Although the answer isn't definite, the book FYI: Does Size Matter? 15 Questions We Aren't Afraid to Answer lends an important clue. In 1578, the word appeared in the book The historie of man. It read: "A certaine Glandulous part, called Thimus, which in Calues...is most pleasaunt to be eaten. I suppose we call it the sweete bread."
That explains why, the first time I witnessed the word sweetbreads being used, I mistakenly thought they were dessert or breakfast breads. Did you ever fall into the same trap?
Source: Flickr User jasonlam