It's weird, because objectively speaking we should think consuming breast milk is fine -- heck, it's designed for human consumption! -- and that drinking something that comes from a cow's or goat's udder is gross. Instead, it's the other way around.
I'm guessing it has something to do with our natural response of disgust to bodily fluids. That disgust is a hardwired trait that protects us against disease. But it's not always rational. It acts independently of the actual disease potential.
Why, in commercials, do they always use blue liquid to demonstrate the absorbency of diapers or feminine-hygiene products? Because if they used fluid that looked more, um, organic, people would be grossed out. I read somewhere about a study showing that people are more repulsed by red or yellow fluids in that kind of context, even if they're just colored water.
Have you ever seen one of those TV segments where they go into a hotel room and use some special light to show all the body fluids that are typically found on the bedspread and elsewhere? Gross and horrifying, right? Yet people typically don't get sick whenever they stay in hotels. Those body fluids generally aren't a real health hazard. But they do trigger our disgust reflex.
Complicating the disgust response for breast milk is that it's the product of an intimate relationship between the mother and baby, so there's a sense of intrusion about drinking it. Most people aren't disgusted by the baby drinking it. Though I'll have to say, it is kind of a weird phenomenon, when you think about it (and I have been on both ends of the process).