Tthere is no reason for you to take meds for something that "may" happen to you, just because it happens to other people.
Family history is important. If you have a very close relative, especially a parent, who has or had a particular condition, such as heart disease, statistically, you are more at risk for developing the same condition. Of course, that does not mean you will develop the condition, but your risk is greater. (I once shocked a cardiologist by mentioning that you cannot say that something will happen to an individual based on a group statistic; he responded, "You get that!")
Because of my father's severe heart disease, on my doctor's recommendation, I've been taking an 81-mg aspirin a day for more years than I can now remember, and I think nothing of it.
By the same token, so far I have never had a colonoscopy because there is absolutely no history of gastrointestinal cancer in my family. My mother died of a stroke, and my father, as I mentioned above, has severe heart disease. In my own opinion, I am at greater risk for heart disease than I am for gastrointestinal cancer.
I guess my father's pancreatic cancer qualifies as "gastrointestinal," since he's now under the care of a gastroenterologist, but that has nothing to do with his stomach or intestines, and in March he will be 88 years old. My own opinion is that it's more likely that his heart disease will kill him before the pancreatic cancer will.
Of course, anyone has the right to refuse to go on any medication (my father refuses chemotherapy for his pancreatic cancer), but that does not mean the doctor is wrong about risk because of age, especially if a parent had the condition in question.
In my opinion, this is not an example of ageism. This is responsible medical practice.
ETA: I should have said that my comments are based on what I see on my job. However, I stand by my opinion that this is not an example of ageism.