Why scientific reports in the media must be read properly

As someone who works in higher education and is responsible for “sexy” titles for scientific research articles, I have held a certain skepticism for the media when they rush out with incredible claims about scientific research. This was particularly the case today, when The Guardian breathlessly announced that “Up to 25 cups of coffee a day safe for heart health, study finds.”

This study, by Queen Mary University of London and the British Heart Foundation was released yesterday at a conference. That was among my first red flags of many, and there were many.

1. Where was the paper published?
2. Is the paper peer-reviewed?
3. What was the sample size?
4. What was the original hypothesis?
5. What is the definition of “safe”?
6. What was their random sampling procedure?

Now, because I am THAT guy, I went and found the paper. To be honest, it wasn’t too hard to find – https://heart.bmj.com/content/105/Suppl_6/A8.2

The first thing I noticed is that it is an extremely short paper. It was presented at a conference, which means it has not been peer-reviewed. That doesn’t mean it’s a bad paper, but it certainly hasn’t been subjected to any rigorous examination by independent scientists at this point.

The sample size was pretty decent, at more than eight thousand people, but my concern was the people that were chosen. They selected more than eight thousand middle aged people, who drank anywhere between 0 and 25 cups of coffee a day.

Okay, fine, Fry wasn’t middle aged, but the point still stands. How many people were drinking more than 25 cups of coffee a day? It turns out not many. Even the scientists grouped people into people that drank:

• 1 or less cups of coffee a day;
• 1 to 3 cups of coffee a day; and
• 3 or more cups of coffee a day.

Now, on that basis alone, the 3-25 cups of coffee group creates some enormous variables within it, and to say that that group are “heavy” coffee drinkers seems a little on the absurd side.

I don’t drink much in the way of coffee – I drink a cold brew coffee once or twice a week. However, 3 cups of coffee a day does not seem like a “heavy” amount to me.

The study also pointed out that moderate and heavy coffee drinkers were more likely to be male, smoke and consume alcohol regularly.

So what?

Are the researchers trying to indicate that drinking coffee will reverse the effects of smoking and drinking alcohol? Of course not, but they’ve now put that idea out there.

They apparently adjusted for “age, sex, ethnicity, Townsend deprivation index, current smoking, higher levels of education, height, weight, regular alcohol consumption (≥3 times/week), systolic blood pressure, resting heart rate, presence of hypertension, hypercholesterolaemia or diabetes, intake of vegetable, meat, water and tea consumption.” However, there’s no data to show the impact of moderate to heavy coffee consumption on the young, the old, female, the tall, the short, the fat, the thin… I could go on.

This study fundamentally stinks of p-hacking. If you think you’ve heard this phrase before, Last Week Tonight with John Oliver discussed its in a segment on scientific studies a few years ago.

P-hacking is fundamentally the practice of looking for a hypothesis that suits some data (look at the 3:18 mark for a brief discussion of p-hacking).

What are we supposed to do?

The first thing you should do is read the article properly, not just the headline. The headline is made to be catchy for the specific purpose of grabbing media attention. It is known that the wider society does not truly understand some of the science done at universities and research institutions around the world.

Ask difficult questions about the article. Was it published in a reputable journal? Was it peer-reviewed? How big was the sample size? Were the tests done on animals or humans?

Sometimes you get all the right answers to all of these things, and good answers to other questions.

Make sure you always ask questions – it’s a fundamental part of science