Healthy Skepticism Library item: 1771

Warning: This library includes all items relevant to health product marketing that we are aware of regardless of quality. Often we do not agree with all or part of the contents.

Publication type: news

Hawkes N.
How to spot a 'misleading' health story
2002 Jun 5

Full text:

American researchers have found that press releases from medical journals can often lead to misleading stories appearing in the press. Nigel Hawkes, left, Health Editor, reports on how to spot a “dud” story.

What problems were found with press releases from medical journals?

A study from the Dartmouth Medical School in New Hampshire has found that journals tend to emphasise the positive and gloss over any limitations of research. They fail to put results in context, and they seldom mention any potential conflicts of interest, such as whether the author is a consultant with the drug company whose product he has been testing.

In your experience, are the press releases misleading?

No, not especially. They are short and to the point, and seek out the best story – just as a journalist would. It is up to the reporter to read the whole article and confirm that the press release is accurate. Medical journals are highly competitive, and seek to print the best studies. One way of increasing a journal’s prominence is by issuing press releases. So long as a journalist is aware of that, he or she should not be led too far astray.

What are the “tell tale” signs you look out for in a potentially misleading press release?

I am always sceptical if the methods or results are skimped, or if effects are quoted only as percentages. A 33 per cent improved survival after two years, for example, could mean that eight patients survived instead of six – but if 55 were treated, this wouldn’t be a very impressive result. Any studies involving very small numbers of patients should be treated with caution.

What advice would you give readers reading a story based on a journal report?

Look at the numbers involved, the method used and the biological plausibility of the result. A double-blind trial, in which none of the participants or researchers know who is taking the test and who is not (the control group), is the gold standard method.

The source of the study can also help – the more reputable the university, the more likely the work is good. Alone among the top journals, the New England Journal of Medicine does not issue press releases. Reports that quote an independent expert are also a good sign, though there isn’t always space.

Are there areas of research that are particularly prone to exaggeration?

Epidemiology, the study of disease, is liable to produce some wild results, especially if studies are small and poorly designed.

To determine a small effect of, say, passive smoking, requires a huge sample and a really good control group against whom to compare them. Relatively few published studies reach these standards and few of those who write press releases – or stories – have the statistical know-how to gauge the validity of the results.

Press releases issued by biotech companies (not examined by the researchers) are designed to provide “news flow” which the companies use in lieu of profits to keep their investors happy. They are far more unreliable than anything the journals issue.

What pressures encourage press releases to be exaggerated?

It is a normal human instinct to emphasise the positive. Most studies that are published have positive results, or they wouldn’t be published at all. And editors, like anybody else, like studies that tend to go against received wisdom.