Healthy Skepticism Library item: 7103

Warning: This library includes all items relevant to health product marketing that we are aware of regardless of quality. Often we do not agree with all or part of the contents.

Publication type: Journal Article

Deeks JJ.
Word limits best explain failings of industry supported meta-analyses.
BMJ 2006 Nov 11; 333:(7576):1021
http://www.bmj.com/cgi/content/full/333/7576/1021-a

Abstract:

Although few doubt that industry funded systematic reviews sometimes use poor methods and misrepresent findings, Jørgensen et al overestimate this bias and misattribute differences in methods and reporting to it.1 Median quality scores of the included reviews were 7 for Cochrane reviews, and 2, 2, and 3 for industry funded, undeclared funding and non-profit or no funding journal reviews. These results are best explained by word restrictions, not financial support. For example, the BMJ paper and online versions of the included celecoxib review were restricted to 2211 and 3425 words,2 whereas the unrestricted Cochrane review has 6002 words.3

The reliability of unblinded quality assessments raises concern, at least for the celecoxib reviews. Contrary to the findings of Jørgensen et al, the reviews gave equivalent detail concerning allocation concealment, and the industry funded review contained four paragraphs with reservations about results. Interestingly, the BMJ deleted two of these paragraphs to shorten the review for the paper journal. Had Jørgensen’s study only included published reviews, the comparison by funding source could have been blinded and controlled for word length.

The industry celecoxib review was produced by an experienced Cochrane reviewer, protected from industry interference by a contract allowing freedom to publish (including results of previously unpublished trials). Data were extracted from full industry reports, avoiding problems extracting detail from abridged journal articles, such as the JAMA report of the CLASS trial 4. The Cochrane review did not have access to this level of information.

Assessment of the likelihood of bias in reviews, including Cochrane reviews, should always be based on the methods and completeness of results, not on prejudices about the organisations from which they emanate. The lesson from the paper by Jørgensen et al may be more that journals should reverse the trend of reducing word lengths, and give authors the opportunity to explain methods in the detail afforded by the Cochrane review format.

Keywords:
Bias (Epidemiology) Drug Industry* Meta-Analysis* Research Support*