Healthy Skepticism Library item: 7074

Warning: This library includes all items relevant to health product marketing that we are aware of regardless of quality. Often we do not agree with all or part of the contents.

Publication type: Journal Article

Reeves BC.
Parachute approach to evidence based medicine: as obvious as ABC.
BMJ 2006 Oct 14; 333:(7572):807
http://www.bmj.com/cgi/content/full/333/7572/807

Abstract:

EDITOR-I strongly oppose the proposition of Potts et al.1 For all the difficulties of doing high quality randomised controlled trials of some research questions, or in challenging settings, making arbitrary decisions about the “obviousness” of the effectiveness of interventions based on observational evidence cannot be the way forward. This point is well made by the readers with expert knowledge who have already responded about the specific examples described by Potts et al.2

Other readers have criticised the naïveté of the argument but seem to accept the underlying sentiment that the world should not always wait for evidence from randomised controlled trials. This point of view presupposes some method, or criteria, for judging obviousness of effectiveness and merits further consideration.

I propose that deciding an intervention is obviously beneficial on the basis of non-randomised evidence requires benefit that clearly outweighs all possible adverse effects (whether frequent and mild, or rare and severe); benefit that could not be explained by bias; and an intervention that is obviously beneficial in all contexts in which it will be applied.

Increasingly, researchers talk about the value of an intervention-the extent to which an intervention does more good than harm.3 Judging value can be extremely difficult. Those who cite the example of the obviousness of the benefits of pencillin often forget that some people may have a serious or fatal allergic reaction to this drug. Historically, even randomised controlled trials have tended to report adverse effects less well than benefits. Although non-randomised or uncontrolled studies may raise the alarm about possible adverse effects, measuring or attributing harm on the basis of these sources of evidence is often difficult. Even when harm is obvious, judging value usually requires quantitative estimates of benefit and harm which the decision maker can regard with equal trust.

There is some evidence about the possible size of bias in non-randomised studies. In prospective studies designed as randomised controlled trials but where concealment of randomised allocation was not done (or where reviewers were unclear about what was done), bias leads on average to exaggeration of effects by 30%4; since some studies with uncertain concealment of allocation will have been unaffected, the true effect of lack of concealment is almost certainly larger. In other examples, such as the effect of hormone replacement therapy on cardiovascular disease, biases in nonrandomised studies seem to have led to a doubling/halving of the effect size found in randomised controlled trials. Interestingly, controversy still exists among topic experts about possible reasons for the differences in the effects of hormone replacement therapy observed in such trials and non-randomised studies.5 Other biases, such as publication and outcome reporting biases, are expected to be worse for non-randomised evidence.

Advocates of the obvious effects of an intervention need to be confident about its benefits in all contexts. Once a decision has been made that an intervention has obvious benefits, its widespread implementation will act to prevent further evaluation.

So next time something seems obvious, see if it passes the ABC test.

Keywords:
Bias (Epidemiology) Evidence-Based Medicine* Harm Reduction Randomized Controlled Trials