Healthy Skepticism International News

October 2006

Don’t judge a paper by its abstract

ABSTRACT

Objective:  The author is a consultant child & adolescent psychiatrist who became interested, perplexed and then troubled after closely comparing the abstract of a highly important paper to his clinical practice with the paper’s own results in the body of its text. The paper in question was the “Fluoxetine, Cognitive-Behavioural Therapy and Their Combination for Adolescents with Depression - Treatment for Adolescents with Depression Study (TADS) Randomised Control Trial”. His analysis of the paper was discussed via email with several colleagues.

Conclusion: The author believes the TADS paper highlights problems in current research and medical publication. Abstracts may reflect bias of the researchers or study sponsors and distort findings of the research. Readers need to go beyond abstracts to critically analyse results and methodology.

BACKGROUND
I write this paper as a consultant child & adolescent psychiatrist, a clinician not a researcher. Having worked for most of the past decade on a busy inpatient unit and also a mood disorders unit for young people, I have treated many depressed adolescents and am no stranger to prescribing antidepressants. However, like many of my colleagues, in response to controversies in the literature and clinical experience, I now find myself doubting the safety and perhaps efficacy of these drugs.

Major depressive disorder (MDD) and dysthymia are recognised as common disorders in this age group (1), (2). Child & adolescent mental health services are under-resourced to cope with demand (3), (4).  A review of the efficacy of tricyclic antidepressants (5) found them to lack efficacy above placebo, except for perhaps a small benefit in older adolescents. Furthermore, the risk of death from overdose and case reports (6) of sudden cardiac arrest have together caused tricyclic antidepressants to be dropped for this indication in the under 18 age group. Thus the advent of the apparently safer selective serotonin reuptake inhibitors (SSRIs) offered hope of a useful therapeutic option for young people with MDD. 

Placebo effects are strong in this age group (7) and even if efficacy is weak, prescribed medication may enhance therapeutic alliances with contrary adolescents and may warrant a place in an overall treatment plan that includes psychotherapies plus attention to relevant family, social and substance misuse factors.

However, trials of SSRIs (when full data were eventually published) raised serious questions about the safety of these drugs. A small but significant number of children and adolescents in trials of paroxetine, sertraline, citalopram, venlafaxine and fluoxetine showed increases in suicidal and aggressive ideation and behaviour relative to placebo (8). While numbers in each individual trial were too small for statistical significance the trends were consistently in the direction of more events in the SSRI treatment groups.

Aside from safety issues, do SSRIs work? A series of industry sponsored studies were said to indicate “yes – a bit”, but this was challenged by a meta-analysis (8) which failed to detect convincing effects on primary outcome measures. Analysis of suicide trends and SSRI prescribing has been argued to support the case for efficacy (9), (10), (11), but this too is a matter of debate (12), (13).

The UK’s Medicines and Healthcare products Regulatory Authority (MHRA), which presumably had reviewed all available trials issued an alert on SSRIs and venlafaxine on 9 December 2003, which exempted fluoxetine (14). At the time I was working in a community CAMHS clinic in the UK, I was generally prescribing citalopram where I felt pharmacotherapy needed.  I briefed the patients and their parents about the risks, and none reported agitation on treatment; I ceased medication for some and changed others to fluoxetine and left some on citalopram.

TREATMENT OF ADOLESCENTS WITH DEPRESSION STUDY (TADS)

Thus it was with great interest that I viewed the Treatment of Adolescents with Depression Study (TADS) (15) in The Journal of the American Medical Association (JAMA). This US$17million NIMH run multicentre trial randomised 439 12-17 year olds with MDD to fluoxetine alone, placebo alone, CBT alone, or fluoxetine with CBT. The abstract for TADS is the main subject of study in this paper and (with JAMA’s permission) is reprinted here:

Context Initial treatment of major depressive disorder in adolescents may include cognitive-behavioral therapy (CBT) or a selective serotonin reuptake inhibitor (SSRI).

However, little is known about their relative or combined effectiveness.

Objective To evaluate the effectiveness of 4 treatments among adolescents with major depressive disorder.

Design, Setting, and Participants Randomized controlled trial of a volunteer sample of 439 patients between the ages of 12 to 17 years with a primary Diagnostic and

Statistical Manual of Mental Disorders, Fourth Edition, diagnosis of major depressive disorder. The trial was conducted at 13 US academic and community clinics between spring 2000 and summer 2003.

Interventions Twelve weeks of (1) fluoxetine alone (10 to 40 mg/d), (2) CBT alone,

(3) CBT with fluoxetine (10 to 40 mg/d), or (4) placebo (equivalent to 10 to 40 mg/d).

Placebo and fluoxetine alone were administered double-blind; CBT alone and CBT with fluoxetine were administered unblinded.

Main OutcomeMeasures Children’s Depression Rating Scale-Revised total score and, for responder analysis, a (dichotomized) Clinical Global Impressions improvement score.

Results Compared with placebo, the combination of fluoxetine with CBT was statistically

significant (P=.001) on the Children’s Depression Rating Scale-Revised. Compared

with fluoxetine alone (P=.02) and CBT alone (P=.01), treatment of fluoxetine

with CBT was superior. Fluoxetine alone is a superior treatment to CBT alone (P=.01).

Rates of response for fluoxetine with CBT were 71.0% (95% confidence interval [CI],

62%-80%); fluoxetine alone, 60.6% (95% CI, 51%-70%); CBT alone, 43.2% (95%

CI, 34%-52%); and placebo, 34.8% (95% CI, 26%-44%). On the Clinical Global Impressions

improvement responder analysis, the 2 fluoxetine-containing conditions were

statistically superior to CBT and to placebo. Clinically significant suicidal thinking, which

was present in 29% of the sample at baseline, improved significantly in all 4 treatment

groups. Fluoxetinewith CBT showed the greatest reduction (P=.02). Seven (1.6%)

of 439 patients attempted suicide; there were no completed suicides.

Conclusion The combination of fluoxetine with CBT offered the most favorable tradeoff

between benefit and risk for adolescents with major depressive disorder.

JAMA. 2004;292:807-820 www.jama.com

ANALYSIS OF THE ABSTRACT
I, and my child psychiatric colleague in our CAMHS team relaxed a bit about our patients on SSRIs and considered changing them all to fluoxetine to feel more medicolegally secure. But asked by a colleague* to critique the paper, I spent a weekend reading it and became increasingly perplexed about the abstract. Chasing the issue with colleagues (see acknowledgements) my concerns were both clarified and amplified. Our letter of concerns about the TADS study and its interpretation was refused by JAMA and later published by the British Medical Journal (BMJ) (16).

Our concerns were plural:

Interventions:  Despite informing us that two of the arms of the trial were blinded and two were not, the Abstract reports direct, and therefore invalid, statistical comparison between these. Essentially there are two studies: a double-blind RCT of fluoxetine vs placebo and an unblinded study of 12 sessions of CBT vs 12 sessions of CBT plus ‘open label’ fluoxetine. The second study would require CBT plus placebo to evaluate the efficacy of CBT plus fluoxetine, both arms could have been blinded for the medication/placebo aspect, but this was deemed too expensive and “not naturalistic”.

Main outcome measures:  The primary outcome measure used was the Children’s Depression Rating Scale-revised (CDRS-R), a 17 to 113 point scale. The self-report Reynold’s Adolescent Depression Scale (RADS) and Suicidal Ideation Questionnaire –junior high school version (SIQ-jr) were also used. Clinical Global Impression (CGI) was used as a categorical (dichotomised) outcome measure. The Food and Drug Administration (FDA) stipulates that pharmaceutical trials should use dimensional measures in preference to categorical, given their typically greater sensitivity and validity (17).

Results: CBT + fluoxetine is stated to be statistically superior to placebo alone using the CDRS-R—but this comparison of a non-blinded with a blinded arm is invalid. The same criticism invalidates the second claim that CBT + fluoxetine is superior to both fluoxetine alone and CBT alone. The third claim is that fluoxetine is superior to CBT; again invalid for the same reason.

The percentage differences between fluoxetine and placebo quoted in sentence 4 are impressive, but they derive from the CGI and the CDRS-R used categorically. These are not reflected in the CDRS-R used dimensionally, where fluoxetine fails to statistically differentiate itself from placebo (though a trend is present: p = 0.10). Apart from the issue of statistical significance, a 3-point difference on a 96 point scale is hardly likely to be clinically significant.

The last 3 sentences report suicidal thinking to improve in all 4 groups with best effect from CBT + fluoxetine. In the body of the paper, the authors split adverse events into multiple groups (harm related, suicide related and non psychiatric) so there is reduced power to detect significant differences. Nevertheless, the total fluoxetine vs no fluoxetine comparison found a significant doubling of harm events (odds ratio 2.19, CI 1.03-4.62), and for psychiatric related adverse events (table 4 of the paper) the OR was 2.57 (CI 1.11-5.94) for fluoxetine vs placebo. The 7 mentioned suicide attempts include 6 from the fluoxetine treated groups and 1 from the placebo group. In the comments section of the paper the authors discount this result as impossible to analyse given the small numbers, but chi square analysis (not conducted by the authors) is significant at p=0.05, and it is of interest that this result was neglected (18).

Furthermore the authors of the TADS paper could have referred to previous trials involving fluoxetine in the paediatric age group. Compilation of such safety data was what prompted the United States Food and Drug Administration (FDA) to its later application of a warning to fluoxetine along with the other SSRIs.  This data is referenced in the current information statement of the Adverse Drug Reactions Advisory Committee (ADREC) of Australia’s Therapeutic Goods Administration (TGA) (19).  As one finds through links from the ADREC statement online, Dr Mosholder, epidemiologist with the FDA had specifically recommended incorporating data from the TADS with his meta-analysis of SSRI’s and suicidality - but this is another story beyond scope of this paper.

Conclusion:  CBT + fluoxetine is best. The implication is that adolescents with MDD should receive fluoxetine (as other SSRIs have accrued warnings for this age group) and possibly CBT, if available. 

OPINION

Readers are encouraged to read the full text to make up their own minds, but I concluded that this abstract, on a highly important topic to my clinical practice, was a work of considerable ‘spin’. Lenzer (20) comes to the same conclusion and points out this spin was not detected by most medical journalists; reports went out in the popular press with headlines such as: “Prozac plus talk is best for teen depression” (Washington Post).

I doubt I’m alone amongst clinicians in being guilty of reading mainly abstracts and conclusions, sometimes discussions and comments, rarely methodology and results.  This practice is reinforced by online Pubmed/Medline searches as a convenient, immediate way to find needed information. A little voice over the years nagged, an echo of a half-forgotten tutorial: “always read introduction, methodology, and results first, make up your own conclusion and only then read the authors’ conclusion and abstract.” One rationalises not doing this because of time pressure, convenience, not always having immediate access to full text of papers and an implicit belief in the wisdom and benevolence of researchers, journal editors and reviewers.

These sloppy habits persist despite my awareness and indeed membership of Healthy Skepticism, which I joined when working as an intern alongside the organisation’s founder, Dr Peter Mansfield in 1983. We had each done medical electives on the Indian subcontinent; whilst I’d been immersed in surgery as well as curry, he had noted the dubious drug promotion and prescribing practices in the Third World. I was horrified to hear of well known pharmaceutical companies dumping anabolic steroids in children’s growth tonics (Pfizer ), arsenic, strychnine, vitamins and alcohol for stress (Bayer),  and chloramphenicol/ streptomycin for children with diarrhoea (Parke Davis / Warner Lambert ) (21). But after some early enthusiasm my erratically financial membership didn’t amount to activism, until asked to closely read and critique the TADS paper. Just as a death on one’s own street has more impact than thousands in a foreign disaster zone, I was prompted by my clinical proximity to TADS to tackle the Abstract issue head-on.

Journals are not unaware of the problem and the looming chasm of a credibility gap.  Bias (and it often may be justified or rationalised in the mind of authors) in the published literature appears to be widespread (22). The BMJ declared registration of clinical trials should be a precondition for publication, noting “undisclosed trials and duplicate and selective publication sting government agencies, clinicians, researchers, and journals ever more frequently and painfully.” (23). I have heard prominent colleagues say “journal X is not worth reading”, after my reading of the TADS paper in JAMA I wondered if I needed to add JAMA to the list? Bias in research and editing need not always be consciously intentional, there are many systemic factors at play. Nonetheless bias affecting research threatens public faith in the credibility of our whole profession (24), (25).

Of course some degree of ‘spin’ is an integral part of daily practice in psychiatry.  CBT, family therapy, parent training and other talking therapies are often ways to help our patients take a “glass half full” view of their world. But good psychotherapy is always about helping people cope with reality. It is not to have them see a half glass as fully full.

Which brings us back to the TADS. The abstract only mentioned the positive findings, weak as they seem to be on closer analysis. The empty or even harmful findings in the study with respect to the active pharmaceutical substance received no prominence. To my closer reading I can only conclude that the abstract and comments sections were a case of smoke and mirrors, of making a more than half empty glass appear full. We and certainly our patients need far better from our journals and research community.

The FDA, in its hearings on use of antidepressants in children and adolescents ended up including fluoxetine with the other SSRI antidepressants in its black box warning of increased risk of suicidal ideation, despite having the results of the TADS paper. This suggests that the FDA did not fully agree with the conclusions in the TADS paper abstract (26).

The principal author of the TADS paper replied to the criticisms of methodology (16) in the rapid response letters in the BMJ (27). However this did not appear to adequately address the main criticism of comparing double blinded and unblended arms in the study. Readers are encouraged to read this correspondence and make up their own minds.

After the TADS back to the dilemma

So much for trusting abstracts for guidance. But as a clinician wondering what if any is the place of SSRIs in child and adolescent psychiatry, I am still left with the original dilemma I had briefly hoped the TADS study resolved.

Determination of whether the safety risks of SSRIs outweigh (largely placebo) benefits awaits accurately informed debate – though it is good to see this commencing (28).  Epidemiological studies in Scandinavia (29), (30) and USA (31), (32), (11) suggest SSRIs have not caused suicides and in fact have possibly prevented suicides, however confounding variables abound in population studies of SSRI use at a time when many resources have been marshalled to prevent youth suicide and suicide in general (13). Australian data (33) suggests a decrease in suicide correlates with increased SSRI prescribing in older adults, though the converse in younger adults and no change in the 15 to 24 age group.  On a different, but highly important topic in the decision to prescribe, recent papers (34), (35) suggest adverse effects extend to problems with growth hormone and testicular atrophy – not problems one wishes to bestow on adolescents.

Like other colleagues I have seen patients with depression who appear to respond well to SSRIs, is this always mainly placebo effect? Part of the problem is likely to be as Parker suggests – that MDD is a basket for heterogenous disorders, those likely to truly respond to antidepressants are submerged in research samples of many depressive syndromes that do not (36).

In terms of guidance and a seemingly unbiased summary of this area, I found the paper by Licinio and Wong (37) worth reading. Finally in my current practice I still prescribe SSRIs when family, school and individual psychosocial interventions fail to be sufficient: for OCD and where anxiety is prominent, for which there is a stronger evidence base (38), but much more selectively in depression, very conscious of the risk/benefit ratio and wondering on what evidence base I do so, or even what evidence base to trust.

Dr Peter I H Parry

MBBS, FRANZCP, Cert. Accred. Child & Adol. Psychiatry

Consultant Child & Adolescent Psychiatrist

Eastern CAMHS, 5 Darley Rd, Paradise SA 5075, Australia.

61 8 82078999ph 61 8 83652221fax, .(JavaScript must be enabled to view this email address)

Acknowledgements
The following colleagues were part of the email discussion about the TADS paper that deepened my concerns about the way the results were presented and offered comment on my first draft of this paper:
*Prof David B Menkes, Dept Psychiatry Wrexham, Wales, UK – also made initial request to me to read and comment on the TADS paper.
Dr Jon Jureidini, Dept Psychological Medicine, Women’s & Children’s Hospital, Adelaide, South Australia

Dr Peter Mansfield, Research Fellow, Department of General Practice, University of Adelaide, South Australia; Director, Healthy Skepticism Inc.

Dr Anne Tonkin, Clinical Pharmacologist, University of Adelaide and Royal Adelaide Hospital

A/Prof Chris Doecke, Pharmacy Department, Royal Adelaide Hospital and School of Pharmacy and Medical Sciences, University of South Australia, Adelaide, South Australia

Conflict of Interest
The author is a member of Healthy Skepticism.  He has received payment on 3 or 4 occasions in the late 1990s for giving presentations to general practitioners at pharmaceutical company sponsored dinners on child psychiatric topics (ADHD, Crisis Intervention, Depression).
——————————————

View/Hide References

HS Int News index

Page views since 15 March 2010: 7814

Comments

Our members can see and make comments on this page.