Healthy Skepticism Library item: 16139

Warning: This library includes all items relevant to health product marketing that we are aware of regardless of quality. Often we do not agree with all or part of the contents.

Publication type: Journal Article

Avorn J, Schneeweiss S.
Managing Drug-Risk Information — What to Do with All Those New Numbers
NEJM 2009 Jul 29;
http://healthcarereform.nejm.org/?p=1235

Full text:

In the late 1970s, many pharmacies and governmental and private insurers began electronically recording patient-specific data from filled prescriptions, as physicians, hospitals, and insurers began capturing computerized information on diagnoses, admissions, office visits, and procedures. By the mid-1980s, researchers began trying to use these data to relate drug exposures to adverse events.1 Since then, the volume, clinical detail, and accuracy of such information have increased dramatically, as has the sophistication of methods for discerning patterns of causality in these collections of data.2

Despite the increasing availability of information on medication use and clinical outcomes in tens of millions of patients, the Food and Drug Administration (FDA) has relied primarily on analysis of individual case reports submitted by health care professionals or pharmaceutical manufacturers to assess side effects of approved drugs. Critics, including the Institute of Medicine and the Government Accountability Office, argued that this method was inadequate for systematic postmarketing safety surveillance, and in 2007 Congress passed a law instructing the FDA to develop a system to make use of the terabytes of data on drug exposure and clinical events that the health care system generates. Nearly 2 years later, the FDA requested proposals to build such a “Sentinel” system, to be restricted to private-sector U.S. health insurers and funded at a level somewhere between $7 million and $120 million over 5 years.

Assembling this volume of data from disparate payers will present daunting challenges in informatics and organizational politics, but these issues will be addressable. If someone can go to an ATM in Peru and obtain funds from a bank in Peoria, it should be possible to ascertain whether someone who filled a prescription for rosiglitazone is the same person who was later hospitalized with a myocardial infarction. Tougher administrative hurdles will have to be overcome to analyze data from different insurers and to protect patients’ privacy. Hard work, but not neurosurgery.

Once these problems are resolved, the system will quickly begin to generate “signals,” or indications of higher than expected rates of particular adverse events after the use of specific medications – such as hospitalization for bradycardia in users of a calcium-channel blocker, cardiac valvulopathy in patients prescribed a diet pill, or fulminant hepatotoxic effects in diabetic patients taking an oral hypoglycemic agent. These were all adverse events that eventually led to the removal of drugs from the market3 and represent the sort of effects that the new system should identify more rapidly.

But any such system can also generate errant signals that turn out to be false alarms. This can occur because other factors, not adequately adjusted for, confound the apparent relationship. Physicians generally select treatments on the basis of the subtleties of a patient’s clinical status, as well as their own practice preferences. As automated algorithms search for associations between medication use and adverse events in large observational data sets, rigorous analytic techniques will be necessary to ensure that confounding does not produce spurious associations that could generate safety signals warning of nonexistent hazards. Or – equally problematic – inadequate analysis could conceal a true risk signal that might have been evident with more careful adjustment. Simplistic data mining yielding inadequately adjusted drug–event associations could thus be counterproductive even for first-step signal-generation analysis. Fortunately, more sophisticated approaches are available to mitigate these risk-assessment risks. Partially automated processes based on epidemiologic principles can be used to derive relevant covariate information from large, comprehensive data sets and use them for advanced multivariable adjustment procedures.4

Another challenge will be determining when to consider a signal likely enough to be real to warrant follow-up. If automated “everything-run-against-everything” analyses were performed on data for millions of people taking thousands of drugs, statistically significant associations could emerge as data on a drug–event relationship accumulate, even after adjustment for repeated testing. Such P-value–driven thresholds could result from the size of the population (which will often be enormous) and the strength of the supposed association. But assessments relying on such criteria alone may not consider the severity of the adverse event, whether a safe alternative treatment is available, or how much benefit the drug provides.

The result is a standard decision-analysis problem: How much more knowledge or certainty is likely to be gained by continued monitoring and delaying a warning decision, and will this delay expose patients to unnecessary harm? Conversely, taking action prematurely on the basis of inadequate data could result in unnecessary confusion and harmful discontinuations of useful treatments. We cannot know now what inputs will be optimal for each decision analysis. But stating such inputs transparently up front will help to clarify the decision-making process of regulators who will have to act on these signals. It will also facilitate the communication of decisions, by enabling regulators to frame recommendations or actions in terms of prestated assumptions about acceptable risks for a given product. If such tools are applied well, the system will be able to provide early notice of adverse drug effects that have previously taken years to discover.

The system’s likely computational and analytic accomplishments will quickly present additional challenges. Meeting them will require drawing on disciplines beyond pharmacology and epidemiology, ranging from psychology to ethics to communication. If a drug triples the risk of a potentially fatal outcome (say, a relative risk of 3.0 with a 95% confidence interval [CI] of 2.5 to 3.6), there may be little ambiguity. But many identified risks will be more modest or less certain; what should the FDA do with that information? A relative risk of 1.3 (95% CI, 1.0 to 1.6) might approximate the usual definition of statistical significance, but 1 in 20 such associations may be due to chance alone. Should they, too, lead to a public health advisory or a labeling change? And how should the FDA’s reaction differ if the tripling of risk is for an event that normally occurs only once in a million patient-years, so that few patients are affected in absolute terms – or, conversely, if a 30% increase occurs in the risk of a common event (e.g., myocardial infarction) in a drug used by millions?

Some will argue that the public has a right to know about all possible adverse drug effects. But frequent announcements of possible hazards that may not be real can themselves harm public health. An excessively high threshold for warnings would keep real risks hidden too long, but an excessively low threshold could undermine public trust in drugs, in the surveillance system itself, and in the entire medical enterprise. In Britain in the 1990s, poor management of public cautions about the thrombogenicity of third-generation oral contraceptives resulted in widespread noncompliance with all oral birth-control regimens, which appears to have led to more health problems due to unwanted pregnancies and abortions than would have been caused by the drugs’ side effects.5 Proper implementation of the Sentinel system will require expertise in intelligibly communicating information about risks – in relation to benefits – to clinicians and patients alike.

The Sentinel system will have the potential to identify and quantify adverse-event signals with unprecedented power and speed. In doing so, it could help to optimize medications’ safety and benefit–risk relationships. Getting the system to function will be daunting but achievable, but making sure the numbers it generates are epidemiologically rigorous and clinically helpful will be of paramount importance. Ultimately, knowing what those numbers mean for practice and communicating that meaning effectively will present the biggest challenges of all.

Drs. Avorn and Schneeweiss report being named as participating faculty on an application for a research grant from HealthCore and on a proposal to the FDA for implementation of the Sentinel system. Dr. Schneeweiss reports receiving consulting fees from HealthCore, RTI International, and World Health Information Science Consultants. No other potential conflict of interest relevant to this article was reported.

Source Information

From the Division of Pharmacoepidemiology and Pharmacoeconomics, Department of Medicine, Brigham and Women’s Hospital and Harvard Medical School, Boston.

This article (10.1056/NEJMp0905466) was published on July 27, 2009, at NEJM.org.

References

Avorn J, Everitt DE, Weiss S. Increased antidepressant use in patients prescribed beta-blockers. JAMA 1986;255:357-360. [Free Full Text]
Schneeweiss S, Avorn J. A review of uses of health care utilization databases for epidemiologic research on therapeutics. J Clin Epidemiol 2005;58:323-337. [CrossRef][Web of Science][Medline]
Avorn J. Powerful medicines: the benefits, risks, and costs of prescription drugs. New York: Alfred A. Knopf, 2005.
Joffe MM. Exhaustion, automation, theory, and confounding. Epidemiology 2009;20:523-524. [CrossRef][Web of Science][Medline]
Wood R, Botting B, Dunnell K. Trends in conceptions before and after the 1995 pill scare. Popul Trends 1997;89:5-12. [Medline]