Healthy Skepticism Library item: 1764

Warning: This library includes all items relevant to health product marketing that we are aware of regardless of quality. Often we do not agree with all or part of the contents.

Publication type: news

Kolata G.
In Public Health, Definitive Data Can Be Elusive
New York Times 2002 Apr 23
http://www.nytimes.com/2002/04/23/science/in-public-health-definitive-data-can-be-elusive.html?pagewanted=1

Full text:

A debate over using hormone replacement therapy in middle-aged women is focusing new attention to a quandary in science and medicine – when different types of studies reach very different conclusions, what can be called truth? And what constitutes evidence?

The issue pits disparate results from observational studies, in which large groups of people are tracked, often for years, against randomized clinical trials, in which participants are randomly assigned to receive, for example, a drug being tested or a dummy pill.

Observational studies have been invaluable in showing that smoking leads to cancer and heart disease or that air
pollution can be dangerous to health. But they have limitations not shared by randomized studies, in which participants are randomly assigned to have a treatment or not. On the other hand, randomized studies are time consuming, expensive and not always feasible.

So on many questions in public health and medicine, researchers have relied on observational studies alone to reveal the truth.

“This is the big issue in epidemiology,” Dr. James Robins, an epidemiologist and a statistician at the Harvard School of Public Health said. “The question is what effects can we reliably detect?”

Though they are not the first randomized studies to contradict findings in observational studies, the estrogen studies are leading to a problem. Some would call the disparity a crisis. The conclusions that hormonal therapy may have important benefits arose from observational studies that were models in medicine. If those studies were wrong, then why? How many other widely held beliefs that emerged from the same studies or ones like them have to be held up to question?

It is, Dr. Robins said, “a research question,” one whose outcome affects almost everything that is known about public health and preventive medicine. The most recent turn in the estrogen debate occurred when a prestigious group of 28 scientists and doctors, authors of the International Position Paper on Women’s Health and Menopause, said hormone replacement therapy’s established benefits were much more limited than many doctors and women had believed.

Estrogen can ease hot flashes and night sweats in women going through or who have passed through menopause, and it can stem the bone loss that accelerates with menopause. But, the group said, it remains to be established whether it protects against heart disease, Alzheimer’s disease, broken bones from osteoporosis, severe depression and urinary incontinence, theories that observational studies suggested.

Committee members said they were absolutely convinced that they were right to voice their skepticism, given that so many women are taking hormone therapy in the belief that it has been proved to be helpful.

“It is mind boggling to give 20 million American women a drug with major side effects without definitive proof that it works,” said Dr. Deborah Grady, a professor of medicine and epidemiology at the University of California in San Francisco.

Others say they are deeply troubled and puzzled. They note that observational studies are hardly trivial research and that they are in any event the only way scientists can ethically address questions questions like whether day care affects children’s development, whether watching violence on television leads to criminal behavior or whether environmental or occupational exposure to chemicals like benzene, arsenic, asbestos and lead is dangerous.

“For the majority of questions, we have no choice but to use observational studies,” Dr. Karin B. Michels, an epidemiologist at the Harvard Medical School, said. “They are our most important tool in public health.”

But such studies have a fundamental drawback. People choose their treatments or behaviors, and those who choose to take vitamin pills or exercise, for example, are quite different from those who do not.

Women who use hormone replacement therapy, for example, are thinner, more likely to eat healthy diets, less likely to smoke or drink heavily and more likely to exercise. If they are healthier than other women, is it the hormones, some other behavior or a combination of factors?

Statisticians have accounted for such differences in their analyses. But they have to hope that they did not miss other equally important factors and that they have measured all the important factors when they acquired their data.

In randomized studies, participants are randomly assigned to have treatments or not, and the outcomes in the two groups are compared. The advantage is that treatments are not chosen by individuals but assigned by a figurative toss of a coin. But there are many questions that such studies cannot answer, because they would be infeasible or unethical, among them, whether heavy drinking leads to liver disease.

With hormone replacement, the conclusion from studies that followed hundreds of thousands of women for years was that women who used it were healthier, with less heart disease and longer lives. In contrast, the emerging results of randomized studies are not finding any such effects.

“This is a bizarre time in history,” Dr. Robins said. If ever there was a situation where he would have believed data from observational studies, he said, it was with hormone replacement therapy, because the studies were so well conducted.

Even the acting director of the largest of the new randomized trials of hormone replacement, the Hormone Replacement Therapy Trial of the National Institutes of Health’s Women’s Health Initiative, expected the two types of studies to agree.

“I was quite surprised,” said the official, Dr. Jacques E. Rossouw, who is at the National Heart Lung and Blood Institute. Hormone replacement “may be something of a classic.”

It will be “taught in the future to illustrate the pitfalls” of relying on observational studies to reveal the truth.

Yet despite their generally exalted status in medicine, randomized studies have been wrong, said Dr. Richard Peto, an epidemiologist at Oxford University in England. Small randomized studies, Dr. Peto said, concluded that vitamin E protected against heart disease, as did large observational studies. It was only very recently, when the question was studied with a randomized study involving 50,000 people that the truth emerged, he said, adding, “Vitamin E does not work.”

There is a reason the truth is so hard to find, he said. Increasingly, researchers are looking for very small effects like a tiny edge in a battle against heart disease. If an effect is huge like the increased risk that a person who smokes will develop lung cancer, an observational study will correctly find it. If a treatment is truly sensational, Dr. Peto said, its benefits will be so clear that any study, randomized or observational, will find them.

The problems occur in questions like the one on estrogen, he said, “when the effects aren’t very big.” If there is no effect or just a tiny one, chance comes into play, making one study turn out slightly positive and another slightly negative, and any flaw in a study or its analysis can throw the results into the wrong column.

Dr. Michels has a particular stake in the estrogen debate because she works with data from one of the biggest and,
most experts say, one of the best observational studies, the Nurses’ Health Study. It saw, correctly, that smoking led to heart disease and lung cancer. It also found that smokers who stopped reduced their risk. But it also found the vitamin E effect on heart disease and it found the benefits of hormone replacement therapy that are now under question.

The nurses’ study began at Harvard in 1976 with 121,000 nurses ages 30 to 55. The women regularly describe their medications, diets, smoking histories, medical problems, exercise habits and general health. In 1989, the study’s directors enrolled a second generation of 116,000 nurses, ages 25 to 42. Researchers said they could hardly imagine a better group to observe. The women’s responses were exemplary in accuracy and completeness. But there remained the thorny problem of correcting for factors other than the hormone therapy that might account for the results.

Dr. Rossouw explained some complications, saying: “One-third of the women who start taking estrogen are no longer on it at the end of one year, and half are no longer on it by the end of two years. The small group that takes it for many years is very unusual. They are very conscious of their health and of doing things to promote their health.

“It may not be the pill you are taking, but the fact that you are the kind of person who keeps taking pills.”

Although he said the nurses’ and other observational studies overestimated the benefits of estrogen, Dr. Rossouw said it was too soon to say what the randomized study would find. The plans for the initiative’s study called for more than 27,000 women from 50 to 79 to be followed for eight to 12 years, unless evidence emerged that hormone therapy was overwhelmingly beneficial or overwhelmingly harmful. With about three years to go, neither case has occurred, and the trial continues.

The women taking estrogen seem to have slightly more heart attacks and strokes. The study directors notified them of that finding, but the effects are too small to halt the trial, its monitoring board found.

Other recent randomized studies with women who already had heart disease found that estrogen increased the risk of subsequent heart attacks and strokes. Women who had early Alzheimer’s were not helped by taking estrogen.

A general rule in medicine, Dr. Rossouw said, has been that treatments that reduce the risk of disease in healthy people are even more effective in people who already have the disease.

At Harvard, statisticians, including Dr. Michels and Dr. Robins, are planning ways to look at the data from the nurses and find what went wrong. Did the analyses miss something? Was there a better way to make statistical adjustments?

“There must be an explanation,” Dr. Michels said. “There is only one truth.”