When Medical Studies Collide

Contradictory reports? Meta-analysis may make things more confusing

Two years ago, the headlines blared that echinacea was a bust. Millions of people who believed the best-selling herbal remedy was warding off colds were probably deluding themselves, according to The New England Journal of Medicine. Now echinacea is back in the news. This time, it works! So says a study in The Lancet Infectious Diseases.

How could two studies come to such different conclusions—especially when there have been no new trials of the herb? While the New England Journal reported on one clinical trial, authors of the latest report combined data from previous studies, a controversial approach called a meta-analysis. Its conclusion is dramatically different—not just from that of the New England Journal paper, but also from a review last year of the same studies.

No wonder people are confused. And not just about echinacea. On this roller coaster of health advice, caffeine, carbohydrates, and replacement hormones for women after menopause are bad one day, good the next. One meta-analysis was enough to send sales of diabetes blockbuster Avandia plunging when it suggested heart side effects. "The average reader can't sort it out," says Jeanne P. Goldberg, director of the Center on Nutrition Communication at Tufts University.

The problem is, the world of medical and health research is messier than most people realize. Black-and-white answers are rare, even when it comes to a single drug trial. In hormone replacement therapy, "people from the same study are disagreeing with each other," says Julie Buring, professor of medicine at Brigham & Women's Hospital in Boston. The uncertainties deepen when studies are inconclusive or contradictory. That's when researchers often lump data from a number of trials together in a meta-analysis, hoping the sum will be greater than the parts. But the approach often has pitfalls.

In the case of echinacea, there are dozens of studies, all too small to be definitive. In 2006, Dr. Bruce P. Barrett of the University of Wisconsin School of Medicine and his colleagues examined all the evidence and concluded the herb didn't seem to prevent colds and had only a small effect on duration. Now, a team led by professor Craig I. Coleman of the University of Connecticut School of Pharmacy has done a meta-analysis on the same body of evidence. Echinacea not only makes colds shorter, they conclude, the herb can also prevent people from catching a cold.

The disparity illustrates the hard choices researchers must make when analyzing batches of studies. "The biggest issue is how to go about selecting studies to go into the analysis," says biostatistician Susan S. Ellenberg of the University of Pennsylvania School of Medicine. In Barrett's review, the team threw out some studies in which researchers exposed volunteers to cold viruses, and others in which people were taking supplements such as vitamin C in addition to echinacea. Coleman's meta-analysis included both types.

In addition, Coleman used only published studies, while Barrett included two unpublished ones. That decision can have a big impact. "We know there is publication bias," says Frank E. Harrell Jr., chair of biostatistics at Vanderbilt University. It's much easier to get a study published that says, "something works!" than one saying, "Oops, the treatment had no effect." Using published data alone thus typically makes the final result more positive.


What's more, meta-analysis is still a bit of a black art because it combines data from studies done with different methods and protocols as though the data came from one large trial. Sometimes the data are too different. The latest study on echinacea "overstepped" in tossing together "apples, oranges, and grapefruit," says Barrett. A legitimate criticism, Coleman concedes. "To be honest, we are lumping apples of different varieties," he says. "It is a limitation we admit to."

Meta-analyses may also mislead by relying on data reported in papers rather than on original raw data, which are usually kept secret. "Good raw data from one study can be worth 50 studies in a meta-analysis," says Vanderbilt's Harrell.

With echinacea, it's impossible to know who is right. Even Coleman advises caution. "This is not the end-all, be-all that echinacea works," he says. "We are just trying to say that it may work."

That, of course, doesn't make good news copy. Laments Goldberg: "I believe passionately that headlines do more damage than anything else." But if people understand the process of science better, they'll be able to spot the gray reality behind the next black-and-white headline.

By John Carey with Arlene Weintraub in New York

— With assistance by Arlene Weintraub

    Before it's here, it's on the Bloomberg Terminal.