Is a meta-analysis always stronger evidence than a single randomised controlled trial?

Not automatically. A meta-analysis sits at the top of the traditional evidence hierarchy in principle, but a poorly conducted meta-analysis — one that pools highly heterogeneous studies or fails to account for publication bias — can actually be less informative than a single well-designed, adequately powered RCT. The design confers potential, not guaranteed quality.

What is I² and why does it matter?

I² is a statistic that quantifies the proportion of variability in a meta-analysis that's due to genuine differences between studies rather than chance. An I² above roughly 75% is considered high heterogeneity, suggesting the included studies are measuring something sufficiently different that combining them into a single pooled estimate may be misleading, even if it's mathematically possible.

How do I know if a meta-analysis has accounted for publication bias?

Look for a funnel plot and a formal statistical test for asymmetry (such as Egger's test or Begg's test) in the methods or results section. Also check whether the authors registered a protocol on PROSPERO before conducting the review — pre-registration makes it harder to selectively report favourable findings after the fact.

What does GRADE mean when I see it in a systematic review?

GRADE (Grading of Recommendations Assessment, Development and Evaluation) is a framework for rating the overall certainty of a body of evidence. It grades evidence as high, moderate, low, or very low certainty based on factors including risk of bias, inconsistency between studies, indirectness of the evidence, imprecision of results, and publication bias. A statistically significant pooled result can still receive a 'very low certainty' GRADE rating.

Why do two news articles sometimes cite the same meta-analysis to support opposite conclusions?

Usually because both articles are working from the abstract rather than the full paper, and abstracts often emphasise the statistically significant headline finding without adequately conveying the effect size, heterogeneity, certainty rating, or limitations. A result can be real and statistically significant while being small in magnitude, applying only to a narrow subgroup, or resting on low-certainty evidence — all context that gets stripped away in a news cycle.

Meta-Analysis: Pooling the Evidence

By Hannah Bui · 22 April 2026 · 7 min read

A microscope sitting on top of a table next to a bottle

Somewhere between the fourth and fifth time I saw the same single meta-analysis cited to support completely contradictory claims in the same week, I decided I needed to write this piece. That's not an exaggeration , same paper, same pooled result, two opposing headlines. Once you understand how meta-analyses work, that kind of thing stops being surprising. It starts being expected.

So let's talk about what a meta-analysis actually is, what it can and cannot tell us, and how to read one without getting played by whoever's spruiking it.

What a meta-analysis is (and what it isn't)

A meta-analysis is a statistical technique that combines the results of multiple independent studies on the same question into a single pooled estimate. The idea is that by aggregating data across many trials, you reduce the noise any single study carries and get a more stable signal. On paper, it sits near the top of the traditional evidence hierarchy , above individual randomised controlled trials, observational studies, and definitely above expert opinion or case reports.

But here's the thing: a meta-analysis is only as good as the studies that go into it. Garbage in, garbage out is not a cliché here; it is a methodological truth. If the underlying trials have design flaws, small sample sizes, inconsistent outcome measures, or publication bias baked in, pooling them doesn't fix any of that. It just averages the flaws.

This matters enormously for anyone reading research in phytomedicine, cannabinoids, or peptides , fields where many trials are small, heterogeneous, and not always peer-reviewed to the same standard as cardiovascular drug trials. Our Phytomedicine glossary entry covers some of that methodological complexity if you want the broader picture.

The forest plot: learning to read the picture

Most meta-analyses present their findings as a forest plot; a visualisation showing each included study as a horizontal line (representing the confidence interval) with a square on it (representing the effect size). Bigger squares mean larger sample sizes, carrying more weight in the pooled estimate. At the bottom sits a diamond shape: the pooled result.

A few things to check immediately when you see one.

First, where does that diamond sit relative to the line of no effect? If the confidence interval of the diamond crosses that line, the pooled result is not statistically significant, full stop, regardless of what the abstract says. Second, look at how spread out the individual study squares are. If they're scattered across both sides of the line of no effect, the studies are telling different stories. That's heterogeneity, and it matters, a lot.

Heterogeneity: the word that should make you pause

Heterogeneity is the degree to which included studies differ from each other. Differences in participant populations, intervention doses, outcome measurement tools, and follow-up periods all contribute. Researchers typically report it using a statistic called I², expressed as a percentage. An I² above roughly 75% is generally considered high, the studies are so different from each other that combining them may be misleading, even if mathematically possible.

I'll be honest: I got this wrong for years. I used to see "statistically significant pooled effect, p < 0.05" and think, job done. It took a very patient biostatistician colleague to walk me through why a highly heterogeneous meta-analysis with a significant result can still be almost uninterpretable. You're averaging apples and engine parts and calling it fruit salad.

This is particularly relevant in cannabinoid research, where studies vary wildly in the compound examined (is it a cannabidiol isolate, a full-spectrum extract, or a broad-spectrum formulation?), in route of administration, and in the population studied. A meta-analysis lumping all of those together and reporting a single pooled effect should be read with serious caution.

Publication bias and the file drawer problem

Here's a structural issue that has nothing to do with researcher incompetence: journals have historically been far more likely to publish trials that found something than trials that found nothing. This means the studies that make it into a meta-analysis skew positive. The null-result trials are sitting in someone's file drawer, or languishing in a preprint server, uncited.

A well-conducted meta-analysis tries to account for this using tools like funnel plots or Egger's test. Funnel plots display each study's effect size against its precision; in an unbiased sample, the plot should look like a symmetrical inverted funnel. Asymmetry suggests publication bias. Not every meta-analysis includes these checks. When one doesn't, ask why.

The Cochrane Collaboration, whose reviews are published at cochranelibrary.com, has some of the most rigorous protocols for addressing this, including mandatory pre-registration of search strategies. Cochrane reviews aren't perfect, but as a benchmark for how a systematic review should handle bias, they're the standard I'd point anyone to first.

GRADE: the framework that goes beyond the p-value

One development I genuinely think changed how evidence-based medicine reads systematic reviews is GRADE, the Grading of Recommendations Assessment, Development and Evaluation framework. Rather than just summarising effect sizes, GRADE explicitly rates the certainty of evidence across five domains: risk of bias, inconsistency (that's heterogeneity again), indirectness, imprecision, and publication bias.

A meta-analysis with a statistically significant result can still come out with a GRADE rating of "very low certainty", meaning the true effect may be substantially different from the estimate. This happens all the time in early-stage research fields. Seeing GRADE applied rigorously is, to me, a good sign that the authors are being intellectually honest rather than overselling their findings.

For context: when you're reading about a compound's documented pharmacological profile; say, research into how a standardised extract behaves differently from an isolate, GRADE helps you understand whether the supporting evidence is solid or still genuinely preliminary. The distinction matters if you're trying to interpret what a headline actually means.

What "statistically significant" does not mean

Statistical significance and clinical or practical significance are not the same thing. A pooled effect can be real, replicable, and statistically significant while being so small in absolute terms that it's essentially meaningless outside a research context. Effect sizes, Cohen's d, odds ratios, risk ratios; tell you the magnitude. The p-value only tells you the probability of seeing a result this large if the null hypothesis were true.

A meta-analysis can also report a statistically significant effect on a surrogate outcome (a biomarker, say) rather than a patient-relevant outcome. Whether changes in surrogate outcomes translate into anything meaningful is a separate question, and often one the meta-analysis cannot answer.

My cattle dog, Ted, has no patience for nuance and will bark at the forest plot regardless. I have slightly more patience, but the same instinct to interrogate rather than accept.

Reading the methods section, not just the abstract

The abstract of a meta-analysis is a marketing document. The methods section is the actual paper. Before you cite anything, check: what databases were searched, and with what search terms? What were the inclusion and exclusion criteria? Were two independent reviewers involved in study selection? Was there a pre-registered protocol (PROSPERO is the main registry for systematic reviews, worth Googling)?

Honestly, most people, including a lot of science journalists; never read past the abstract. And that's precisely how a meta-analysis with significant limitations becomes "science proves X" in a news cycle.

In my view, the uncritical deference to meta-analyses as the final word on anything is almost as dangerous as ignoring evidence entirely. The design is powerful. That doesn't make every example of the design trustworthy.

If you want to understand how these evidentiary questions play out in the regulatory context, how agencies like the TGA weigh pooled evidence when assessing submissions under the Special Access Scheme or the Authorised Prescriber pathway, the methodological standards above are exactly the kind of quality filters being applied. Evidence hierarchy matters to regulators, not just academics.

A quick checklist when you encounter a meta-analysis

Is the forest plot's diamond crossing the line of no effect? If so, there is no statistically significant pooled finding.
What is the I² value? High heterogeneity warrants real caution about the pooled estimate.
Is there a funnel plot or bias assessment? Absence is a flag.
Is GRADE applied? What certainty rating did the evidence receive?
Is there a pre-registered protocol (PROSPERO)?
What is the effect size, not just the p-value?
Are the included studies actually comparable in design, population, and intervention?

None of this is about dismissing meta-analyses. They're among the most useful tools we have. But reading them critically; actually reading them, not the press release, is a skill worth building. The research landscape in plant-based medicine and cannabinoids is genuinely exciting right now, and that's exactly the context in which an uncritical headline can do the most damage.

Sources

, Hannah Bui, Evidence & Research Literacy Writer

]]>

Common questions

Is a meta-analysis always stronger evidence than a single randomised controlled trial?: Not automatically. A meta-analysis sits at the top of the traditional evidence hierarchy in principle, but a poorly conducted meta-analysis — one that pools highly heterogeneous studies or fails to account for publication bias — can actually be less informative than a single well-designed, adequately powered RCT. The design confers potential, not guaranteed quality.
What is I² and why does it matter?: I² is a statistic that quantifies the proportion of variability in a meta-analysis that's due to genuine differences between studies rather than chance. An I² above roughly 75% is considered high heterogeneity, suggesting the included studies are measuring something sufficiently different that combining them into a single pooled estimate may be misleading, even if it's mathematically possible.
How do I know if a meta-analysis has accounted for publication bias?: Look for a funnel plot and a formal statistical test for asymmetry (such as Egger's test or Begg's test) in the methods or results section. Also check whether the authors registered a protocol on PROSPERO before conducting the review — pre-registration makes it harder to selectively report favourable findings after the fact.
What does GRADE mean when I see it in a systematic review?: GRADE (Grading of Recommendations Assessment, Development and Evaluation) is a framework for rating the overall certainty of a body of evidence. It grades evidence as high, moderate, low, or very low certainty based on factors including risk of bias, inconsistency between studies, indirectness of the evidence, imprecision of results, and publication bias. A statistically significant pooled result can still receive a 'very low certainty' GRADE rating.
Why do two news articles sometimes cite the same meta-analysis to support opposite conclusions?: Usually because both articles are working from the abstract rather than the full paper, and abstracts often emphasise the statistically significant headline finding without adequately conveying the effect size, heterogeneity, certainty rating, or limitations. A result can be real and statistically significant while being small in magnitude, applying only to a narrow subgroup, or resting on low-certainty evidence — all context that gets stripped away in a news cycle.