Reading a Systematic Review

The study everyone quotes , and why you should read the fine print

Forty-three included studies. Pooled data from six countries. A forest plot that stretches across two pages. And a conclusion so hedged with qualifiers that the abstract and the headline contradict each other. Sound familiar?

Systematic reviews get cited constantly in plant-based medicine, cannabinoid research and emerging peptide science. They're invoked as the final word, the settled science, the gold standard. But a systematic review is only as good as the studies inside it , and I've spent enough time in PubMed to know that's a wildly variable thing.

This is a practical guide to what a systematic review actually contains, what signals to look for, and where to be sceptical. No statistics degree required, though it doesn't hurt.

What a systematic review actually is

A systematic review answers a pre-specified research question by exhaustively searching for all relevant studies, applying explicit inclusion and exclusion criteria, and synthesising what they found. The "systematic" part is the key word. It's meant to remove the arbitrary "I'll cite the three papers I already know" approach that plagues narrative reviews and popular-press summaries alike.

When quantitative data from multiple studies is pooled statistically, that's called a meta-analysis. A systematic review may or may not include a meta-analysis; the two terms get conflated, but they're not the same thing. You can have a systematic review without pooling numbers. You cannot responsibly run a meta-analysis without having done a systematic review first.

The PRISMA statement , Preferred Reporting Items for Systematic Reviews and Meta-Analyses, maintained by a group of methodologists and accessible at prisma-statement.org; is the reporting checklist most journals require. If a review claims to be systematic but doesn't mention PRISMA compliance (or its predecessor QUOROM), that's worth noting.

The first thing I always check: PICO

Before I read anything else, I look at the PICO framework the authors declared: Population, Intervention, Comparison, Outcome. This tells me exactly what question the review was designed to answer. And it tells me, instantly, whether the conclusion I saw shared on social media actually answers that question.

You'd be surprised how often they don't match. A review scoped to healthy adults in controlled laboratory settings gets cited as evidence about clinical populations. A review using a particular extract formulation, say, a standardised extract with a defined concentration, gets cited as though it applies to any product with the same name on the label. These are not the same thing.

In cannabinoid research especially, bioavailability differences between routes of administration matter enormously when comparing studies. Oral administration subjects a compound to first-pass metabolism, which meaningfully changes the active concentration reaching systemic circulation. A review that pools inhaled and oral studies without stratifying them is not wrong to exist; but you should know it's doing that before you accept its pooled result at face value.

Risk of bias: this section is not optional reading

Every decent systematic review includes a risk-of-bias assessment. The Cochrane Risk of Bias tool (RoB 2 for randomised trials) and the ROBINS-I tool for non-randomised studies are the most common frameworks. They assess things like: was randomisation properly concealed? Were outcome assessors blinded? Were there incomplete outcome data?

Here's the part most popular summaries skip entirely: the overall quality of the evidence base. If the review rates 80% of included studies as high risk of bias, which is not unusual in early-phase research areas, then the pooled result inherits that uncertainty. A statistically significant pooled estimate from mostly high-risk-of-bias trials is not the same as a statistically significant result from well-designed, low-risk trials. The number is not the same kind of number, even if it's printed the same way.

The GRADE framework (Grading of Recommendations Assessment, Development and Evaluation) goes further. It rates the overall certainty of evidence as high, moderate, low or very low. Cochrane reviews routinely use it. When you see "low certainty evidence" or "very low certainty evidence" in a GRADE summary, that is a meaningful statement; not boilerplate modesty. It means the true effect could be substantially different from the estimate.

Heterogeneity: when pooling stops making sense

I'll be honest, I got this wrong for years. I used to see a meta-analysis forest plot and focus entirely on the pooled diamond at the bottom. The point estimate. The p-value. And I paid barely any attention to how spread out the individual study results were.

Heterogeneity is the statistical measure of how much variation exists between studies beyond what you'd expect from chance alone. It's reported as the I² statistic. An I² of 0% means the studies are telling basically the same story. An I² of 90% means they are telling dramatically different stories, and pooling them into one number is doing real epistemic violence to the data.

In my experience, and in the cannabinoid and phytomedicine literature I read most closely; high heterogeneity is common. Populations differ. Dosing regimens differ. Formulations differ enormously; a full-spectrum preparation and an isolate of the same primary compound are not interchangeable, and studies using them shouldn't be blithely merged. When I² is above 75%, I personally treat the pooled result as background context rather than a definitive signal. That's my line. Yours might sit somewhere else. But have a line.

Publication bias: the studies that never showed up

A systematic review can only include studies that were published or otherwise accessible. Trials with null or negative results are less likely to be published, a well-documented phenomenon in clinical research. This is publication bias, and it inflates the apparent effect size in almost every meta-analysis that hasn't actively accounted for it.

Funnel plots (and statistical tests like Egger's test) are the standard methods for detecting it. Not every review reports them; their absence doesn't prove bias exists, but it's another flag.

The Australian and New Zealand Clinical Trials Registry (ANZCTR), maintained at anzctr.org.au, is one of the registers where trials should be prospectively registered before they begin. Pre-registration reduces the risk that outcomes were changed or cherry-picked after the data came in. If you're reading a review and want to sanity-check whether a major included trial was pre-registered, it's worth a look.

What systematic reviews can't tell you

This is the section I feel most strongly about, so I'll be direct. A systematic review synthesises existing evidence. It cannot create evidence that doesn't exist. If the available literature consists of short-duration, small-sample studies in highly selected populations, which describes most of the early literature on cannabinoids, many peptide compounds with current regulatory scrutiny (the TGA has been active in enforcing supply rules around unapproved peptide products, which is relevant context for what research has been possible), and plant-based compounds where pharmacokinetics data in humans is still thin; then the review can tell you that the evidence base is limited. That's genuinely useful. But the next step, translating that limited evidence into guidance for individuals, is not something a systematic review does. That's a clinical judgement, and it belongs with a qualified practitioner.

The Special Access Scheme and Authorised Prescriber pathways that govern access to certain unapproved therapeutic goods in Australia exist precisely because the evidence base for many compounds is evolving. Systematic reviews inform regulatory thinking; they don't replace it.

Between sea swims last winter I went back through several Cochrane reviews on cannabinoid-related research. Honestly, the consistency between them on one point was striking: nearly all flagged the same methodological problems, small samples, short follow-ups, undeclared conflicts of interest in industry-funded trials. The research community knows what needs to be done. The question is whether the funding and the patience exist to do it.

My measured view, for what it's worth: I think we're in a period where enthusiasm for certain plant-based and cannabinoid compounds has substantially outrun the quality of the available systematic evidence. That's not a reason to dismiss the research. It is a reason to read it carefully and resist the pull of a confident-sounding headline.

A quick reading checklist

When I sit down with a new systematic review, I work through roughly this order:

• Is the PICO question clearly stated, and does the conclusion actually answer it?
• Does the review report PRISMA compliance or equivalent?
• What does the risk-of-bias assessment say? What proportion of studies are rated high risk?
• Is GRADE certainty reported? What level?
• What is the I² for the primary outcome? Is heterogeneity explained or just noted?
• Is there a funnel plot or publication bias assessment?
• Were included studies pre-registered?
• Who funded the included trials, and is that disclosed?

None of these questions invalidates a review on their own. But together, they tell you what kind of confidence you can actually hold in the pooled result, and that's the thing the headline never tells you.

Sources

• PRISMA Statement; PRISMA Group
• About Cochrane Reviews, Cochrane Library
• GRADE: an emerging consensus on rating quality of evidence, NCBI/PubMed Central
• Good Manufacturing Practice; Therapeutic Goods Administration

, Hannah Bui, Evidence & research-literacy writer