The Hierarchy of Evidence

Roughly 85% of biomedical research has been estimated to be wasteful or misleading , a figure that comes from John Ioannidis's 2005 PLOS Medicine paper, one of the most-cited pieces in all of scientific publishing. I've been sitting with that number for years, and it still stops me mid-scroll whenever I see a headline that reads "Scientists discover X does Y."

Understanding where any given study sits in what researchers call the hierarchy of evidence won't make you a cynic. It'll make you a better reader. There's a difference.

What the hierarchy of evidence actually is

The hierarchy of evidence is a framework that ranks study designs by how well they can establish that one thing actually causes another, rather than merely appearing alongside it. It's been formalised in various ways , the Oxford Centre for Evidence-Based Medicine publishes one of the most widely used versions , but the core idea is consistent: some designs are more prone to bias and confounding than others, and we should weight our confidence accordingly.

At the top sits the systematic review with meta-analysis. Below that, randomised controlled trials (RCTs). Then cohort studies, case-control studies, cross-sectional surveys, and, at the base, expert opinion and anecdote. The pyramid metaphor is common, though honestly I prefer to think of it as a confidence gradient rather than a rigid pecking order; more on that shortly.

Starting at the bottom: why anecdote isn't worthless, just limited

Case reports and anecdotal accounts are where a lot of scientific curiosity begins. A clinician notices something unusual. A patient reports an unexpected response. This is genuinely useful signal, it's how we first noticed drug side effects that RCTs missed, and how several important biological mechanisms were first flagged for investigation.

But anecdote cannot establish causation. Too many variables are moving at once. The person reporting an outcome may have changed four things at the same time. Memory is reconstructive. Placebo responses are real and often large. None of that makes a person's experience wrong or unimportant, it makes it insufficient, on its own, to draw broad conclusions from.

This matters a lot in plant-based medicine research. When you read about compounds like cannabidiol or anandamide, you'll encounter layers of preclinical data, mechanistic hypotheses, and scattered case reports. They're worth knowing about. They're not worth treating as settled science.

Animal studies and in vitro work: mechanism, not outcome

Preclinical research; cell culture (in vitro) and animal models (in vivo), sits low on the hierarchy for human outcomes, but it's the engine room of early-stage science. This is where researchers probe mechanisms: how does a compound interact with a receptor, at what concentrations, via what pathway?

The endocannabinoid system, for instance, was largely mapped through animal research and receptor-binding studies before human trials became feasible. Understanding how CB1 receptors are distributed in the nervous system, or how CB2 receptors appear in peripheral immune tissue, required painstaking preclinical work across decades.

The failure rate between promising animal results and confirmed human outcomes is, however, enormous. Estimates vary, but somewhere around 90% of compounds that look good in animals don't translate to humans in trials. There are structural reasons for that, rodent metabolism differs from ours, disease models are approximations, dosing is rarely directly comparable. Preclinical results justify further research. They don't justify claims about what something does in a human body.

Observational studies: the confounding problem

Cohort studies follow a group of people over time and look for associations. Case-control studies compare people who have experienced an outcome with those who haven't, looking backward for exposures. Cross-sectional studies take a snapshot of a population at one moment in time.

These designs are much more feasible than RCTs; cheaper, longer-horizon, ethically simpler, and they've produced some of the most important findings in public health. But they cannot, on their own, prove causation. The reason is confounding: a third variable that's associated with both the exposure and the outcome, making the two look related when they may not be.

Correlation really isn't causation, and it sounds trite to say it, but I've watched smart people get tripped up on this more times than I can count. When an observational study finds that people who use a particular supplement have better some-outcome than people who don't, you have to ask: are those groups actually comparable in everything else? Diet? Exercise? Socioeconomic status? Health-seeking behaviour in general? Usually, they're not. That's selection bias and confounding walking in together.

Randomised controlled trials: the gold standard, with caveats

The RCT is the design that best controls for confounding, because randomisation distributes unknown variables roughly equally across groups. When it works, you can make a much stronger case for causation than observational designs allow.

But RCTs have real limitations that headlines almost never mention. They're often short, six weeks, twelve weeks, rarely more than a year. Sample sizes in early-phase trials are small, sometimes twenty or thirty people. Industry-funded trials show higher rates of favourable findings than independently funded ones, a pattern documented repeatedly in the literature. Blinding is hard to maintain when a substance has noticeable physiological effects. And trials are usually conducted in highly selected populations that don't reflect the full diversity of real-world users.

For something like bioavailability research; how much of a compound actually reaches systemic circulation and via which route, RCT designs can be particularly tricky, because first-pass metabolism varies substantially between individuals. A mean result in a small trial may tell you very little about what happens in a specific person.

The short version is: an RCT is much better than an observational study, but a single RCT is still just one data point. Replication matters enormously.

Systematic reviews and meta-analyses: powerful, but not infallible

A systematic review pools all the available evidence on a research question using a pre-specified, reproducible search strategy. A meta-analysis goes further and statistically combines results across studies to produce an overall estimate. Done well, these designs give the highest-confidence picture of what evidence currently shows.

Done poorly, they're garbage-in garbage-out. If the underlying RCTs are small, short, and heterogeneous in their methods, combining them doesn't magically produce a clean answer, it just combines the noise with extra steps. "Garbage in, garbage out" is a cliché for a reason. The Cochrane Collaboration maintains strict methodological standards for this reason, and it's worth looking at whether a review follows something comparable before trusting its conclusions.

I'd also flag publication bias: the tendency for studies with positive or statistically significant results to be published more readily than null results. This means a meta-analysis that only captures published literature is drawing from a biased pool. Some researchers pre-register trials on platforms like ClinicalTrials.gov precisely to reduce this problem, but it's not universal.

Reading the evidence hierarchy in practice

When you encounter a claim about a compound; whether that's a phytomedicine, a peptide, or a standardised extract, ask these questions in order:

What study design generated this finding? A cell culture study is not the same as a human trial. An animal study is not the same as a human trial. One human trial is not the same as ten replicated human trials.

Who funded it? Funding source doesn't automatically invalidate research, but it should inform how carefully you read the interpretation sections, where bias most often creeps in.

How large was the sample, and how long did it run? Thirty people over eight weeks tells you something modest. Three hundred people over two years tells you more.

Has it been replicated? One study, however well-designed, is a starting point. Science advances through replication and attempted falsification, not single demonstrations.

What did the outcome measure actually measure? A surrogate endpoint (a biomarker, a score on a scale) is not always the same as a clinically meaningful outcome. This distinction matters.

I'll be honest, working through this checklist for every claim you read online is genuinely exhausting. But even running through the first two questions will catch most of the worst offenders. Between sea swims last winter I started keeping a rough note of how many headlines I see each week that misrepresent study design, and the number is, let's say, humbling.

The hierarchy is a tool, not a verdict

Here's where I'll push back slightly on how the hierarchy is sometimes taught: it's a ranking of design quality for answering causal questions in humans. It's not a comprehensive ranking of all the ways knowledge can be useful. Mechanistic research that explains how something works can be highly valuable even if it sits low on the causal hierarchy. Qualitative research that captures lived experience doesn't fit neatly into the pyramid at all.

The hierarchy also can't rescue you from a badly formulated question. A perfect RCT answering the wrong question is still the wrong answer. Knowing the framework means knowing what it does and doesn't tell you; which is, I reckon, the whole point of being research-literate in the first place.

Sources

• Why Most Published Research Findings Are False, PLOS Medicine / NCBI
• Oxford Centre for Evidence-Based Medicine Levels of Evidence, CEBM, University of Oxford
• How Cochrane Reviews Work; Cochrane Collaboration
• Publication Bias and the Distortion of Evidence, NCBI / PMC

, Hannah Bui, Evidence & Research-Literacy Writer

]]>