Understanding Statistical Significance

Plenty of published studies contain a result that clears the usual bar for statistical significance and still tells us almost nothing useful. That's not a cynical take, it's a structural problem with how research gets filtered through headlines, press releases and, honestly, sometimes the discussion sections of the papers themselves.

Statistical significance is one of the most misunderstood concepts in science communication. In a field like plant-based medicine, where early-phase research is often cited as if it were settled evidence, that matters a lot. Understanding what a p-value actually means, and what it doesn't, is probably the single most useful skill for reading any study critically. This article is educational only; it describes how to read research, not what any substance does.

What a p-value is (and the part everyone skips)

The p-value is a probability. Specifically, it's the probability of observing your data, or something more extreme, if the null hypothesis were true. The null hypothesis is typically "there is no effect here." So a p-value of 0.04 means: if there were genuinely no effect, you'd still see data this extreme about 4% of the time just by chance.

That's it. That's what it says. It does not tell you the probability that the finding is real. It does not measure effect size. It does not confirm the hypothesis is correct.

The common mistake is to flip the sentence around. "There is a 4% chance this result is a fluke" sounds almost identical to the correct definition, yet it answers a different question. The p-value starts by assuming there is no effect and asks how surprising the data would be. It cannot run backwards and tell you how likely the effect is to be real, because that depends on things the p-value never sees, such as how plausible the idea was before anyone tested it and how many other things were tested alongside it. The general Wikipedia overview of the p-value sets out the formal definition and the usual misinterpretations if you want a neutral reference.

The conventional threshold, p < 0.05, was largely popularised by statistician Ronald Fisher in the 1920s. He proposed it as a rough rule of thumb for when results might warrant further investigation. It was never meant to be a binary pass/fail gate for deciding whether something is "true." And yet, for decades, that's exactly how journals used it.

Statistical significance versus clinical (or practical) significance

Here's where things get genuinely interesting, and where a lot of science journalism falls apart.

A result can be statistically significant and practically meaningless. With a large enough sample, almost any tiny difference will cross the p < 0.05 threshold. Studies with thousands of participants can report a "significant" difference between groups that amounts to a fraction of a unit on a scale whose known measurement error is larger than the difference itself. The maths checks out. The real-world relevance doesn't.

The reverse is also true. Small studies, common in early cannabinoid and phytomedicine research, may fail to reach significance not because the effect is absent, but because the study was underpowered to detect it. That's a sample size problem, not a verdict on the compound. If that idea is new, our piece on sample size and statistical power walks through why small studies are so easy to misread in both directions.

This is why effect size measures matter as much as p-values. Cohen's d, odds ratios, number needed to treat; these tell you how big the observed difference actually is. A well-reported study gives you both. When only the p-value appears in a summary, it's fair to ask what the size of the difference was and whether anyone would notice it outside a spreadsheet.

The replication crisis and what it means for this field

Between roughly 2011 and 2015, a wave of high-profile failures to replicate well-known psychology and biomedicine findings shook confidence in the p < 0.05 framework. The Open Science Collaboration's 2015 project, published in Science, attempted to replicate 100 psychology studies and found that only about 36% reproduced the original result with comparable effect sizes.

The drivers were familiar: small samples, flexible analysis decisions made after data collection (sometimes called "p-hacking"), publication bias toward positive results, and a culture that rewarded novelty over rigour. Publication bias deserves a second look. If journals favour positive findings, the studies that show nothing quietly stay in a drawer, and the published record ends up looking more convincing than the full body of work really is. None of this means science is broken. It means science is a process that self-corrects, slowly, imperfectly, but genuinely.

For anyone following cannabinoid or phytomedicine research, this context is worth sitting with. Much of the existing literature involves small samples, animal models, or in-vitro work. These studies generate hypotheses; they don't confirm treatments. The gap between those stages is explained in preclinical versus clinical research. And when you read about a compound, say, something affecting bioavailability or pharmacokinetics; the question to ask first is always: what was the study design, and has it been replicated?

Confidence intervals: often more useful than p-values

I genuinely prefer confidence intervals to p-values as a communication tool. A 95% confidence interval tells you the range of values within which the true effect likely falls, given the data. It carries more information than a binary significant/not-significant verdict, because it shows both the estimate and how much uncertainty surrounds it.

If a study reports a mean difference of 3.2 units with a 95% CI of [0.1, 6.3], that interval is wide, it's consistent with an effect that's trivial at the low end. That uncertainty should temper how confidently you interpret the finding. If the CI is [2.9, 3.5], the estimate is much more precise. Same p-value might appear in both. Very different stories.

Width is the thing to watch. A narrow interval usually reflects a larger or better-measured sample, while a wide one signals that the data are compatible with many different true effects. It's also worth checking whether the interval comes close to the "no difference" value. An interval that only just excludes it is telling you the same thing a borderline p-value would.

The American Statistical Association released a statement in 2016, updated in 2019, explicitly cautioning against using p < 0.05 as a dichotomous decision rule. They're not alone. The journal Basic and Applied Social Psychology banned p-values entirely in 2015 and required effect sizes and confidence intervals instead. Whether that's the right call is debated, but the underlying frustration is legitimate.

Pre-registration and what to look for

One of the more meaningful reforms in research practice over the last decade is trial pre-registration. Researchers commit their hypotheses, sample sizes, and analysis plan to a public registry, like ClinicalTrials.gov or the Australian New Zealand Clinical Trials Registry (ANZCTR); before data collection begins. This makes post-hoc rationalisation much harder and gives readers a way to check whether the outcomes reported match the outcomes planned.

A careful reader of a clinical study will check whether it was pre-registered and whether the primary outcomes in the paper match those filed in the registry. Discrepancies aren't automatically damning, sometimes protocols change for legitimate reasons, but unannounced outcome switching is a red flag worth noting. The usual pattern is that the planned main outcome came up empty, so a secondary measure gets promoted to the headline. Registries make that swap visible.

Studies examining compounds that interact with the endocannabinoid system; including ligands for the CB1 receptor and CB2 receptor, are increasingly being run under pre-registered protocols, particularly in later-phase clinical work. That's a good sign for the field's maturity.

Multiple comparisons: the hidden inflator

Run enough statistical tests on the same dataset and one of them will eventually cross p < 0.05 by chance. This is the multiple comparisons problem, and it's rife in exploratory studies.

If you test 20 independent hypotheses simultaneously at p < 0.05, on average one will appear significant even if none of the effects are real. Corrections exist, Bonferroni is the most conservative, Benjamini-Hochberg is more commonly used in genomics and multi-outcome trials; but they're not always applied, and they're rarely mentioned in headlines. In essence, these methods raise the bar each individual result has to clear, so that the overall chance of a false alarm stays under control.

Honestly, I'd argue this is the single most underreported methodological issue in health science journalism. When a study tests fifteen outcomes and one of them reaches significance, that needs to be front and centre in any write-up. It almost never is.

A practical checklist for reading a study

Rather than bullet points, a short prose list reads more naturally. Here's what I'd run through when opening a paper.

First, what was the study design? Randomised controlled trial, observational cohort, case series, animal model, in-vitro? Each sits at a different point on the evidence hierarchy and tells you how far you can extend the conclusions; observational studies versus RCTs covers the main differences. Second, what's the sample size, and was the study powered to detect the effect they're claiming? Third, is the primary outcome pre-registered? Fourth, what are the effect sizes and confidence intervals, not just the p-values? Fifth, who funded it, and is there a conflict of interest disclosure?

None of this requires a statistics degree. It requires slowing down and asking the unsexy questions before you get to the interesting bits. A well-written abstract can bury the sample size deep in the methods, so it pays to read past the summary. And when several studies on one question exist, a systematic review can help, as long as you know how to read one. Single stories, however vivid, sit lower still, as why anecdotes are not evidence explains.

Why this matters for plant-based medicine specifically

Research into compounds like cannabidiol and anandamide, or the proposed entourage effect, is genuinely fascinating and genuinely early. The honest position is that we have plausible mechanisms, some promising signals in controlled studies, and a long way to go before the evidence base is what you'd call settled. That's not a reason to dismiss the research. It's a reason to read it carefully.

Statistical significance is a starting point for a conversation, not the end of one. A p-value below 0.05 means "this is worth looking at more closely", not "this is proven." The sooner that distinction becomes common knowledge, the better the public conversation about emerging science will be.

Sources

• Open Science Collaboration (2015), Estimating the reproducibility of psychological science; NCBI/PubMed
• ASA Statement on Statistical Significance and P-Values (2016), American Statistical Association
• Wasserstein, Schirm & Lazar (2019), Moving to a World Beyond "p < 0.05"; NCBI/PubMed
• Australian New Zealand Clinical Trials Registry (ANZCTR), anzctr.org.au

Hannah Bui, Evidence & Research Literacy Writer