Pre-Registration and Publication Bias

Roughly half of all clinical trials that are completed never get published. That number has been kicking around the literature for years , it comes from a 2012 analysis by Jones and colleagues in PLOS Medicine , and it still makes me a bit furious every time I think about it.

It's not fraud, mostly. It's something subtler and, I'd argue, more insidious: the systematic disappearance of inconvenient results. Studies that find a positive effect get written up, submitted, and published. Studies that find nothing, or find the opposite of what the researchers hoped for, get quietly filed away. The scientific record that the public sees , and that other researchers build on; ends up looking a lot more convincing than it actually is.

That's publication bias. Pre-registration is one of the main tools researchers have developed to push back against it. Understanding both will change how you read any study, in any field.

What publication bias actually is

Picture a filing cabinet, the "file drawer", as the statistician Robert Rosenthal named it back in 1979. Every null result, every equivocal finding, every study that returned a shrug, goes into that drawer. The journals only receive the studies that came out looking clean and positive.

This creates a compounding problem. Systematic reviews and meta-analyses, which are supposed to give us the clearest view of a body of evidence; pool the published literature. But if the published literature already skews positive, the meta-analysis will too, regardless of how carefully it's constructed. You end up with scientific consensus built partly on selection.

The incentives driving this are structural, not personal. Academic careers depend on publications. Journals have historically preferred positive results. Funding bodies want to see returns. No single researcher needs to behave badly for the whole system to bend in the same direction.

In fields where the evidence base is still developing, cannabinoid research, phytomedicine, novel peptide compounds, this matters enormously. When the total number of trials is small, a handful of unpublished null results can make a modest or absent effect look like a strong signal. Readers who don't know about the file drawer assume the published studies are a representative sample of the evidence. They almost never are.

How pre-registration works

Pre-registration is straightforward in principle. Before collecting any data, a research team records their hypotheses, methods, sample size calculations, and primary outcomes in a public registry. The two most widely used are ClinicalTrials.gov (run by the US National Institutes of Health) and the Australian New Zealand Clinical Trials Registry (ANZCTR), which covers locally conducted work.

The timestamp matters. Once the pre-registration is public, any deviation between what the researchers said they would do and what they actually report becomes visible. The registry entry becomes a baseline; a version of the study that existed before anyone knew what the results would look like.

This addresses publication bias in a couple of ways. First, if trials are registered before they start, reviewers can search for completed-but-unpublished trials and contact the authors (or flag the gap). Second, it makes a specific class of misbehaviour much harder: outcome switching, which is when researchers move a secondary outcome to primary position after seeing that the primary didn't pan out.

Outcome switching is more common than you'd hope. A 2015 study published in PLOS ONE by Mathieu and colleagues tracked a sample of trials and found that a substantial proportion reported primary outcomes differently from their registered protocol. Sometimes this is innocent, methodological adjustments happen for legitimate reasons, and should be transparently disclosed. But without the pre-registered record, readers have no way to tell.

HARKing: the problem that pre-registration directly targets

HARKing stands for Hypothesising After Results are Known. It describes the practice, again, not always conscious; of looking at your data, identifying the most interesting pattern, and then writing the paper as though you'd hypothesised that finding from the start.

The statistical consequence is severe. When you test one pre-specified hypothesis, a p-value of 0.05 means roughly a 1-in-20 chance of a false positive, as advertised. When you've actually tested twenty different combinations of your data and reported only the most significant, you've inflated your false positive rate dramatically, but the paper reads as though you ran one clean test. This is sometimes called p-hacking, and it doesn't require any bad intent. Researchers can genuinely forget which analyses felt exploratory at the time.

Pre-registration forces separation between confirmatory and exploratory analyses. A well-pre-registered study will note, clearly, which findings were specified in advance and which emerged post-hoc. Both can be interesting. But they're not equally strong evidence, and treating them the same way is how confident-sounding claims end up failing to replicate.

What pre-registration doesn't fix

Honestly, this is the bit I find most useful to think about, because pre-registration is sometimes discussed as though it solves the problem rather than partially addresses it.

A pre-registered study can still be methodologically poor. The sample can still be too small. The comparator can still be badly chosen. Participant blinding can fail. The outcome measures can be crude or poorly validated. Pre-registration tells you what the researchers intended; it says nothing about whether the study was well-designed or well-executed.

It also doesn't prevent industries from selectively funding only those trials most likely to produce favourable results in the first place. If ten companies independently run small trials of the same compound, register all of them, and seven get null results that sit unpublished on ClinicalTrials.gov without anyone looking, the three that worked still dominate the public conversation. The problem shifts; it doesn't disappear.

And pre-registration is still far from universal. Observational studies, animal studies, and laboratory research are rarely pre-registered. A lot of the early-stage research on compounds like those in the endocannabinoid system, work exploring how cannabidiol interacts with specific receptor pathways, or investigating the bioavailability of various formulations; doesn't go through clinical trial registries at all. That's not inherently a problem, but it means those findings carry a different standard of evidentiary weight than a well-powered, pre-registered RCT.

How to actually check for pre-registration when you're reading a study

This is the practical bit. I went back through this last month when I was looking at some trial data for a piece on pharmacokinetics, and the check takes about ninety seconds once you know where to look.

First, check the methods section for a registration number. It should look like NCT followed by eight digits (ClinicalTrials.gov), or ACTRN followed by fourteen digits (ANZCTR). If there's no registration number, the study wasn't pre-registered, or wasn't registered in a publicly searchable way, which amounts to the same thing from a reader's perspective.

Second, if there is a registration number, look it up. Search the registry directly and pull up the original entry. Compare the listed primary outcomes with the ones reported in the paper. If they've changed, the paper should explain why. If it doesn't, that's a red flag.

Third, check the registration date against the trial start date. A trial registered after data collection has begun is sometimes called "retrospective registration", it captures some information but loses the pre-specification benefit entirely, because the researchers already know what direction the data went before they wrote down their hypotheses.

Fourth, look for the full results on the registry itself. Some registries now require results to be posted, and checking whether a registered trial reported results (or didn't) can reveal whether the registry-to-publication pipeline is intact. Trials that registered but never published and never posted results are part of the file drawer problem even when they're technically findable.

Why this matters for the fields we cover here

Plant-based medicine, cannabinoids, and novel compounds like GLP-1 receptor agonists all sit in fields where the evidence base is comparatively young and the commercial interest is growing fast. That's precisely the combination that makes publication bias most dangerous. Positive early results get amplified. Null replications don't land in top journals. Patients and prescribers base decisions on a literature that may be skewed in ways nobody intended but everyone contributed to.

This isn't a reason to dismiss emerging research. Some of the most genuinely interesting mechanistic work; on how compounds like anandamide interact with endogenous receptor systems, or how first-pass metabolism shapes the activity of orally administered cannabinoids, comes from exactly these early-stage, often unregistered studies. The honest position is: interesting signal, watch for replication, track the registry.

Between sea swims, I spend probably more time than is healthy trawling ClinicalTrials.gov for registered-but-never-published trials in areas I'm following. It's a slightly grim hobby. But the discrepancy between what's registered and what's published is the most honest measure of how much we don't know that we don't know.

The short version is: pre-registration raises the cost of moving goalposts after the whistle. It's not a guarantee of good science. But a study without it is carrying an extra layer of uncertainty that most research write-ups simply don't mention.

Sources

• Jones CW et al., "Declining rates of clinical trial reporting", PLOS Medicine (2012)
• Mathieu S et al., "Comparison of Registered and Published Primary Outcomes in Randomized Controlled Trials"; PLOS ONE (2015)
• Australian New Zealand Clinical Trials Registry (ANZCTR), anzctr.org.au
• Rosenthal R, "The file drawer problem and tolerance for null results", Psychological Bulletin, via NIH/NLM

, Hannah Bui, Evidence & Research Literacy Writer

]]>