What a p-value is (and isn't)
If the drug did nothing, how surprising would this result be?
This page compares one group of patients on the drug with a known placebo rate. (Module 0 compared two groups.)
1. The trial
2. Simulate trials of a drug that does nothing
Orange: trials with as many responders as yours, or more. Their share of all trials is the p-value.
The rule used from here on: a trial wins if the one-sided p-value (counting only results that favor the drug) is below 0.025. This is the same line as the familiar two-sided 0.05. Different question: the chance the true rate beats placebo chip starts from no opinion (flat prior, module 1) and uses the same data.
Two kinds of interval
Same data, 9 of 12 patients got better. Two ways to give a range for the true rate.
Credible interval (flat prior, module 1):
Given the data and a flat prior, the true rate is in here with 95% probability.
Confidence interval (exact Clopper-Pearson):
If we ran this trial many times, 95% of intervals built this way would catch the true rate.
Try it. The true rate is hidden from the method: 75%. Each run is 20 trials of 12 patients. The exact method catches the truth at least 95% of the time, so misses are rare.
Predict, then check
Does p = 0.02 mean a 98% chance the drug works?
Test 1,000 drugs. 100 of them (10%) truly work and the trial detects a working drug 80% of the time (power). The other 900 do nothing. A trial wins when the one-sided p is below 0.025.
Peeking at the data
The drug does nothing: half of patients get better, the same as on placebo. Compute p at each look and stop at the first one-sided p below 0.025 (significant). The height of the line shows how extreme the result is, in units of luck. One unit of luck is how far a trial's result typically lands from the truth by chance alone (statisticians call it a standard error). Exactly p = 0.025 sits at 1.96 units.
Fishing in many comparisons
The drug does nothing, but you test it in several groups of patients. Each dot is one test.
Orange dot = a false alarm (one-sided p below 0.025). Each number is a p-value.
A real example: EOLIA
EOLIA tested ECMO (a machine that does the lungs' work) against usual care in severe lung failure. The two-sided p-value for 60-day deaths was 0.09. The two-sided cutoff is 0.05, so the trial was called "not significant".
A later reanalysis combined the trial data with a prior (prior times likelihood gives the posterior, module 1). Depending on the prior, it put the chance that ECMO lowers deaths at 88% to 99%. "Not significant" did not mean "no effect".
Regulator's corner
Why do trials write the main outcome down before they see any data?
"To avoid multiplicity concerns arising from post hoc definitions, it is critical to specify in the protocol the precise definition of the primary variable ..."
Protocol: the trial's written plan, fixed before it starts. Primary variable: the main outcome. Post hoc: chosen after seeing the data. Multiplicity: many tests, as in Fishing above.
ICH E9, Statistical Principles for Clinical Trials, section 2.2.2. Read the guideline
Takeaway
- A p-value asks: if the drug did nothing, how often would luck give a result this good?
- It is not the chance that the drug works, and it depends on the plan you had before you looked.
- Peeking and fishing make luck look like a discovery.
- A credible interval speaks about this trial, given a prior. A confidence interval speaks about the method: over many trials, about 95% of the intervals catch the truth.