Trial simulator: test the rule before you trust it
A trial may stop early when the trial records look great. How often is that a false alarm?
Each trial has two groups (arms): drug and placebo. A look is a pause to check the trial records. A false positive is a trial that calls a useless drug a success.
1. The truth (hidden from the trial)
2. The trial plan
The arm that is more likely better gets more patients.
3. The stopping rule
This lab starts every arm from no opinion, a flat prior. At each look, find the chance the drug is better. Above the success bar: stop and say success.
Below it at an early look: stop and give up. Off = never give up.
Fixed and classic designs always use equal groups.
Fixed: one look, at the end. Classic rule: looks set so a useless drug wins about 2.5% under the bell-curve approximation. The simulated false-win rate in the table is the number to trust. Your bar: the rule set on the left.
When do the trials stop?
Share of trials stopping at each look.
Classic rule
Your bar
Truth versus what the trial sees
One simulated trial: the true rates, then the patients it enrolled before stopping.
Hidden truth
What the trial sees
filled = improved outline = did not. Only the first 80 per arm are shown.
The chance the drug is better after each patient, for this one simulated trial. Grey lines are the looks; the trial can stop only there. Dotted: where the chance would have gone if the trial had continued.
What the trial says when it stops
Each trial reports its estimated gap when it stops. Dashed line: the true gap.
Find a fair bar
Calibrate finds the lowest success bar at which a useless drug wins 2.5% of trials or fewer (the one-sided 0.025 line, counting only wins for the drug). It checks one placebo rate only. A real design checks the false-win rate at every plausible placebo rate.
It tests bars on one batch of simulated trials, then checks the best one on a new batch.
Predict, then check
A real trial that set its bar by simulation
The Pfizer-BioNTech COVID-19 vaccine trial (2020) used a Bayesian success rule: the chance that the vaccine cuts COVID cases by more than 30% had to pass 98.6% at the final analysis.
“to compensate for the interim analysis and to control the overall type 1 error rate at 2.5%”
(Its looks would otherwise raise false positives above 2.5%.)
The bar is above 97.5% for the same reason as above. The final count was 8 cases among vaccinated people and 162 on placebo (95.0% efficacy).
Source: Polack et al. (2020), doi:10.1056/NEJMoa2034577 (full text).
Regulator’s corner
FDA guidance (2019) and its Bayesian draft (2026) ask sponsors to show by simulation that a trial with looks keeps false positives in check. The draft international guideline ICH E20 (2025) agrees.
Go further: the studio
Need more designs: many arms, dose finding, borrowing from history, a full simulation report? Open the free trial simulator studio.
Takeaway
- More looks mean more chances for luck to cross a fixed bar, so the bar must rise.
- Simulation finds the bar: run the do-nothing drug thousands of times and count.
- Stopping early on good news overstates the drug's benefit.