Decisions: the price of avoiding false wins
Every rule for saying "yes" trades false wins against missed drugs. What does safety cost in patients?
A. Two ways to be wrong
A regulator can approve a drug that does nothing or reject a good one. In one trial, the first is a false win. Approval itself needs substantial evidence (module 11). Which mistake is worse?
A false win is ... times worse
Approve only if we are at least 39/(39+1) = 97.5% sure the drug works.
We start with no opinion about the drug. Costs are made up.
Start with no opinion and look once: “approve if 97.5% sure” is the same rule as the one-sided 0.025 test in the simple bell-curve model used here. For yes/no outcomes it is only approximate, which is one more reason to check by simulation. It is also the same as treating a false win as 39 times worse than a missed drug. Interim looks, adaptation and informative priors break this match, which is why regulators ask for simulations.
B. The price of safety: loss of power
A stricter bar for "yes" protects against false alarms. It also makes real drugs harder to find.
Example
Patients
Alpha (one-sided)
Endpoints tested (Bonferroni)
Looks at the data (module 7)
C. What FDA asks for
FDA’s 2026 draft gives three ways to decide a trial succeeded. Read it at fda.gov.
Cap false wins
Default: at most a 2.5% chance of any false win across all endpoints (module 4).
Judge what we believe after the trial
Start from what is already known, then ask how sure we are now.
Win if we are sure enough (say 97.5%) the drug beats placebo by a set amount.
Weigh benefit against risk
Weigh what each mistake costs, as in section A.
Benefit-risk: do the gains outweigh the harms?
D. Checking a rule with simulated trials
10,000 simulated trials per case. Old drug: 30% improve. Up to 100 patients per arm.
Chance the rule picks the new drug
Average patients per arm
Rules are set up differently, so this is not a ranking. Costs are made up.
E. Predict, then check
Takeaway
- Every rule for saying yes trades false wins against missed drugs and trial size.
- FDA's default bar: at most a 2.5% false win chance, one-sided.
- Simulated trials show what a rule really does: here 0.026 to 0.038 (the source aimed for 0.05). FDA's Bayesian draft allows other routes too.