Peek without penalty
Can you look at your trial after every patient and still not cheat?
1. The bet
Start with $1 and bet part of it on each pair of patients (one on drug, one on placebo). If only the drug patient improves you win the bet, if only the placebo patient improves you lose it.
2. Add patients
40 simulated trials. A trial that reaches $40 stops and declares the drug works. Your money at any moment is the e-value. Each step up the axis is 10 times more money.
Why a useless drug rarely wins
If the drug does nothing, each bet is fair, so the money does not grow on average. Ville (1939) proved it reaches $40 in at most 1 trial in 40 (2.5%), however often you look. So a useless drug rarely wins.
A Bayes factor (how much better one story fits the data than another) against a simple “no effect” null is itself an e-value.
Peeking: p-value versus wealth
One trial of a useless drug, checked after every pair.
Usual test: checked after every pair
E-value: wealth after every pair
Same patients in both panels.
How often does each method cry wolf?
3,000 trials of a useless drug, 200 pairs each.
The price: power
If you fix the check times in advance, a planned-checks test finds a working drug more often. The e-value trades some of that for checking any time.
Power = chance a working drug (45% vs 30%, 200 per arm) reaches the bar.
Predict, then check
Two real trials
RECOVERY, dexamethasone: in simulated trials with 2,000 patients per arm and RECOVERY's survival rates (77.1% vs 74.3%), only about a third reach $40. Small effects need big trials.
Ulcer surgery trial (illustrative counts based on Novick and Grizzle, 1965): 100 patients per arm, four surgery types A to D, deaths 7, 1, 1, 3. Same deaths, different patient order: A vs B and A vs C reach different highest e-values.
| Comparison | Deaths | Highest e-value |
|---|---|---|
| A vs B | 7 vs 1 | 15.3 |
| A vs C | 7 vs 1 | 45.8 |
| A vs D | 7 vs 3 | 7.2 |
| D vs B | 3 vs 1 | 1.7 |
| D vs C | 3 vs 1 | 5.2 |
The real patient order was not recorded, so the order here is simulated. An illustration, not a test. Bars show the highest e-value during the trial (log scale). Source: Sokolova and Sokolov, arXiv:2602.06379, Table 7. RECOVERY: RECOVERY Collaborative Group (2021).
Regulator's corner
ICH E20 is a draft international rulebook for trials that change their plan midway. It says:
"controlling the chances of false positive efficacy conclusions" is "an essential element of regulatory decision-making"
ICH E20 draft, section 3.3. E-values give this at every look, but regulators rarely use them yet.
Extension: is it random? (Kolmogorov's test)
Are the points spread evenly along a line? Count the points to the left of each spot: if they are even, that staircase hugs the straight line.
Click the line to place points (up to 15).
Random wiggles always come home
Each path is the staircase minus the line for one simulated random set of 15 points. It starts at 0 and ends at 0, with wiggles in between.
How big is a normal biggest gap?
Scores of 10,000 random sets. Orange: scores at least as big as the selected set's.
Kolmogorov's test catches points that bunch up in one region, not points that are too evenly spaced.
Takeaway
- Evidence is wealth from fair bets: if the drug does nothing, it does not grow on average, so it rarely reaches $40 (at most 1 time in 40).
- You may peek after every patient. Usual p-values cry wolf far more than 2.5% of the time.
- The price: with planned check times, a planned test finds a working drug more often.