Module 11

What counts as proof

One trial wins. If you ran it again, would it win again?

Try an example:

1. First trial result

The two-sided p-value of trial 1:

A trial wins if the drug looks better with one-sided p below 0.025 (the same line as the familiar two-sided 0.05). A useless drug wins by luck 1 time in 40.

2. Run trial 2

What the picture shows

Each bar counts repeat trials of the same drug. Teal: the repeat wins.

The Black curve: the same answer from a formula.

This assumes the drug is really as good as trial 1 showed. Real drugs are often a bit worse, so real chances are lower.

Useless drugs that win by luck

Each dot is a useless drug. Orange dots got lucky and "won". Pick a rule.

Predict, then check

How the rule changed

Timeline. Sources: Clinical Trials in Practice, chapters 1, 9 (history, Phase 3) and the FDA guidance below.

Faster is not easier

FDA can speed up drugs for serious diseases. It does not lower the proof.

Source: Clinical Trials in Practice, chapter 29 (submission); FDA, Expedited Programs for Serious Conditions (2014).

Real cases: when "it helps" was wrong

Thalidomide (1950s to 1962). About 10,000 babies were born with birth defects, mostly outside the US, where FDA's Frances Kelsey had refused it.

Congress then required proof that drugs work (1962), with adequate and well-controlled trials.

CAST (late 1980s). Heart drugs fixed a rhythm marker on a heart monitor, but more patients died than on placebo. The trial stopped early.

Sources: Sokolova and Sokolov (2026), Clinical Trials in Practice, chapters 1 (history), 17 (endpoints) and 16 (randomization); Rossouw et al., Women's Health Initiative, JAMA 2002; CAST Investigators, N Engl J Med 1989.

Regulator's corner

"Substantial evidence" is the law's phrase for this page's question: adequate, well-controlled trials. FDA's revised draft guidance (June 2026; it revises the December 2019 draft and will replace the 1998 guidance) (section IV.A) describes one adequate, well-controlled trial plus confirmatory evidence. The evidence can come from a highly persuasive trial or from strong confirmatory evidence:

"a highly persuasive trial or a source of strong confirmatory evidence"

FDA draft guidance, Demonstrating Substantial Evidence of Effectiveness for Human Drug and Biological Products. Revised draft. FDA also weighs trial quality and what the result means for patients.

Takeaway

If the true effect equals the first estimate, an equally large repeat trial barely wins about half the time.
Two wins by luck are rare (1 in 1,600). One very strong win plus other evidence can also count.
A fast review is not a weaker standard. A marker that improves is not proof that patients do better.