Nytel Testing 4. Many tests, false alarms ← PrevNext →
Module 4

Many tests, false alarms

None of 20 drugs works. Will one still look like it does?

Try an example:

The truth, then what the trials see

Twenty drugs, each with a true response rate of %, the same as placebo (a dummy pill). None works.

Each drug gets a trial: patients on drug, on placebo. Results differ by luck.

filled = improvedoutline = did not improve

The p-value asks how often a useless drug would look this good. A useless drug that passes is a false alarm.

1. How many tests?

2. Cutoff per test

A test whose p-value is under the cutoff gives a false alarm. (One-sided: only results that favor the drug count.)

3. Run

One trial tests a useless drug this many times. Every false alarm is luck.

One trial, test by test

false alarmno false alarmnever tested

Same trial, p-values against the cutoff

Predict, then check

The climb: more tests, more false alarms

Dotted line: your number of tests.

Fixed order

Most important test first. Go on only while each test is under the cutoff.

A real case: star signs and aspirin

In 1988, the ISIS-2 trial gave aspirin or placebo to 17,187 patients with a suspected heart attack. Aspirin clearly helped overall. To warn about slicing the data, its authors split patients by star sign. Aspirin seemed to fail for two signs, Gemini and Libra.

Twelve signs, twelve chances for luck. Below: a drug that really helps every sign equally, split by sign. Some bars still fall short.

Bars are simulated, not ISIS-2 data. The drug works equally in all 12 signs.

Another one: a site that looks bad

A drug helps 30% of patients at every site. One site enrolls 19 patients and only 1 improves. Look at card 5 above for how often luck alone does that when the drug is tested at many sites.

The page's own binomial formula gives the 1-in-100 number. We treat each site as separate luck.

Regulator's corner

The trial rulebook ICH E9 (section 5.6) lists where many tests come from: several main outcomes, several treatments, and repeated looks at the data before the trial ends.

Write the fix into the trial plan before any results come in. ICH E9, Statistical Principles for Clinical Trials (PDF)

Takeaway