Case: the ulcer surgery trial
Four operations for a gut ulcer (duodenal ulcer), 100 patients each. Which one would you pick?
1. The results
Edit any count. Excellent = healed well. Fair = some trouble.
The chi-square test gives one mismatch score: how far the table is from "all four the same".
2. Compare two
A vs B is the worst against the best, picked after seeing the table. Predict 6 shows how picking fools you. The 98% from "No guess" assumes each operation has nothing to do with the others, so it is fooled the same way. Shrinkage, which pulls extreme arms toward the average, gives a smaller number.
3. Starting guess (prior)
More in module 1.
What happened to the patients
The two you compare
Predict, then check
Pulling extreme results toward the middle (shrinkage)
The best and worst arms are partly luck. Shrinkage pulls each arm toward the overall average, and pulls harder when the arm has fewer patients. Same counts as above (illustrative); edit the table to see it move.
A hierarchical model does this pulling in a principled way. Module 9 shows the same idea as borrowing from history.
Where the numbers come from
Illustrative counts based on Novick and Grizzle (1965), "A Bayesian approach to the analysis of data from clinical trials".
The operations were not named.
Regulator's corner
Why decide the comparison before looking? Put the main analysis choices in the protocol before the trial. Finish the detailed plan before unblinding (ICH E9, section 5.1).
Takeaway
- A p-value just above 0.05 does not mean the treatments are equal.
- Each test answers one question. Ask which one, before you look.
- Shrinkage pulls extreme arms toward the average, most for small arms, so picking the best and worst fools you less.
- Pick best vs worst after the fact, and chance fools you about 1 time in 20 (5.2% here), over ten times more than with a pair chosen in advance (0.4%).