Module 15

Reading a Bayesian trial

A trial report says "probability of benefit 99%". What does that mean, and should you trust it?

A prior is what we believe before the trial. The posterior is what we believe after it (module 1). A 95% credible interval is a range that holds the true value with 95% chance, given the prior and the data.

Try an example:

Case 1: the Pfizer/BioNTech vaccine

Among the people who caught COVID in the trial, how many were in the vaccine group? Call the share of cases that were vaccinated \(\theta\) (theta).

If the vaccine does nothing, both groups catch it equally, so \(\theta = 1/2\). (Equal group sizes and follow-up time are assumed here.) Vaccine efficacy, VE, is how much the vaccine cuts the risk: \[\mathrm{VE} = \frac{1 - 2\theta}{1 - \theta}\]

1. The prior

2. The data

8 cases in the vaccine group, 162 in the placebo group (Polack et al. 2020).

3. The rule

Win if the chance that VE is above 30% is above 98.6%.

The bar is above 97.5% because the trial looked at its data early (module 7). The higher bar keeps false wins at 2.5%.

Published (Polack et al. 2020): VE 95.0%, 95% credible interval 90.3% to 97.6%, chance VE is above 30% more than 99.99%. The paper also adjusts for time under watch, so the numbers above differ slightly.

Case 2: two drugs for critically ill COVID patients

REMAP-CAP is a platform trial (module 10). It tested tocilizumab and sarilumab against usual care. The outcome was organ-support-free days: days alive without, for example, a breathing machine.

The analysis compares outcomes ordered from worst (death) to best (more days alive without organ support). An odds ratio above 1 means patients on the drug tended to land on better outcomes. 1 means no difference.

Tocilizumab: chance of benefit above 99.9% Sarilumab: chance of benefit 99.5%

Reported odds ratios with 95% credible intervals, Gordon et al. (REMAP-CAP) 2021. Not computed on this page.

Case 3: a borderline trial, EOLIA

EOLIA gave "not significant" in module 2 (p = 0.09). A later Bayesian reanalysis put the chance that ECMO lowers deaths at 88% to 99%, depending on the prior.

When the data are borderline, the prior decides. So who picks it? One way is to ask doctors.

Five doctors:

Ask five doctors

Each guess is how many fewer deaths per 100 patients the drug causes. Each doctor is unsure by about 5.

Made-up trial for this widget: the drug group had 11 fewer deaths per 100, give or take 6.5 from chance. Illustrative, not the EOLIA data.

The five guesses are blended into one prior (orange). The data move it to the posterior (teal).

Takeaway

A Bayesian result is three things: a probability of benefit, a credible interval, and a check with other priors.
All three vaccine priors clear the 30% bar, but the strong skeptical prior still lowers the estimated benefit. With borderline data (EOLIA), the prior can decide the verdict.