Shark Tank Activity: Riverside General Hospital

Background

Riverside General is a 600-bed teaching hospital with a busy ED and med-surg units. Sepsis is a major quality challenge with ~12% in-hospital mortality and long delays to treatment. The current average time-to-first antibiotic is ~6 hours, with large variation across units.

The data science team has developed an AI sepsis early-warning alert that analyzes vitals, labs, and nursing flowsheets every 15 minutes in the EHR. A silent 8-week pilot (no alerts shown) on two wards showed a 0.5–1.0 hour faster antibiotic start versus historical data, but results were not randomized and may be confounded.

Clinicians are divided—some expect earlier care, others fear alert fatigue and overtreatment. The hospital can randomize at the patient level, record outcomes within 72 hours, and monitor safety continuously. The DSMB will meet at scheduled interim looks.

We can enroll up to 1,500 patients over 6 months (~250/month). The Board and IRB demand a rigorous, practical, and adaptive statistical plan: clear early-stop rules, Type I error control, and real-time safety protection.

Goals:

  1. Reduce time-to-antibiotics by ≥ 0.75 hours (clinically meaningful).

  2. Ensure mortality is not worse by > +2.5 percentage points.

  3. Stop immediately for credible signs of harm.

Baseline Data

Aspect Details
Primary outcome Time-to-first antibiotic (hours)
Baseline mean (SD) 6.0 (SD=3.0) hours
Target improvement ≥ 0.75 hours faster
Safety outcome In-hospital mortality (baseline 12%)
Safety margin ≤ +2.5 percentage-point increase
Timeline / Cap ≤ 6 months or ≤ 1,500 patients
Randomization Individual patients
Ethics Immediate stop for harm; allow early stop for benefit/futility
Error control Overall hospital risk = 0.05 ; patient risk = 0.10 (

Sample Design

Fixed Sample Size Calculation

The study has the following hypothesis:

\[ H_0: \mu_1 - \mu_2 \leq 0.75, vs. H_1: \mu_1 - \mu_2 \geq 0.75 \]

Where \(\mu_1\) and \(\mu_2\) are the average response time in the control and treatment. So the effect to detect \(\Delta = 0.75\) (the faster the better). Plug in \(\sigma = 3, \alpha = 0.05, \beta = 0.1\) , the fixed sample size for each arm is

\[ n = 2 \left(( \frac{z_{1-\alpha} + z_{1-\beta})\sigma}{\Delta}\right)^2 \approx 275 \]

So we need a total of 550 patients.

Similarly if the power drops to 80%, then we only need about 400 patients in total.

Group Sequential with Alpha Spending Function

Suppose in total there will be 3 interim looks at \(t_1 = 0.33, t_2 = 0.67\) and \(t_3 = 1\) , roughly after 180, 370, 550 total patients, (note: any fraction and rounding should be fine), and use Lan DeMets (it’s a continuous version of O’Brien Fleming ) as the alpha spending function:

\[ \alpha(t) = 1-\Phi(\frac{z_{1-0.05}}{\sqrt{t}}) \]

Then at each look, the cumulative \(\alpha\) would be 0.0021, 0.0222, 0.05. Then the actual \(\alpha\) for the test for each look would be

  • 0.0021

  • 0.0222-0.0021 = 0.02014

  • 0.05 - 0.0222 = 0.0278

For one-sided test, calculate the test statistics at each look

  • Look 1 at \(t=1/3\): \(z_{1-\alpha} = z_{1-0.0021} \approx 2.8637\)

  • Look 2 at \(t=2/3\): \(z_{1-\alpha} = z_{1-0.02014} \approx 2.0508\)

  • Look 3 at \(t=1\): \(z_{1-\alpha} = z_{1-0.0278} \approx 1.9141\)

Action:

Calculate the test statistics at each look: (sample mean and sample size for each arm)

\[ Z=\frac{\bar{X}_1 - \bar{X}_2}{\sqrt{ \frac{3^2}{n_1} + \frac{3^2}{n_2}}} \]

  • After recruiting 180 patients, stop if \(Z>2.8637\).

  • After recruiting 370 patients, stop if \(Z>2.0508\).

  • After recruiting 550 patients, stop if \(Z>1.9141\).

SPRT for Safety Monitoring

Safety monitoring does not spend efficacy α; it’s an emergency brake and a gate for rollout. Check the safety outcome at each patient(or every 3 patients, or daily batch).

There are different approaches at each check point, either one-sided confidence interval or one sided SPRT should work.

One-sided Confidence Interval

At each look, calculate the upper 95% confidence interval for mortality. For \(n\) patients, use the central limit theorem:

\[ \hat{p} = \bar{X} \]

where \(X\) is a binary outcome(Bernoulli distribution) labeling death or not. Then

\[ Var(\hat{p})=Var(\bar{X})=\frac{Var(X)}{n} \]

In the Bernoulli distribution, \(Var(X)=p(1-p)\), thus

\[ Var(\hat{p})\approx = \frac{\hat{p}(1-\hat{p})}{n} \]

So the standard deviation is \(SE=\sqrt{\frac{\hat{p}(1-\hat{p})}{n}}\) The upper 95% bound is

\[ \hat{p} + z_{0.95} \sqrt{\frac{\hat{p}(1-\hat{p})}{n}} \]

Suppose we observed a mortality rate of 13% with 50 patients, the upper bound of the confidence interval is

\[ 0.13 + 1.645 \times \sqrt{\frac{0.13 \times 0.87}{50}}\approx 0.208 \]

Since our upper limit for mortality rate is 0.145, this estimate does not meet the safety gate. We should stop the trial immediately.

Action

At each look, take down the mortality rate \(\hat{p}\) and the current number of patient \(n\), stop if \(\hat{p} + 1.645\times \sqrt{\frac{\hat{p}(1-\hat{p})}{n}}>0.145\) .

SPRT

Typically we have simple hypothesis in the sequential probability ratio test. In this class, let’s test on

\[ H_0: p = 0.12, vs. H_1: p = 0.145 \]

At each look, update the likelihood ratio statistics:

\[ \text{log}\Lambda_n = (\sum_{i=1}^nX_i) \text{log}\frac{0.145}{0.12} + (n-\sum_{i=1}^nX_i) \text{log}\frac{1-0.145}{1-0.12} \]

Where \(X_i=0,1\) are the indicator of a death event. Then calculate the upper limit \(B\) and lower limit \(A\)

\[ \text{log}(A) = \text{log}(\frac{\beta}{1-\alpha}), \text{log}(B) = \text{log}(\frac{1-\beta}{\alpha}) \]

Action

At each look, stop if \(\text{log}(\Lambda_n)\geq \text{log} (B)\) . For all other situation, keep sampling.