Evidence-Based Medicine 4

.title[
# Evidence-Based Medicine 4
]
.subtitle[
## Meta-studies
]
.author[
### Austin Meyer, MD, PhD, MS, MPH, MS, FAAP
]
.date[
### May 24, 2024
]

---

# Agenda

1. Gain an intuition about systematic reviews and meta-analyses
 
2. Understand their strengths and shortcomings
 
3. Review relevant measures and tests
 
4. Go over some questions

---

# Basic Statistical Things

---

# There are some important random terms

<html>
--

- **Generalizability**
- How applicable is a finding from a particular sample to the population
--
 
- **P-value**
- Probability of finding a value this extreme by random chance (given the assumption of a true null)
--
 
- **Confidence Interval**
- Interval over which population (true?) value is expected to be found with a specified probability (e.g. 95%)
--
 
- **Efficacy**
- Performance of treatment under ideal circumstances
--
 
- **Effectiveness**
- Performance of treatment under real world circumstances

---

# Hypothesis Testing

- `\(H_0\)`: There is **no** difference in the groups or no slope in regression **(these are the same thing, fyi)**

- `\(H_A\)`: There is difference because the probability of `\(H_0\)` being true randomly is low **(you define how low)**

---

# Was on my boards for both Peds and IM

.pull-right[.left[**Which of the following is the best method to assess the association between OCP use and blood clots?** A. Two sample T-test B. Analysis of variance C. Pearson correlation D. Chi-square test E. Spearman correlation ]]

<table style="margin-left:6%;">
<tr><th align="left"></th><th align="right">Clot</th><th align="right">No Clot</th><th align="right">Total</th></tr>
<tr><td align="left">OCP Use</td><td align="right">500</td><td align="right">400</td><td align="right">900</td></tr>
<tr><td align="left">No OCP Use</td><td align="right">80</td><td align="right">20</td><td align="right">100</td></tr>
<tr><td align="left">Total</td><td align="right">580</td><td align="right">420</td><td align="right">1000</td></tr>
</table>

---

# Was on my boards for both Peds and IM

---

# Why?

The data are counts.

---

# Data type goes with hypothesis test

- **Counts of 2 or more groups:** Chi-square, Fisher exact (only 2x2 contingency)
 
- **Quantitative variable + grouping variable(s):** T-test (data approximates a t-distribution), U-test (distribution not well defined), ANOVA
 
- **Quantitative variable + ranking variable:** Spearman correlation
 
- **Quantitative and 2 variables:** Pearson correlation

---

# Hypothesis testing: 4 possible outcomes

.pull-right[
- Correct - Reject a false `\(H_0\)`
  - Probability of success is called "power"
  - Power depends on sample size
  - bigger sample = bigger power

- Correct - Fail to reject a true `\(H_0\)`
  - Probability determined by `\(\alpha\)` as `\(1−\alpha\)`

- Type 1 - Incorrect rejection of a true `\(H_0\)`
  - False Positive

- Type 2 - Failure to reject a false `\(H_0\)`
  - False Negative]

---

# Relevant Question

.pull-right[.left[**An 18-month-old girl with a history of sickle cell disease is evaluated for 2 days of fever, cough, and vomiting. A rapid influenza diagnostic test is performed due to her symptoms and the high incidence of influenza A in the community. The test has 55% sensitivity and 95% specificity for influenza. Of the following, the MOST likely error type to occur when using this test is** A. type I B. type II C. type a D. type b E. type III ]]

---

# Hint

`\(sn = \frac{TP}{TP + FN}\)` 
`\(sp = \frac{TN}{TN + FP}\)`

---

# Relevant Question

.pull-right[.left[**An 18-month-old girl with a history of sickle cell disease is evaluated for 2 days of fever, cough, and vomiting. A rapid influenza diagnostic test is performed due to her symptoms and the high incidence of influenza A in the community. The test has 55% sensitivity and 95% specificity for influenza. Of the following, the MOST likely error type to occur when using this test is** A. type I **B. type II** C. type a D. type b E. type III ]]

---

# Foundation of 'Evidence-Based Medicine'

---

# The same image we always see repeated

---

**Two things to keep in mind:**

1. Medical school emphasizes the physiological-model-as-evidence for medicine which is the lowest part of the pyramid
 
2. `#`1 occurs because randomized trials and their derivatives (meta-analyses) are *extremely* inefficient means of knowledge generation

---

# Systematic Reviews

---

# What is a Systematic Review?

1. Rigorous and structured approach to reviewing available literature on a specific topic.

---

# Conducting a Systematic Review

<html>
--
1. Define the research question. 
--
2. Develop a protocol for identifying and including/excluding studies. 
--
3. Conduct a literature search. 
--
4. Screen and select studies. 
--
5. Extract and synthesize data. 
--
6. Assess the quality of the evidence.

---

# Meta-Analyses

---

# What is a Meta-Analysis?

- Statistical technique that combines results from multiple studies.
- Generally used within a systematic review.

---

# Pros of Systematic Review

1. **Reduces bias** - Follow a rigorous and predefined method, reducing the potential for bias in the identification, selection, and appraisal of studies.
 
2. **Completeness** - Synthesize many studies to one result - With meta-analysis can generate combined (pooled) estimates and errors
 
3. **Identify Gaps** - Can reveal areas where research is lacking or not of high quality, providing direction for future research

---

# Cons of Systematic Review

1. **Extraordinarily Resource Intensive** - Take an incredible amount of prior work to achieve useful conclusions
 
2. **Can Be Difficult to Interpret** - Publication bias and study heterogeneity can make it difficult to synthesize a result. 
 
3. **Lag Time** - Due to #1, it take a lot of time to get an answer and sometimes the field is settled before the review is written

---

# A thing to know

**Which of the following tools is commonly used to assess the risk of bias in studies included in a systematic review?**

A. Forest plot 
B. Funnel plot 
C. Cochrane Risk of Bias tool 
D. Kaplan-Meier curve 
E. I² statistic

---

# A thing to know

**Which of the following tools is commonly used to assess the risk of bias in studies included in a systematic review?**

A. Forest plot 
B. Funnel plot 
C. **Cochrane Risk of Bias tool **
D. Kaplan-Meier curve 
E. I² statistic

---

## 72 pages of pure fun

---

# All reviews are not systematic reviews

- A standard review is a narrative review and it is quite different

---

# Common Statistical Methods

---

# Key Statistical Methods and Evaluating Meta-analysis

1. Odds Ratio (OR)
2. Risk Ratio (RR)
3. Risk Difference (RD)
4. Number Needed to Treat (NNT)
5. Heterogeneity (I2)
6. Publication Bias (Funnel plots, Egger's test)

---

# Here is some data

---

# Here is the same data

<table>
<tr><th align="left"></th><th align="right">Disease</th><th align="right">No Disease</th><th align="right">Total</th></tr>
<tr><td align="left">Exposed</td><td align="right">30</td><td align="right">70</td><td align="right">100</td></tr>
<tr><td align="left">Not Exposed</td><td align="right">20</td><td align="right">80</td><td align="right">100</td></tr>
<tr><td align="left">Total</td><td align="right">50</td><td align="right">150</td><td align="right">200</td></tr>
</table>

---

# Odds Ratio (OR)

.pull-right[
.left[
`\(OR = \frac{D_E * ND_{NE}}{D_{NE} * ND_E} = \frac{30 * 80}{20 * 70} =\)` **1.71**
]
]

---

# Risk Ratio (RR)
**Only use on random samples**

.pull-right[
.left[
`\(R_E = \frac{D_E}{D_E + ND_E} = \frac{30}{30 + 70} = 0.3\)` 
`\(R_{NE} = \frac{D_{NE}}{D_{NE} + ND_{NE}} = \frac{20}{20 + 80} = 0.2\)` 
`\(RR = \frac{R_E}{R_{NE}} = \frac{0.3}{0.2} =\)` **1.5**
]
]

---

# Risk Difference (RD)

---

# Number Needed to Harm or Treat (NNH or NNT)

.pull-right[
.left[
`\(R_E = \frac{D_E}{D_E + ND_E} = \frac{30}{30 + 70} = 0.3\)` 
`\(R_{NE} = \frac{D_{NE}}{D_{NE} + ND_{NE}} = \frac{20}{20 + 80} = 0.2\)` 
`\(RD = R_E - R_{NE} = 0.3 - 0.2 = 0.1\)` 
`\(NNH = \frac{1}{0.1} =\)` **10**
]
]

---

# What if we use these 4 studies
## We'll make a meta-analysis

<table>
<tr><th align="left"></th><th align="right">Disease</th><th align="right">No Disease</th><th align="right">Total</th></tr>
<tr><td align="left">Exposed</td><td align="right">39</td><td align="right">61</td><td align="right">100</td></tr>
<tr><td align="left">Not Exposed</td><td align="right">20</td><td align="right">80</td><td align="right">100</td></tr>
<tr><td align="left">Total</td><td align="right">59</td><td align="right">141</td><td align="right">200</td></tr>
</table>
]

<table>
<tr><th align="left"></th><th align="right">Disease</th><th align="right">No Disease</th><th align="right">Total</th></tr>
<tr><td align="left">Exposed</td><td align="right">33</td><td align="right">67</td><td align="right">100</td></tr>
<tr><td align="left">Not Exposed</td><td align="right">27</td><td align="right">73</td><td align="right">100</td></tr>
<tr><td align="left">Total</td><td align="right">60</td><td align="right">140</td><td align="right">200</td></tr>
</table>

<table>
<tr><th align="left"></th><th align="right">Disease</th><th align="right">No Disease</th><th align="right">Total</th></tr>
<tr><td align="left">Exposed</td><td align="right">36</td><td align="right">64</td><td align="right">100</td></tr>
<tr><td align="left">Not Exposed</td><td align="right">19</td><td align="right">81</td><td align="right">100</td></tr>
<tr><td align="left">Total</td><td align="right">55</td><td align="right">145</td><td align="right">200</td></tr>
</table>
]

---

# Calculate odds ratios and standard error

`\(OR = \frac{A * D}{B * C}\)` 
`\(SE = sqrt(\frac{1}{A} + \frac{1}{B} + \frac{1}{C} + \frac{1}{D})\)`

`\(OR_1 = 1.71\)` 
`\(OR_2 = 1.33\)` 
`\(OR_3 = 2.56\)` 
`\(OR_4 = 2.4\)` 
]

`\(SE_1 = 0.33\)` 
`\(SE_2 = 0.31\)` 
`\(SE_3 = 0.32\)` 
`\(SE_4 = 0.33\)` 
]

`\(\begin{equation} \begin{split} CI & = e^{ln(OR) \pm t_{crit} * SE} \\ & = e^{ln(OR) \pm 1.96 * SE} \end{split} \end{equation}\)` 
`\(t_{crit} = 1.96\)` for 95% CI

---

# Question about heterogeneity

**In a meta-analysis, the term "heterogeneity" refers to:**

A. The similarity in study design among included studies. 
B. The difference in study outcomes across included studies. 
C. The uniformity in sample sizes of included studies. 
D. The consistency in publication dates of included studies. 
E. The methodological quality of included studies.

---

# Question about heterogeneity

**In a meta-analysis, the term "heterogeneity" refers to:**

A. The similarity in study design among included studies. 
**B. The difference in study outcomes across included studies. **
C. The uniformity in sample sizes of included studies. 
D. The consistency in publication dates of included studies. 
E. The methodological quality of included studies.

---

## According to a JAMA

Imrey, P. Limitations of Meta-analyses of Studies With High Heterogeneity. JAMA Network Open.

---

# Heterogeneity (I2) for our Meta-analysis

---

# Another slide on heterogeneity

> "The quantity, which we call I2, describes the percentage of total variation across studies that is due to heterogeneity rather than chance. I2 can be readily calculated from basic results obtained from a typical meta-analysis as `\(I2 = 100\%×(Q - df)/Q\)`, where Q is Cochran's heterogeneity statistic and df the degrees of freedom. Negative values of I2 are put equal to zero so that I2 lies between 0% and 100%."

- Generally people break heterogeneity like this:
 - Low Heterogeneity: <25%
 - Medium Heterogeneity: 50%
 - High Heterogeneity: >75%
- If heterogeneity is a significant effect you might look for further investigation
- They might perform a random effects regression to include stratification or subgroup analysis

---

# Forest Plot of our Meta-analysis

---

# Here is a Cochrane forest plot

---

# Question about funnel plots

**What is the primary purpose of a funnel plot in a meta-analysis?**

A. Display the individual results of each study included in the analysis. 
B. Assess the overall effect size of the included studies. 
C. Evaluate the presence of publication bias. 
D. Show the time-to-event data for the included studies. 
E. Summarize the methodological quality of the included studies.

---

# Question about funnel plots

**What is the primary purpose of a funnel plot in a meta-analysis?**

A. Display the individual results of each study included in the analysis. 
B. Assess the overall effect size of the included studies. 
**C. Evaluate the presence of publication bias. **
D. Show the time-to-event data for the included studies. 
E. Summarize the methodological quality of the included studies.

---

## now I make up different data to facilitate plotting

---

# Publication Bias - Unbiased Funnel Plot

**midline represents median effect** 
**horizontal lines represent 95% CI**

---

# Publication Bias - Unbiased Testing

---

# Publication Bias - Biased Funnel Plot

---

# Publication Bias - Biased Testing

---

# Caveats - there are some subtle points
 
---

# Caveats

- people are **not** particularly good at determining bias from visual inspection of a forest plot
- Egger's test is just an example
  - it has been extensively studied on binary outcomes
  - there are more sophisticated methods for continuous outcomes (random effects modeling)
- would be good to generally avoid systematic reviews that attempt to pool metrics from methodologically different studies
  - eg trying to combine odds ratios from case-control trials and cohort is not a great idea
  - it would be better to match like-studies
- you can only calculate risk ratios from random population samples (not case-controls)
- be sure to evaluate whether efforts to mitigate bias are sufficient
- subgroup analysis is very useful for hypothesis generation
  - **just be aware that if it is your primary endpoint, most people will expect multiple testing adjustment**

---

# Multiple testing can be a *real* problem

- every p-value is a hypothesis test

- given a true null, how many tests would you expect to reject the null randomly?
]
]

**this is subgroup testing and why it is better for hypothesis generation**

---

# The End