**POLSCI 4SS3**

Winter 2023

Schedule group meetings before break!

Need help with R? Check the Data Analysis Support Hub at Mac

You can book virtual research consultations with an expert

We discussed and explored techniques to reduce sensitivity bias

Some techniques are

**observational**`(e.g. randomized response)`

Some techniques are

**experimental**`(e.g. list experiment)`

**Today:**Discuss surveys using experiments more generally

Data strategy |
||
---|---|---|

Inquiry | Observational | Experimental |

Descriptive | Sample survey | List experiment |

Causal | Panel survey | Survey experiment |

Data strategy |
||
---|---|---|

Inquiry | Observational | Experimental |

Descriptive | Sample survey | List experiment |

Causal | Panel survey | Survey experiment |

Data strategy |
||
---|---|---|

Inquiry | Observational | Experimental |

Descriptive | Sample survey | List experiment |

Causal | Panel survey | Survey experiment |

Data strategy |
||
---|---|---|

Inquiry | Observational | Experimental |

Descriptive | Sample survey | List experiment |

Causal | Panel survey | Survey experiment |

Data strategy |
||
---|---|---|

Inquiry | Observational | Experimental |

Descriptive | Sample survey | List experiment |

Causal | Panel survey | Survey experiment |

Survey experiments are **experimental** data strategies that answer a **causal** inquiry

Assign respondents to

**conditions**Usually by

**random assignment**Each condition is a different version of a

**question**or**vignette****Goal:**Understand the effect of different conditions on the outcome question if interestHow does this work?

- Two ways to express functional relations

**Structural causal models**`(two weeks ago)`

**Potential outcomes framework**`(today)`

\(i\): unit of analysis

`(e.g. individuals, schools, countries)`

\(Z_i = \{0,1\}\) indicates a condition

`(1: Treatment, 0: Control)`

\(Y_i(Z_i)\) is the individual

**potential outcome**\(Y_i(0)\): Potential outcome under control

\(Y_i(1)\): Potential outcome under treatment

ID | Female | \(Y_i(1)\) | \(Y_i(0)\) |
---|---|---|---|

1 | 0 | 0 | 0 |

2 | 0 | 1 | 0 |

3 | 1 | 1 | 0 |

4 | 1 | 1 | 1 |

- \(\tau_i = Y_i(1) - Y_i(0)\) is the
**individual causal effect**

ID | Female | \(Y_i(1)\) | \(Y_i(0)\) | \(\tau_i\) |
---|---|---|---|---|

1 | 0 | 0 | 0 | 0 |

2 | 0 | 1 | 0 | 1 |

3 | 1 | 1 | 0 | 1 |

4 | 1 | 1 | 1 | 0 |

- \(\tau_i = Y_i(1) - Y_i(0)\) is the
**individual causal effect**

- \(\tau = (1/n) \sum_{i=1}^n \tau_i = E[\tau_i]\) is the
**inquiry**

- We call \(\tau\) the
**Average Treatment Effect (ATE)**

Letters like \(\mu\) denote

**estimands**A hat \(\hat{\mu}\) denotes

**estimators**

Letters like \(X\) denote

**actual variables**in our dataA bar \(\bar{X}\) denotes an

**estimate**calculated from our data

\(X \rightarrow \bar{X} \rightarrow \hat{\mu} \xrightarrow{\text{hopefully!}} \mu\)

\(\text{Data} \rightarrow \text{Estimate} \rightarrow \text{Estimator} \xrightarrow{\text{hopefully!}} \text{Estimand}\)

We want to know the ATE \(\tau\)

This requires us to know \(\tau_i = Y_i(1) - Y_i(0)\)

But when we assign treatment conditions we only observe one of the potential outcomes \(Y_i(1)\) or \(Y_i(0)\)

Meaning that \(\tau_i\) is impossible to calculate!

This is the

**fundamental problem of causal inference**

Unobserved |
||||
---|---|---|---|---|

ID | Female | \(Y_i(1)\) | \(Y_i(0)\) | \(\tau_i\) |

1 | 0 | 0 | 0 | 0 |

2 | 0 | 1 | 0 | 1 |

3 | 1 | 1 | 0 | 1 |

4 | 1 | 1 | 1 | 0 |

- We can randomly assign conditions \(Z_i\)

Unobserved |
Observed |
|||||
---|---|---|---|---|---|---|

ID | Female | \(Y_i(1)\) | \(Y_i(0)\) | \(\tau_i\) | \(Z_i\) | \(Y_i\) |

1 | 0 | 0 | 0 | 0 | 1 | 0 |

2 | 0 | 1 | 0 | 1 | 0 | 0 |

3 | 1 | 1 | 0 | 1 | 1 | 1 |

4 | 1 | 1 | 1 | 0 | 0 | 1 |

We observe outcome \(Y_i\) depending on assigned condition \(Z_i\)

We can use this to approximate the ATE with an

**estimator**

**Additive property of expectations:**

\[ \tau = E[\tau_i] = E[Y_i(1) - Y_i(0)] \\ = \underbrace{E[Y_i(1)] - E[Y_i(0)]}_{\text{Difference in means between potential outcomes}} \]

- We cannot calculate this, but we can calculate

\[ \hat{\tau} = \underbrace{E[Y_i(1) | Z_i = 1] - E[Y_i(0) | Z_i = 0]}_{\text{Difference in means between conditions}} \]

If we can claim that units are selected into conditions \(Z_i\) independently from potential outcomes

Then we can claim that \(\hat{\tau}\) is a good approximation of \(\tau\)

In which case we say that \(\hat{\tau}\) is an

**unbiased**estimator of the ATERandom assignment of units into conditions guarantees this

*in expectation*

- We cannot evaluate these assumptions with data but we can convince our audience with careful research design

Surveys in the UK (\(n = 762\)) and US (\(n = 1273\))

April-May 2010

**Outcome:**Support for military strike2x2x2 survey experiment

**Political regime:**Democracy/not a democracy**Military alliances:**Ally/not an ally**Military power:**As strong/half as strong

**Political regime:**Democracy/not a democracy**Military alliances:**Ally/not an ally**Trade:**High level/not high level

Factor | MP | Challenger |
---|---|---|

Party | Labour, Conservative | Labour, Conservative, Liberal Democrat |

Age | 45, 52, 64 | 40, 52, 64 |

Gender | Male, Female | Male, Female |

Previous job | General practitioner, journalist, political advisor, teacher, business manager | General practitioner, journalist, political advisor, teacher, business manager |

**Focus on:** Should findings generalize?