POLSCI 4SS3
Winter 2023

## Announcements

• Schedule group meetings before break!

• Need help with R? Check the Data Analysis Support Hub at Mac

• You can book virtual research consultations with an expert

## Last week

• We discussed and explored techniques to reduce sensitivity bias

• Some techniques are observational (e.g. randomized response)

• Some techniques are experimental (e.g. list experiment)

• Today: Discuss surveys using experiments more generally

# Survey experiments

## Types of survey research design

Data strategy
Inquiry Observational Experimental
Descriptive Sample survey List experiment
Causal Panel survey Survey experiment

## Types of survey research design

Data strategy
Inquiry Observational Experimental
Descriptive Sample survey List experiment
Causal Panel survey Survey experiment

## Types of survey research design

Data strategy
Inquiry Observational Experimental
Descriptive Sample survey List experiment
Causal Panel survey Survey experiment

## Types of survey research design

Data strategy
Inquiry Observational Experimental
Descriptive Sample survey List experiment
Causal Panel survey Survey experiment

## Types of survey research design

Data strategy
Inquiry Observational Experimental
Descriptive Sample survey List experiment
Causal Panel survey Survey experiment

Survey experiments are experimental data strategies that answer a causal inquiry

## Survey experiments

• Assign respondents to conditions

• Usually by random assignment

• Each condition is a different version of a question or vignette

• Goal: Understand the effect of different conditions on the outcome question if interest

• How does this work?

## Taking a step back

• Two ways to express functional relations
1. Structural causal models (two weeks ago)

2. Potential outcomes framework (today)

# Potential outcomes framework

## Notation

• $i$: unit of analysis (e.g. individuals, schools, countries)

• $Z_i = \{0,1\}$ indicates a condition (1: Treatment, 0: Control)

• $Y_i(Z_i)$ is the individual potential outcome

• $Y_i(0)$: Potential outcome under control

• $Y_i(1)$: Potential outcome under treatment

## Toy example

ID Female $Y_i(1)$ $Y_i(0)$
1 0 0 0
2 0 1 0
3 1 1 0
4 1 1 1
• $\tau_i = Y_i(1) - Y_i(0)$ is the individual causal effect

## Toy example

ID Female $Y_i(1)$ $Y_i(0)$ $\tau_i$
1 0 0 0 0
2 0 1 0 1
3 1 1 0 1
4 1 1 1 0
• $\tau_i = Y_i(1) - Y_i(0)$ is the individual causal effect
• $\tau = (1/n) \sum_{i=1}^n \tau_i = E[\tau_i]$ is the inquiry
• We call $\tau$ the Average Treatment Effect (ATE)

## A note on notation

### Greek

• Letters like $\mu$ denote estimands

• A hat $\hat{\mu}$ denotes estimators

### Latin

• Letters like $X$ denote actual variables in our data

• A bar $\bar{X}$ denotes an estimate calculated from our data

$X \rightarrow \bar{X} \rightarrow \hat{\mu} \xrightarrow{\text{hopefully!}} \mu$

$\text{Data} \rightarrow \text{Estimate} \rightarrow \text{Estimator} \xrightarrow{\text{hopefully!}} \text{Estimand}$

## Challenge

• We want to know the ATE $\tau$

• This requires us to know $\tau_i = Y_i(1) - Y_i(0)$

• But when we assign treatment conditions we only observe one of the potential outcomes $Y_i(1)$ or $Y_i(0)$

• Meaning that $\tau_i$ is impossible to calculate!

• This is the fundamental problem of causal inference

## Continuing the example

Unobserved
ID Female $Y_i(1)$ $Y_i(0)$ $\tau_i$
1 0 0 0 0
2 0 1 0 1
3 1 1 0 1
4 1 1 1 0
• We can randomly assign conditions $Z_i$

## Continuing the example

Unobserved
Observed
ID Female $Y_i(1)$ $Y_i(0)$ $\tau_i$ $Z_i$ $Y_i$
1 0 0 0 0 1 0
2 0 1 0 1 0 0
3 1 1 0 1 1 1
4 1 1 1 0 0 1
• We observe outcome $Y_i$ depending on assigned condition $Z_i$

• We can use this to approximate the ATE with an estimator

## Estimator for the ATE

$\tau = E[\tau_i] = E[Y_i(1) - Y_i(0)] \\ = \underbrace{E[Y_i(1)] - E[Y_i(0)]}_{\text{Difference in means between potential outcomes}}$

• We cannot calculate this, but we can calculate

$\hat{\tau} = \underbrace{E[Y_i(1) | Z_i = 1] - E[Y_i(0) | Z_i = 0]}_{\text{Difference in means between conditions}}$

## Randomization

• If we can claim that units are selected into conditions $Z_i$ independently from potential outcomes

• Then we can claim that $\hat{\tau}$ is a good approximation of $\tau$

• In which case we say that $\hat{\tau}$ is an unbiased estimator of the ATE

• Random assignment of units into conditions guarantees this in expectation

## Assumptions

1. Ignorability

Assignment to conditions does not depend on potential outcomes. This is guaranteed if randomization works properly.

2. Non-interference

Individual potential outcomes do not depend on the treatment assignment of others. If they do, then we need a more complicated model.

• We cannot evaluate these assumptions with data but we can convince our audience with careful research design

# Discussion

## Tomz and Weeks (2013): “Public Opinion and the Democratic Peace”

• Surveys in the UK ($n = 762$) and US ($n = 1273$)

• April-May 2010

• Outcome: Support for military strike

• 2x2x2 survey experiment

## Vignette design

### UK

• Political regime: Democracy/not a democracy

• Military alliances: Ally/not an ally

• Military power: As strong/half as strong

### US

• Political regime: Democracy/not a democracy

• Military alliances: Ally/not an ally

• Trade: High level/not high level

## Profile variants

Factor MP Challenger
Party Labour, Conservative Labour, Conservative, Liberal Democrat
Age 45, 52, 64 40, 52, 64
Gender Male, Female Male, Female

## Next Week

### Convenience Samples

Focus on: Should findings generalize?