Survey Experiments

 

POLSCI 4SS3
Winter 2023

Announcements

  • Schedule group meetings before break!

  • Need help with R? Check the Data Analysis Support Hub at Mac

  • You can book virtual research consultations with an expert

Last week

  • We discussed and explored techniques to reduce sensitivity bias

  • Some techniques are observational (e.g. randomized response)

  • Some techniques are experimental (e.g. list experiment)

  • Today: Discuss surveys using experiments more generally

Survey experiments

Types of survey research design

Data strategy
Inquiry Observational Experimental
Descriptive Sample survey List experiment
Causal Panel survey Survey experiment

Types of survey research design

Data strategy
Inquiry Observational Experimental
Descriptive Sample survey List experiment
Causal Panel survey Survey experiment

Types of survey research design

Data strategy
Inquiry Observational Experimental
Descriptive Sample survey List experiment
Causal Panel survey Survey experiment

Types of survey research design

Data strategy
Inquiry Observational Experimental
Descriptive Sample survey List experiment
Causal Panel survey Survey experiment

Types of survey research design

Data strategy
Inquiry Observational Experimental
Descriptive Sample survey List experiment
Causal Panel survey Survey experiment

 

Survey experiments are experimental data strategies that answer a causal inquiry

Survey experiments

  • Assign respondents to conditions

  • Usually by random assignment

  • Each condition is a different version of a question or vignette

  • Goal: Understand the effect of different conditions on the outcome question if interest

  • How does this work?

Taking a step back

  • Two ways to express functional relations

  1. Structural causal models (two weeks ago)

  2. Potential outcomes framework (today)

Potential outcomes framework

Notation

  • \(i\): unit of analysis (e.g. individuals, schools, countries)

  • \(Z_i = \{0,1\}\) indicates a condition (1: Treatment, 0: Control)

  • \(Y_i(Z_i)\) is the individual potential outcome

  • \(Y_i(0)\): Potential outcome under control

  • \(Y_i(1)\): Potential outcome under treatment

Toy example

ID Female \(Y_i(1)\) \(Y_i(0)\)
1 0 0 0
2 0 1 0
3 1 1 0
4 1 1 1
  • \(\tau_i = Y_i(1) - Y_i(0)\) is the individual causal effect

Toy example

ID Female \(Y_i(1)\) \(Y_i(0)\) \(\tau_i\)
1 0 0 0 0
2 0 1 0 1
3 1 1 0 1
4 1 1 1 0
  • \(\tau_i = Y_i(1) - Y_i(0)\) is the individual causal effect
  • \(\tau = (1/n) \sum_{i=1}^n \tau_i = E[\tau_i]\) is the inquiry
  • We call \(\tau\) the Average Treatment Effect (ATE)

A note on notation

Greek

  • Letters like \(\mu\) denote estimands

  • A hat \(\hat{\mu}\) denotes estimators

Latin

  • Letters like \(X\) denote actual variables in our data

  • A bar \(\bar{X}\) denotes an estimate calculated from our data

\(X \rightarrow \bar{X} \rightarrow \hat{\mu} \xrightarrow{\text{hopefully!}} \mu\)

 

\(\text{Data} \rightarrow \text{Estimate} \rightarrow \text{Estimator} \xrightarrow{\text{hopefully!}} \text{Estimand}\)

Challenge

  • We want to know the ATE \(\tau\)

  • This requires us to know \(\tau_i = Y_i(1) - Y_i(0)\)

  • But when we assign treatment conditions we only observe one of the potential outcomes \(Y_i(1)\) or \(Y_i(0)\)

  • Meaning that \(\tau_i\) is impossible to calculate!

  • This is the fundamental problem of causal inference

Continuing the example

Unobserved
ID Female \(Y_i(1)\) \(Y_i(0)\) \(\tau_i\)
1 0 0 0 0
2 0 1 0 1
3 1 1 0 1
4 1 1 1 0
  • We can randomly assign conditions \(Z_i\)

Continuing the example

Unobserved
Observed
ID Female \(Y_i(1)\) \(Y_i(0)\) \(\tau_i\) \(Z_i\) \(Y_i\)
1 0 0 0 0 1 0
2 0 1 0 1 0 0
3 1 1 0 1 1 1
4 1 1 1 0 0 1

  • We observe outcome \(Y_i\) depending on assigned condition \(Z_i\)

  • We can use this to approximate the ATE with an estimator

Estimator for the ATE

  • Additive property of expectations:

\[ \tau = E[\tau_i] = E[Y_i(1) - Y_i(0)] \\ = \underbrace{E[Y_i(1)] - E[Y_i(0)]}_{\text{Difference in means between potential outcomes}} \]

  • We cannot calculate this, but we can calculate

\[ \hat{\tau} = \underbrace{E[Y_i(1) | Z_i = 1] - E[Y_i(0) | Z_i = 0]}_{\text{Difference in means between conditions}} \]

Randomization

  • If we can claim that units are selected into conditions \(Z_i\) independently from potential outcomes

  • Then we can claim that \(\hat{\tau}\) is a good approximation of \(\tau\)

  • In which case we say that \(\hat{\tau}\) is an unbiased estimator of the ATE

  • Random assignment of units into conditions guarantees this in expectation

Assumptions

1. Ignorability

Assignment to conditions does not depend on potential outcomes. This is guaranteed if randomization works properly.

2. Non-interference

Individual potential outcomes do not depend on the treatment assignment of others. If they do, then we need a more complicated model.

  • We cannot evaluate these assumptions with data but we can convince our audience with careful research design

Discussion

Tomz and Weeks (2013): “Public Opinion and the Democratic Peace”

  • Surveys in the UK (\(n = 762\)) and US (\(n = 1273\))

  • April-May 2010

  • Outcome: Support for military strike

  • 2x2x2 survey experiment

Vignette design

UK

  • Political regime: Democracy/not a democracy

  • Military alliances: Ally/not an ally

  • Military power: As strong/half as strong

US

  • Political regime: Democracy/not a democracy

  • Military alliances: Ally/not an ally

  • Trade: High level/not high level

Results for democracy

Results for other factors

Eggers et al (2017): “Corruption, Accountability, and Gender”

Profile variants

Factor MP Challenger
Party Labour, Conservative Labour, Conservative, Liberal Democrat
Age 45, 52, 64 40, 52, 64
Gender Male, Female Male, Female
Previous job General practitioner, journalist, political advisor, teacher, business manager General practitioner, journalist, political advisor, teacher, business manager

Results

Next Week

Convenience Samples

Focus on: Should findings generalize?

Break time!

 

Lab