POLSCI 4SS3
Winter 2023

## Announcements

• No class on March 29

• Optional lab that week

• Will use flex session on April 12 to catch up

• No office hours between March 23-29

## Last time

• We learned about implementing field experients

• Lots of details!

• Sometimes cannot simply randomly assign (stepped-wedge design)

• Today: Thinking about how to do better

## Why do better?

• Conducting research is expensive

• Field experiments are very expensive

• Even if you had the resources, we have a mandate to do better

## Research ethics

• Belmont report: Benefits should outweigh costs

• : Researchers have duties beyond getting review board approval

• At a minimum, participating in a study takes time

• Mandate: Find the most efficient, ethical study before collecting data

• Sometimes that means doing more with a smaller sample

# Improving Precision

## Pre-post design

• Similar to panel studies

• Outcomes are measured at least twice

• Once before treatment, once after treatment

Condition $t=1$ Treatment $t=2$
$Z_i=1$ $Y_{i, t=1}$ X $Y_{i, t=2}(1)$
$Z_i=0$ $Y_{i, t=1}$ $Y_{i, t=2}(0)$

## How does this work?

• Standard ATE estimator:

$E[Y_i(1) | Z_i = 1] - E[Y_i(0) | Z_i = 0]$

• Pre-post ATE estimator:

$E[(Y_{i,t=2}(1) - Y_{i,t=1}) | Z_i = 1] - E[(Y_{i,t=2}(0) - Y_{i,t=1}) | Z_i = 0]$

## How does this work?

• Standard ATE estimator:

$E[Y_i(1) | Z_i = 1] - E[Y_i(0) | Z_i = 0]$

• Pre-post ATE estimator:

$E[(Y_{i,t=2}(1) \color{#ac1455} {- Y_{i,t=1}}) | Z_i = 1] - E[(Y_{i,t=2}(0) \color{#ac1455} {- Y_{i,t=1}}) | Z_i = 0]$

• We improve precision by subtracting the variation in the outcome that is unrelated to the treatment

## Block randomization

• Change how randomization happens

• Group units in blocks or strata

• Estimate average treatment effect within each

• Aggregate with a weighted average

## How does it work?

• Within-block ATE estimator:

$\widehat{ATE}_b = E[Y_{ib}(1) | Z_{ib} = 1] - E[Y_{ib}(0) | Z_{ib} = 0]$

• Overall ATE estimator:

$\widehat{ATE}_{\text{Block}} = \sum_{b=1}^B \frac{n_b}{N} \widehat{ATE}_b$

## Illustration

ID Block $Y_i(0)$ $Y_i(1)$
1 1 1 4
2 1 2 5
3 1 1 4
4 1 2 5
5 2 3 8
6 2 4 9
7 2 3 8
8 2 4 9
• Potential outcomes correlate with blocks

• True $ATE = 4$

• Do 500 experiments

• Compare complete and block-randomized experiment

## Reasons to block randomize

1. To increase precision in ATE estimates

2. To account for possible heterogeneous treatment effects

• The more blocking variables correlate with potential outcomes, the more useful block randomization is

• And it rarely hurts when they do not correlate! (more in the lab!)

# Example

## Kalla et al (2018): Are You My Mentor?

• Correspondence experiment with $N = 8189$ legislators in the US

• Cue gender with student’s name

## Data strategy

• Block-randomize by legislator’s gender (why?)

• Outcomes: Reply content and length

## Findings

Outcome Male Sender Female Sender p-value
Meaningful response 0.11 0.13 0.47
Praised 0.05 0.06 0.17
Offer to help 0.03 0.05 0.09
Warned against running 0.01 0.02 0.14