**POLSCI 4SS3**

Winter 2023

**Policy**is an umbrella term to describe government programs or operations at different levels**Examples:**How long should form 57B be?

Should we get help from private clinics to clear surgey backlogs?

Should the education budget increase?

When should the next federal election be held?

Of course we want to base policy on evidence!

But there is no

*objective*evidence when it comes to human behaviorWe say evidence-

*informed*because the best we can do is try to prove ourselves wrong, but we cannot*base*policy on evidence the same way medicine does

- Evidence as
**insight**

- Evidence as
**evaluation**

*A lady declares that by tasting a cup of tea made with milk she can discriminate whether the milk or the tea infusion was first added to the cup*

How do you **evaluate** this claim?

Suppose we have eight milk tea cups

4 milk first, 4 tea first

We arrange them in random order

Lady knows there are 4 of each, but not which ones

True Order |
||
---|---|---|

Lady's Guesses | Tea First | Milk First |

Tea First | 3 | 1 |

Milk First | 1 | 3 |

She gets it right \(6/8\) times

What can we conclude?

How does “being able to discriminate” look like?

Same for policy, we don’t know how the world where the policy works look like

But we

**do know**how a person without the ability to discriminate milk/tea order looks likeThis lets us make

**probability statements**about this**hypothetical world of no effect**

Count | Possible combinations | Total |
---|---|---|

0 | xxxx | \(1 \times 1 = 1\) |

1 | xxxo, xxox, xoxx, oxxx | \(4 \times 4 = 16\) |

2 | xxoo, xoxo, xoox, oxox, ooxx, oxxo | \(6 \times 6 = 36\) |

3 | xooo, oxoo, ooxo, ooox | \(4 \times 4 = 16\) |

4 | oooo | \(1 \times 1 = 1\) |

- We can repeat this for both successes and failures

Count | Possible combinations | Total |
---|---|---|

0 | xxxx | \(1 \times 1 = 1\) |

1 | xxxo, xxox, xoxx, oxxx | \(4 \times 4 = 16\) |

2 | xxoo, xoxo, xoox, oxox, ooxx, oxxo | \(6 \times 6 = 36\) |

3 | xooo, oxoo, ooxo, ooox | \(4 \times 4 = 16\) |

4 | oooo | \(1 \times 1 = 1\) |

- A person just guessing gets \(6/8\) cups right with probability \(\frac{16}{70} \approx 0.23\)
- And
**at least**\(6/8\) cups with \(\frac{16 + 1}{70} \approx 0.24\)

If the lady is not able to discriminate milk-tea order, the chance of observing 6/8 correct guesses or better is 24%

We can translate this to general statements about policies or experiments

If the

**null hypothesis**of no effect is true…… the

**p-value**is the probability of observing a result*equal or more extreme*than what is originally observedSmaller p-values give more evidence

**against**the null, which helps us make a case for the policy having an effect

A convention in the social sciences is to claim that something with \(p < 0.05\) is

*statistically significant*^{1}Committing to a

**significance level**implies accepting that sometimes we will get \(p < 0.05\) by chanceThis is a

**false positive**resultA good answer strategy as a

**controlled**false positive rate`(more in the lab!)`

**Focus on:** Research design alternatives