Sensitive Questions

 

POLSCI 4SS3
Winter 2023

Announcements

  • Groups put in contact later today

  • Topic is highly negotiable

  • Don’t forget to schedule a meeting before break!

Last week

  • Elements of models (signature, functional relations, probability distributions)

  • Panel surveys: Good for understanding before-after effects, expensive to conduct

Today

  • Elements of data strategies and answer strategies

  • Research designs for sensitive questions

  • Lab: Practice writing research designs on your own

Data and Answer Strategies

Elements of a data strategy

  1. Sampling (random, nonrandom)

  2. Treatment assignment (if applicable)

  3. Measurement (validity, reliability)

Measurement as multiple independent draws

Elements of an answer strategy

  1. Answer characterization (domain, units, outcomes, conditions)

  2. Uncertainty (Bayesian, frequentist)

  3. Procedure (e.g. OLS, difference-in-means)

Types of answer strategies

  1. Point estimation

Example

  • What is the proportion of French speakers?

  • What is the effect of taking a political science course on political interest?

Types of answer strategies

  1. Hypothesis testing

Example

  • Has the proportion of French speakers increased?

  • Is there an effect of taking a political science course on political interest?

Sensitive Questions

Why do people lie in surveys?

  1. Inattention

  2. Satisficing

  3. Limited options

  4. Demand effects

  5. Social desirability/sensitivity bias

Why do people lie in surveys?

  1. Inattention

  2. Satisficing

  3. Limited options

  4. Demand effects

  5. Social desirability/sensitivity bias

How to prevent lying?

  • Add noise to the question

  • Two approaches:

  1. Distract from the sensitive attitude/behavior

  2. Guarantee anonymity

  • Different designs vary on how they combine the two

Randomized response

For this question, I want you to answer yes or no.

Randomized response

For this question, I want you to answer yes or no. But I want you to consider the number of your dice throw.

Randomized response

For this question, I want you to answer yes or no. But I want you to consider the number of your dice throw. If shows on the dice, tell me no.

Randomized response

For this question, I want you to answer yes or no. But I want you to consider the number of your dice throw. If shows on the dice, tell me no. If shows, tell me yes.

Randomized response

For this question, I want you to answer yes or no. But I want you to consider the number of your dice throw. If shows on the dice, tell me no. If shows, tell me yes. But if another number shows, tell me your own opinion about the question.

 

[TURN AWAY FROM RESPONDENT]

 

Now you throw the dice so that I cannot see what comes out.

Randomized response

For this question, I want you to answer yes or no. But I want you to consider the number of your dice throw. If shows on the dice, tell me no. If shows, tell me yes. But if another number shows, tell me your own opinion about the question.

 

[TURN AWAY FROM RESPONDENT]

 

Have you thrown the dice?

Randomized response

For this question, I want you to answer yes or no. But I want you to consider the number of your dice throw. If shows on the dice, tell me no. If shows, tell me yes. But if another number shows, tell me your own opinion about the question.

 

[TURN AWAY FROM RESPONDENT]

 

Have you picked it up?

Randomized response

For this question, I want you to answer yes or no. But I want you to consider the number of your dice throw. If shows on the dice, tell me no. If shows, tell me yes. But if another number shows, tell me your own opinion about the question.

 

Now, during the height of the conflict in 2007 and 2008 (in Afghanistan), did you know any militants, like a family member, a friend, or someone you talked to on a regular basis?

Please, before you answer, take note of the number you rolled on the dice.

Answer strategy

  • We know that about \(1/6 \approx 0.17\) respondents said yes because they rolled a

  • So if \(30\%\) in total said yes

  • We have \(\widehat{Y} = 0.3 - 0.17 = 0.13\) as our estimate of the proportion of the population who know a member of a militant organization

  • But we do not know who they are in our survey!

Assumptions

1. Honesty given protection

People respond honestly when guaranteed anonymity.

2. One-sided lying

Those who do not hold the sensitive trait never falsely claim to bear it.

  • These cannot be verified with data!

List experiment

Now I am going to read you three things that make people angry or upset.

List experiment

Now I am going to read you three things that make people angry or upset. After I read all three, just tell me HOW MANY of them upset you.

List experiment

Now I am going to read you three things that make people angry or upset. After I read all three, just tell me HOW MANY of them upset you. I don’t want to know which ones, just HOW MANY.

Control group

  1. The federal government increasing the tax on gasoline
  2. Professional athletes getting million-dollar contracts
  3. Large corporations polluting the environment

List experiment

Now I am going to read you three things that make people angry or upset. After I read all three, just tell me HOW MANY of them upset you. I don’t want to know which ones, just HOW MANY.

Treatment group

  1. The federal government increasing the tax on gasoline
  2. Professional athletes getting million-dollar contracts
  3. Large corporations polluting the environment
  4. A black family moving next door

Answer strategy

  • Respondents are randomly assigned to conditions

  • Differences in responses can only be attributed to the presence/absence of the sensitive item

  • So \(\widehat{Y} = \text{Mean(treatment)} - \text{Mean(control)})\) estimates the proportion in the population of interest who would get upset if a black family moves next door

  • But we do not know who they are in our survey!

Assumptions

1. No liars

Those who do not hold the sensitive item never falsely claim to bear it.

2. No design effects

Including the sensitive item does not change how participants respond to the baseline items

  • These can only be evaluated indirectly

Other research designs

Example

Vignette

Below you will read three things that sometimes people oppose or are against. After you read all three, just tell us HOW MANY of them you OPPOSE. We don’t want to know which ones, just HOW MANY.

Vignette

  1. The federal government increasing assistance to the poor
  2. Professional athletes making millions of dollars per year
  3. Large corporations polluting the environment

Treatment 1

4. Granting citizenship to a legal immigrant who is Muslim

Treatment 2

4. Granting citizenship to a legal immigrant who is Christian

Next Week

Survey Experiments

Focus on: What are they good for?

Break time!

 

Lab