Analysis scripts for REDS (Rural Economic and Demographic Survey) data quality assessment.

No state overlap between SEPRI1 and SEPRI2.

Script | Figure

Village sample sizes vary dramatically: 2 to 5,134 HH (2,567x ratio). This extreme variation affects clustering and inference.

Cluster structure: 92,996 obs across 193 villages, 83 districts, 13 states

Script | Figure

• Only 2.4% of households are panel HH
• 0.3% locked houses overall
• Non-interview reasons: 69% travelling, 24% migrated out

Script | Figure

Impossible Values:

• 990 interviews (1.06%) with impossible duration (<5 min or >4 hours)
• Mostly in SEPRI2 (779 vs 211 in SEPRI1)
• No duplicates found

Missing Data:

• Land ownership (q1_10): 6.6% missing overall
• Caste/religion: <0.2% missing
• Many "3rd choice" variables 99%+ missing (by design)

Script | Figure

Name Sharing:

• SEPRI1: 3,530 names, 130 represent 2+ people, 12 represent 4+ people
• SEPRI2: 2,171 names, 112 represent 2+ people, 10 represent 4+ people

Productivity:

• Median 3 interviews/interviewer-day (both datasets)
• After time-overlap adjustment, max ~17 interviews/person/day

Duration:

• SEPRI1: median 105 min
• SEPRI2: median 120 min
• End-of-day rushing: duration drops to ~60 min after 5pm

Script | Figure

Strong evidence of rounding:

• 40.4% whole numbers (expected ~10%)
• 48.8% multiples of 0.5 acres (expected ~20%)
• 36.6% of interviewers have >60% whole number responses

Script | Figure

Shorter interviews have MORE missing data:

• SEPRI1 <60 min: 15.2% missing land; 90-120 min: 4.6%
• SEPRI2 <60 min: 8.5% missing land; 90-120 min: 2.1%
• Correlation: r = -0.37 (SEPRI1), r = -0.19 (SEPRI2)

Script | Figure

How much variance do interviewers explain beyond village-level differences?

Key finding: Land ownership shows low interviewer effects (expected for objective measures). However, caste variables show high interviewer effects (14-16% ICC), meaning interviewers within the same village get systematically different caste distributions. Possible explanations:

• Caste boundaries are subjective (esp. OBC vs General)
• Non-random HH assignment within villages (caste-segregated hamlets)
• Interviewer bias or fabrication

Script | Figure

With 193 villages across 13 states and highly unequal cluster sizes, proper clustering is critical.

Significance drops from 11/11 (HC1) to 5/11 (state bootstrap). Only 5 specs survive all methods; 6 flip under wild bootstrap. Results should survive both village-level (193 clusters) and state-level (13 clusters) wild bootstrap to be trusted.

Script | Figure

Matching Results (SEPRI2 → SHRUG via LGD):

• 78 of 90 villages fuzzy-matched to LGD (86.7%)
• 72 villages successfully linked to SHRUG (80%)
• Match quality: 31 exact, 27 distance=1, 13 distance=2, 7 distance=3

Validation Correlations:

Matched Village Characteristics (Census 2011):

• Median population: 1,524
• Median SC/ST %: 22.8% (REDS: 26.8%)
• Median literacy rate: available in data/reds_shrug_matched.csv

Re-analysis of published papers using wild cluster bootstrap to assess inference robustness with few clusters.

Script | Figure

Re-analyzed "Development from Representation? A Study of Quotas for the Scheduled Castes in India" (AEJ:Applied, Vol. 7, No. 3, pp. 196-220) with wild cluster bootstrap. With only 15 state clusters, 4 of 10 HC1-significant results flip to non-significant:

With only 15 clusters, standard clustered SEs are unreliable. Wild bootstrap is essential.

Script | Figure

Re-analyzed "Networks and Misallocation: Insurance, Migration, and the Rural-Urban Wage Gap" (AER, 2015). The original paper uses wild cluster bootstrap for Table 8a (15 state clusters) and standard bootstrap for Table 6 (148 caste clusters).

Table 8a (15 state clusters):

Table 6 (148 caste clusters):

The negative jati income effect on migration is robust across specifications and inference methods.

For linking to census/SHRUG:

Matching strategy:

1. Use state codes directly (standard)
2. Fuzzy match village names within state to SHRUG village names
3. Or use REDS documentation for district/village code crosswalk if available

library(haven)
library(dplyr)
library(tidyr)
library(ggplot2)
library(patchwork)
library(stringdist)  # for SHRUG fuzzy matching
library(lme4)        # for interviewer FE analysis
library(fixest)      # for fast FE estimation
library(fwildclusterboot)  # for wild cluster bootstrap
library(sandwich)    # for robust SEs
library(lmtest)      # for coefficient tests

cd reds
Rscript scripts/02_obs_per_village_full.R
Rscript scripts/03_interviewer_analysis.R
Rscript scripts/04_data_quality.R
Rscript scripts/05_temporal_patterns.R
Rscript scripts/06_interviewer_effects.R
Rscript scripts/07_panel_attrition.R
Rscript scripts/08_shrug_merge.R  # requires ../quota/data/lgd/ and ../quota/data/shrug/
Rscript scripts/09_interviewer_fe.R
Rscript scripts/10_inference_robustness.R

Outputs saved to figs/ and data/.