जाली — forged / fake. A jaali ration card is the colloquial term for a fraudulent ration card in India.

jaali is a forensic toolkit for Indian PDS (Public Distribution System / ration) data. It finds duplicate and suspicious ration-card images in a Google Cloud Storage bucket and joins them to the PDS relational tables to surface fraud signals — one photo used across many cards, the same person holding multiple cards, ghost beneficiaries — with reproducible, offline HTML review viewers. Report only — it never modifies the source data.

⚠️ Privacy / PII. The inputs and every output (manifests, duplicate clusters, applicant names, card numbers, face thumbnails, annotations) describe real people and are personal data. They are git-ignored by design — never commit, publish, or share data/, out*/, inspect/, or any *.csv / *.parquet / thumbnail produced here. This repo is code only.

Built for the Rajasthan PDS corpus:

• Card images — gs://ration/{rural,urban}/jpg/<cardnumber>.jpg (~15.85M rural images; each object is named after its 12-digit ration card number). Note: gs://ration is requester-pays, so a non-owner identity must pass a billing user_project.
• Relational tables — Harvard Dataverse doi:10.7910/DVN/FIFZEX ("Transaction Level Ration Data from Rajasthan"): rationcard (Card_Type APL/BPL/AAY, Applicant_Name, FPS, geography), rationcarddata, rationcardfamily, rationcardtransaction (offtake), fps. Join key = Ration_Card_Number = the image filename, zero-padded to 12 digits. Access is token-gated (Dataverse API key; the access endpoint 303-redirects to S3).

Every flag is an edge: the file, the file it is similar to, a score, and a human-readable reason (exact_edges.csv / near_edges.csv).

How "near-duplicate" is decided: two images match if either perceptual hash is within a small Hamming distance — dHash (gradient; robust to re-compression / brightness) ≤ --dhash-max (default 6) or pHash (DCT structure; robust to crop/scale) ≤ --phash-max (default 8), i.e. ≤ ~2–3 % of bits differ. Matches are unioned into groups; a BK-tree avoids O(n²) comparison. Exact (MD5) dups are a strict subset, annotated also_exact.

Joins the duplicate-photo clusters to the PDS card metadata and grades suspicion:

1. Photo impersonation — one byte-identical photo (MD5) registered under many distinct applicant names across multiple shops; strongest on subsidised (BPL/AAY) cards. (Real example in the Rajasthan data: one face on 1,078 cards under 943 different names across 42 fair-price shops.)
2. Duplicate identity — same Applicant_Name + Father_Name in the same village on more than one card (duplicate-beneficiary enrolment).
3. Cross signal — cards that trip both a photo cluster and a duplicate identity (highest confidence).

Outputs: fire1_photo_reuse.csv, fire2_duplicate_identity.csv, forensic_summary.csv. Heuristic note: near-blank template images (~600–1000 B) and the rare corrupt file are flagged separately so they are not mistaken for real impersonation; size + the viewer resolve them.

Static, offline HTML (open index.html — no server). Reproducible seeded weighted sampling (weight ∝ how often an image is duplicated; big clusters always included):

• --mode clusters — all duplicate clusters, concern-first.
• --mode impersonation — Fire-1 ranked by number of distinct names, showing the shared photo, the applicant names it is registered under (one per line), and badges (distinct names / shops / districts / subsidised / likely-blank).

Annotators label each item and Export annotations.json.

python3 -m venv .venv
.venv/bin/python -m pip install -e .          # or: pip install -r requirements.txt
gcloud auth application-default login          # for the image stages

# estimate cost first (no spend)
python -m jaali --dry-run

# cheapest signal: byte-identical re-uploads, no downloads
python -m jaali --stages 0,1 --project <gcp-project>

# full image pipeline
python -m jaali --stages all --project <gcp-project>

# scope/test on a sample
python -m jaali --stages all --max-images 5000 --project <gcp-project>

# forensic join (needs data/rural_rationcard.csv.gz from Dataverse)
python -m jaali.forensics --out out_full --data data

# build the impersonation review viewer (reproducible; seed 42)
python -m jaali.cluster_review --mode impersonation --out out_full --seed 42

At full scale, run the download/compute stages in-region (Cloud Shell or a VM in the bucket's region) to make egress $0. All stages cache to out/ (manifest.parquet, phashes.parquet, CSVs) so reruns never re-download.

• --stages 0,1,2,issues,viewer,report (or all)
• --max-images N — cap objects (testing/scoping)
• --sample N — pairs/clusters in the HTML viewer · --seed N — reproducible sample
• --dhash-max / --phash-max — near-dup thresholds
• --budget $ — abort the download stage if estimated cost exceeds this
• --workers N — parallel download workers (default 32)

The only way this gets expensive is paid per-image cloud APIs (Vision OCR / web-detection) across the full corpus — at Vision rates that is tens of thousands of dollars for ~10M images. jaali deliberately uses local / open-source methods for the full sweep; the image pipeline over the whole rural set costs single-digit dollars (and $0 egress in-region). --dry-run prints an estimate; --budget caps spend.

MIT — see LICENSE.