[⚠️NOTE] Data still needs some finetunning and QA.

A computational linguistics tool to identify "forgotten" Romanian words - terms that exist in official dictionaries but have fallen out of modern usage.

Status: 📚 Definitions + 🔍 Phase 3 + 🌐 Web UI (PHP) — shortlist generated, definitions complete, web validation next

• Generates a list of the least used or forgotten words from Romanian dictionaries
• Compares official dictionaries (including archaisms) against usage frequency data
• Identifies linguistic "dark matter" - words that exist in dictionaries but have fallen out of active use
• Produces curated lists with rarity scores and linguistic metadata

See exploratory UI prototype: lab.gov2.ro/oțios

Vezi și: initial specs / live (google doc)

flowchart TD
    DEX[("DEX Online dump\n1.2 GB MySQL")]

    subgraph P1["Phase 1 · Dictionary Analysis"]
        A["create_sample_db.py"] --> B[("dex-sample.sql")]
        B --> C["extract_lexemes.py"] --> D[("lexemes.db\n315k lexemes")]
        D --> E["analyze_forgotten_words.py"] --> F[("forgotten_words_v1.csv")]
        F --> G["create_curated_list.py"] --> H[("forgotten_words_curated.csv\n~140k candidates")]
        D --> TAX["extract_taxonomy.py\n(run once)"] --> D
    end

    subgraph P2["Phase 2 · Corpus Validation"]
        subgraph P2A["2a · wordfreq — fast, rough"]
            WF["validate_with_wordfreq.py"] --> WF_OUT[("validated_wordfreq.csv\n1,868 rows")]
        end
        subgraph P2B["2b · Diachronic — recommended ✅"]
            WS["process_wikisource.py\n(historical · 14M tokens)"]
            CX["process_culturax.py\n(modern web · 17B tokens)"]
            WS & CX --> CORP[("corpus_frequencies.db")]
            CORP --> DIA["validate_diachronic.py"] --> DIA_OUT[("forgotten_words_diachronic.csv\n130k rows · 4 taxonomy cols")]
        end
        subgraph P2C["2c · Legacy Wikipedia — ⚠️ P0 bug"]
            DL["download_wikipedia_ro.py"] --> PC["process_corpus.py"] --> VFW["validate_forgotten_words.py"] --> LEG[("validated.csv")]
        end
    end

    subgraph P25["Phase 2.5 · Shortlist"]
        SL["make_shortlist.py"] --> SL_OUT[("forgotten_words_shortlist.csv\n23,112 rows · 3 tiers")]
    end

    subgraph P3["Phase 3 · Web Validation"]
        SW["search_wild.py\n--provider ddg | google"] --> WEB_OUT[("forgotten_words_web_validated.csv\nweb_score · last_seen_approx")]
    end

    DEX --> A
    H --> WF
    H --> WS & CX
    H --> DL
    D -.->|"taxonomy tags\n(dex_pos, register,\ndomain, etymology)"| DIA
    DIA_OUT --> SL
    WF_OUT --> SL
    SL_OUT --> SW
    LEG --> SW

Phase 2 paths are alternatives — run 2a for a quick pass, 2b for the recommended diachronic analysis (historical vs modern corpora), or 2c only if reproducing earlier results (it has a known P0 bug). make_shortlist.py (Phase 2.5) filters the 130k diachronic rows down to the ~23k most defensible forgotten words before web validation.

# Activate virtual environment (adjust path to your venv)
source ~/devbox/envs/240826/bin/activate

# Install all dependencies
pip install -r requirements.txt

# 1. Create sample database (reduces 1.2GB to 285MB)
python create_sample_db.py

# 2. Extract lexeme data (creates CSV + SQLite database)
python extract_lexemes.py

# 3. Generate analysis and statistics
python analyze_forgotten_words.py

# 4. Create final curated list
python create_curated_list.py

Output: forgotten_words_curated.csv (~140k candidates)

python validate_with_wordfreq.py

Outputs two files:

• forgotten_words_validated_wordfreq.csv — words with Zipf < 3.0 (tier=forgotten)
• rare_words_wordfreq.csv — words with Zipf 3.0–4.5 and a non-empty dex_register tag (tier=rare_in_use)

The tier column supersedes is_forgotten (kept for backward compatibility). Note: wordfreq's Romanian coverage is binary (0.000 or ≥ 3.0), so treat this as a rough first pass. The rare_in_use gate on dex_register prevents modern unmarked vocabulary (neurologie, cowboy…) from polluting the rare list.

Extracts Tag, ObjectTag, and EntryLexeme tables from the DEX dump into lexemes.db, enabling register/domain/etymology/POS columns in the diachronic output.

# Sample dump (fast, ~5% coverage)
python extract_taxonomy.py

# Full dump (recommended — ~990k ObjectTag rows, full coverage)
python extract_taxonomy.py --sql data/dictionaries/dex-database.sql

Uses Wikisource RO (historical literary baseline) and CulturaX RO (modern web) to compute actual per-corpus frequencies. Designed to find words that appear in 19th-century literature but are absent from modern text.

# Wikisource — test run (500 docs, ~10s)
python process_wikisource.py --test

# Wikisource — full run (best on a VPS)
mkdir -p data/logs
nohup python process_wikisource.py --resume >> data/logs/wikisource.log 2>&1 &
echo $! > data/logs/wikisource.pid

# CulturaX — full run (64 parquet shards, ~40M docs; auto-restarts on network errors)
# Interactive (watch it run):
while true; do
    python -u process_culturax.py --resume
    [ $? -eq 0 ] && break
    echo "[$(date)] restarting in 15s..." && sleep 15
done

# Background (logs to file):
VENV=~/g2-dev/monitorulpreturilor/venv/bin/python
mkdir -p data/logs
nohup bash -c "while true; do $VENV -u process_culturax.py --resume; [ \$? -eq 0 ] && break; echo \"[\$(date)] restarting in 15s...\"; sleep 15; done" \
  >> data/logs/culturax.log 2>&1 &
echo $! > data/logs/culturax.pid

Output: corpus_frequencies.db with corpus_name = 'wikisource_ro' and corpus_name = 'culturax_ro'.

Note: process_culturax.py reads the 64 parquet shards directly via HfFileSystem + pyarrow and checkpoints at file + row-group level. This avoids the datasets streaming ds.skip() cycling bug that triggers when the checkpoint offset exceeds the dataset size.

# Compare historical vs modern frequencies, add taxonomy columns
python validate_diachronic.py
# Output: forgotten_words_diachronic.csv (130k rows)

# Filter down to the most defensible forgotten words
python make_shortlist.py --stats   # preview counts by tier
python make_shortlist.py           # write forgotten_words_shortlist.csv (~17k rows)

The DEX MySQL dump's DefinitionSimple table only covers ~4.6k of the 17.4k shortlist words. scrape_definitions.py fills the remaining gaps by extracting the synthesis (definition) from dexonline.ro for each missing word.

# Smoke test (5 words, no HTTP)
python scrape_definitions.py --dry-run --limit 5

# Small live run (test the scraper)
python scrape_definitions.py --limit 20 --delay 3.0

# Full run (all missing words, ~5–7 hours at 3s/request)
python scrape_definitions.py --delay 3.0 --merge

# Resume an interrupted run
python scrape_definitions.py --delay 3.0 --merge      # automatically skips already-scraped

# Just upsert checkpoint into DB (if scraping completed but merge wasn't run)
python scrape_definitions.py --merge-only

Output: data/processed/scraped_definitions.csv (checkpoint with columns: word, definition, source_url, scraped_at, status). With --merge, all status=ok rows are upserted into definitions.db immediately. Resume is safe — each row is flushed instantly; Ctrl+C stops cleanly.

# Dry run first
python search_wild.py --input data/processed/forgotten_words_shortlist.csv \
    --provider ddg --limit 5 --dry-run

# DDG triage (no API key, good for first pass)
python search_wild.py --input data/processed/forgotten_words_shortlist.csv \
    --provider ddg --limit 200 --delay 2

# Google CSE (cleaner results, needs env vars, 100/day free tier)
export GOOGLE_API_KEY="AIza..."
export GOOGLE_CSE_ID="017576..."
python search_wild.py --input data/processed/forgotten_words_shortlist.csv \
    --provider google --limit 100

flowchart LR
    subgraph JOBS["Long-running corpus jobs"]
        J1["process_wikisource.py"]
        J2["process_culturax.py"]
    end

    subgraph LOGS["data/logs/"]
        L1["wikisource.log / .pid"]
        L2["culturax.log / .pid"]
        L3["run_history.jsonl"]
        L4["health_status.json"]
    end

    subgraph MON["Monitoring scripts"]
        ST["status.py\n(read-only · any time)"]
        HC["health_check.py\n(cron · every 30 min)"]
        AU["audit.py\n(cron · daily 02:00)"]
    end

    subgraph ALERT["Alert channels"]
        AW["webhook\nOTZIOS_ALERT_URL"]
        AE["email\nOTZIOS_ALERT_EMAIL"]
    end

    JOBS --> LOGS
    LOGS --> ST & HC & AU
    HC & AU --> AW & AE

health_check.py, audit.py, and status.py keep tabs on long-running corpus jobs. Run them manually or via cron (see CLAUDE.md for crontab lines).

python status.py                # at-a-glance summary — corpora, artifacts, loops, audit
python health_check.py          # check liveness, stalls, log errors, completion
python audit.py                 # snapshot run history + DB quality checks
python health_check.py --dry-run  # print without alerting or writing state

status.py is read-only — safe to run any time. health_check.py and audit.py write logs and may alert.

Set OTZIOS_ALERT_URL (webhook) or OTZIOS_ALERT_EMAIL to receive push alerts.

Apostrophes in the word column — DEX Online encodes syllable stress using apostrophes (e.g. bucl'e, băt'ârn). These are not real Romanian words; the clean form is in word_no_accent. The validated output from validate_with_wordfreq.py uses word_no_accent for all lookups and moves the raw word column to the end of the CSV for reference.

All generated files live under data/processed/. Columns shared across files have the same meaning everywhere.

How the files relate:

forgotten_words_curated.csv    — 140k dictionary suspects (no corpus signal)
        ↓ validate_diachronic.py
forgotten_words_diachronic.csv — 130k rows with corpus frequencies + taxonomy
        ↓ make_shortlist.py
forgotten_words_shortlist.csv  — ~17k most defensible forgotten words (2 tiers)
        ↓ search_wild.py
forgotten_words_web_validated.csv — shortlist + real-world web presence

Every DEX entry with frequency < 1.0 that passes form filters (length, not a proper noun, has a word-class marker). No corpus evidence — these are suspects, not confirmed forgotten words. Currently ~140k rows.

One row per candidate from forgotten_words_curated.csv, enriched with measured frequencies from both corpora and a verdict. This is the file to use for any downstream analysis — it tells you whether each word is actually missing from modern text, and by how much.

Verdict values:

Generated by make_shortlist.py from the diachronic CSV. Three selection tiers, all with POS exclusions applied:

Tier B: DEX editorial + absent from all corpora — two independent archaism signals. Tier C: highest DEX legitimacy but no corpus trace at all — the "most forgotten" words (e.g. oțios, dex_frequency=0.85). Tune Tier C with --dex-freq-threshold (default 0.85).

All rows carry is_forgotten = true (required by search_wild.py). Columns are a subset of the diachronic CSV plus confidence_tier.

All columns from the shortlist, plus web search results from search_wild.py.

Quick frequency screen via the wordfreq library, without streaming any corpus.

otios/
├── data/
│   ├── dictionaries/       # DEX Online database (download separately)
│   └── processed/          # Generated lexeme data and results
├── docs/                   # Documentation and specifications
│   ├── scripts-guide.md    # Detailed script documentation
│   ├── romanian-forgotten-words-spec.md
│   └── results-summary.md
└── *.py                    # Processing scripts

• docs/scripts-guide.md - Comprehensive guide to all scripts
• docs/romanian-forgotten-words-spec.md - Technical specification
• docs/results-summary.md - Analysis results and findings
• docs/oțios.docx.md - Initial brainstorming document
• more docs: PHASE2_COMPLETE.md; phase2-test-results.md; scripts-guide.md

Top extinct words from the diachronic analysis (high historical frequency, near-zero modern):

DEX-tagged archaic words with no corpus signal at all (Tier B — "dark matter"):

• DEX Online Database: Official Romanian dictionary (1.2 GB MySQL dump)
  • Download: dexonline.ro
  • 315,247 lexemes with frequency data
  • Archaic markers and linguistic metadata

• fix mysql import - try a llm assisted import
• create another sample db with max 3 inserts per table - for analytics

• Database setup and conversion
• Lexeme extraction pipeline
• Frequency-based analysis
• Quality filtering and curation
• CSV export with ~140k candidates (cutoff raised to DEX freq < 1.0)

Output: forgotten_words_curated.csv — ~140k candidates (dictionary suspects, corpus validation is the real gate)

• Wikisource RO corpus — 12,921 docs, 14.3M tokens (historical baseline)
• CulturaX RO corpus — 40.3M docs, 17.0B tokens (modern web)
• Diachronic comparison: log₂(hist_ppm / modern_ppm) per word
• Taxonomy enrichment: dex_pos, dex_register, dex_domain, dex_etymology
• Shortlist generation: 23,112 words across 5 confidence tiers (Tier C added for corpus-absent DEX-canonical words)

Output: forgotten_words_diachronic.csv (130k rows) → forgotten_words_shortlist.csv (23k rows)

• Extract full definitions from DEX database
• Join Definition and DefinitionSimple tables
• Identify archaic markers (înv., arh., reg., dial.) — dex_register column via Tag taxonomy
• Extract etymology information — dex_etymology column (grecism, latinism, turcism…)
• Add part-of-speech tagging — dex_pos column (substantiv neutru, adjectiv, verb…)
• Flag words with no definition body ("Fără definiție." entries like nombrilist)
• Parse first attestation dates
• Temporal analysis (when words fell out of use)
• Link to word families and cognates

• Integrate Romanian lemmatizer (spaCy-ro or nlp-cube)
• Match inflected forms to base words
• Improve recall (find "frumoaselor" when searching "frumos")
• Named entity recognition for better filtering
• Semantic clustering of forgotten words

• DDG triage pass on shortlist (~17k words, no quota)
• Google CSE pass on high-confidence subset (100/day free tier)
• Cross-reference: corpus verdict vs web presence

• Exploratory UI for browsing the shortlist (filter by tier, POS, etymology, domain, verdict, marks)
• Word detail view: DEX definition, corpus stats, dexonline.ro link
• PHP thin-API port — deployable on shared hosting (public/, tools/build_ui_db.py)
• localStorage bookmarks / notes / quick-tags (no server-side auth needed)
• Forgotten / rare-in-use toggle — browse two word tiers in both Flask and PHP UIs
• REST API for programmatic access
• Interactive visualizations
  • Frequency decay curves (hist_ppm vs modern_ppm scatter)
  • Etymological breakdown of extinct words
  • Word cloud weighted by log_ratio

• Revival potential scoring algorithm
• Compare with other Romance languages
• Historical corpus analysis (Project Gutenberg)
• Machine translation of forgotten word contexts
• Crowdsourced validation platform
• Word-of-the-day feature
• Educational tools and quizzes
• Create a reverse, browse news and r/romania and find new words, used more than 3? times that are not in dictionary -> alternative dictionary

• tools: convert texts to archaic form - less used words. with a coeficient of uniqueness (bigger number, harder words)
• filter out uninteresting words. Too domain specific: medicine, biology etc
• one word a day game? quizz, guess what it means?

1. No lemmatization — buclele doesn't match bucle in the corpus. Inflected forms are invisible. Adding simplemma would significantly improve recall (backlog #6).
2. POS tag noise — some words get wrong POS tags due to the ObjectTag join occasionally pulling tags from adjacent dictionary entries. Supplementary metadata only; doesn't affect core analysis.
3. Sparse etymology -ism tags — many words store "limba franceză" not "franțuzism" in DEX. Both are captured but the vocabulary is inconsistent across DEX editors.
4. absent verdict ambiguity — 83k words with no corpus signal conflate "truly unused" with "only appears in inflected forms not tracked". Lemmatization (#6) would reduce this category significantly.

# Web validation — DDG triage on full shortlist
python search_wild.py --input data/processed/forgotten_words_shortlist.csv \
    --provider ddg --limit 500 --delay 2

# Or just the highest-confidence tier first
python make_shortlist.py --limit 1137   # corpus_extinct only
python search_wild.py --input data/processed/forgotten_words_shortlist.csv \
    --provider ddg --delay 2

See Activity History and Backlog for the changelong and open items / roadmap.

See CLAUDE.md for development guidelines and project context.