Wesley Corpus

Training data for the Wesley Index project.

wesley-corpus/
├── raw/                    # Original downloads, untouched
│   ├── john-wesley/
│   └── charles-wesley/
├── cleaned/                # OCR-corrected, standardized text
│   ├── john-wesley/
│   └── charles-wesley/
├── chunked/
│   └── passages.jsonl      # All passages in JSON Lines format
├── metadata/
│   ├── sources.csv         # Tracking spreadsheet for all sources
│   ├── themes.csv          # Theological theme taxonomy
│   └── annotations/        # Human-labeled training data (Phase 3)
├── process_corpus.py       # Processing tools
└── README.md               # This file

Save the raw text to raw/john-wesley/ or raw/charles-wesley/:

# Example: download a sermon
curl -o raw/john-wesley/sermon-001-salvation-by-faith.txt \
  "https://wesley.nnu.edu/john-wesley/the-sermons-of-john-wesley-1872-edition/sermon-1-salvation-by-faith/"

Or just copy/paste from the website into a .txt file.

python process_corpus.py clean \
  raw/john-wesley/sermon-001-salvation-by-faith.txt \
  cleaned/john-wesley/sermon-001-salvation-by-faith.txt

This fixes OCR errors, normalizes whitespace, and removes page numbers.

python process_corpus.py chunk \
  cleaned/john-wesley/sermon-001-salvation-by-faith.txt \
  --source-id jw-sermon-001 \
  --author john-wesley \
  --title "Salvation by Faith" \
  --type sermon \
  --year 1738 \
  --url "https://wesley.nnu.edu/..."

This creates passages in chunked/passages.jsonl.

python process_corpus.py chunk \
  cleaned/charles-wesley/wrestling-jacob.txt \
  --source-id cw-wrestling-jacob \
  --author charles-wesley \
  --title "Wrestling Jacob" \
  --type hymn \
  --year 1742 \
  --hymn

This creates both a full-hymn passage and individual stanza passages.

python process_corpus.py stats
python process_corpus.py validate

Each passage in passages.jsonl is a JSON object:

{
  "id": "jw-sermon-001-003",
  "author": "john-wesley",
  "source_id": "jw-sermon-001",
  "source_title": "Salvation by Faith",
  "source_type": "sermon",
  "year": 1738,
  "chunk_index": 3,
  "text": "The actual passage text...",
  "word_count": 247,
  "source_url": "https://wesley.nnu.edu/...",
  "themes": ["justifying-grace", "faith"],
  "created_at": "2025-01-28T12:00:00"
}

See metadata/themes.csv for the full taxonomy. Key themes:

• prevenient-grace - Grace preceding conversion
• justifying-grace - Pardon and new birth
• sanctifying-grace - Growth in holiness
• entire-sanctification - Christian perfection
• means-of-grace - Prayer, Scripture, Eucharist
• social-holiness - Faith in community
• assurance - Witness of the Spirit

Update metadata/sources.csv as you work:

1. Change status from pending to downloaded, cleaned, or chunked
2. Add download_date when you download
3. Add notes about any issues

• Start with the Standard Sermons - they're the theological core
• Wesley Center Online has the cleanest text versions
• Review chunks manually - the automated chunking isn't perfect
• Add themes as you go - even rough tags help
• Keep raw files - you may need to re-process later

• Wesley Center Online - Primary source
• CCEL - Alternative texts
• Hymnary.org - Charles Wesley hymns
• Internet Archive - Historical editions