Historical‑OCR‑Preprocessor

A demonstration repository for preprocessing blurry and unreadable text from historical scanned documents using a UNet‑based model. Includes scripts to extract test samples from your original .npz dataset and run end‑to‑end evaluations.

• extract_test_data.py: Extracts the first N degraded↔clean image pairs from an .npz file into a test_data/ folder for demo and verification.
• test_model_v13_presentation.py: Loads a trained checkpoint, runs sample visualizations, and performs large‑scale OCR‑based evaluations.
• Checkpoints: Pretrained weights in checkpoints_v12/ (e.g., checkpoint_epoch_272.pth).

• Python 3.7 or later
• Install dependencies:

  pip install -r requirements.txt

  Ensure numpy, pillow, torch, torchvision, opencv-python, pytesseract, pytorch-msssim, rich, matplotlib are installed.

├── checkpoints_v12/                 # model weights
│   └── checkpoint_epoch_272.pth
├── extract_test_data.py             # gets some images for reference
├── test_model_v13_presentation.py   # main evaluation & visualization
├── training_data_v11_english_final_test.npz  # the original dataset
├── test_data/                       # generated by extract_test_data.py
│   ├── degraded/
│   ├── clean/
│   └── index_map.csv
└── README.md                        # this file

Use the original .npz file to dump the first N samples:

python extract_test_data.py --npz path/to/training_data_v11_english_final_test.npz --out test_data --max 500

• --npz: Path to your .npz (must contain degraded and clean arrays).
• --out: Output folder (default: test_data).
• --max: Maximum number of pairs to extract (default: 500).

Note: The test_data/ directory is for demonstration and verification only. It should not be used for training.

In test_model_v13_presentation.py, you can override defaults via command‑line flags or by editing constants at the top of the script:

• checkpoint_path: Path to your .pth checkpoint.
• npz_file: If you prefer loading directly from .npz, update this path.
• num_eval: Total number of image pairs to evaluate (e.g., 10 000).
• num_workers: Number of threads for parallel evaluation.
• transform: Change or add torchvision transforms if needed.

Example CLI:

python test_model_v13_presentation.py \
  --checkpoint checkpoints_v12/checkpoint_epoch_272.pth \
  --npz training_data_v11_english_final_test.npz \
  --num_eval 5000 \
  --num_workers 8

1. Extract demo samples:

   python extract_test_data.py --npz training_data_v11_english_final_test.npz

2. Visualize a few samples:

   python test_model_v13_presentation.py --num_eval 5

3. Evaluate overall performance on full set:

   python test_model_v13_presentation.py --num_eval 10000 --num_workers 20

MIT © Siddharth Rodrigues

Feel free to adapt, extend, and integrate this code into your own workflows!