Lightweight reference-free quality evaluation for decentralized LLM inference, with online dimension calibration.
POQ_4 extends the original PoQ framework with neural Judge models, online dimension calibration, and cascade evaluation. Use the web demo for interactive scoring, or run the experiment scripts for reproduction.
| In repository | Not in repository (download or train) |
|---|---|
| Source code, configs, tests | Fine-tuned Judge weights (*.pt) — ANNTIAN/poq-judge-checkpoints |
| Processed eval data, evaluation metrics | CNN/DailyMail train split (~1.2 GB raw) |
| Training/eval JSONL splits | Hugging Face base encoders (auto-download on first use) |
UltraFeedback HF cache under training/data/openbmb___ultra_feedback/ |
Base models referenced in config/judge_models.yaml (MiniLM, DeBERTa, Qwen) are fetched from Hugging Face Hub when needed. Only the fine-tuned Judge checkpoints must be downloaded separately or retrained locally.
- Python 3.10 (see
environment.yaml) - ~5 GB disk space with checkpoints and dependencies
- NVIDIA GPU optional (CPU works, slower)
- OpenAI API key optional (GPT labeling / API judge only)
git clone <your-repo-url>
cd POQ_4Conda (recommended for GPU):
conda env create -f environment.yaml
conda activate poq_v2
pip install -r requirements.txtvenv + pip:
python -m venv .venv
# Windows: .\.venv\Scripts\Activate.ps1
# Linux/macOS: source .venv/bin/activate
pip install -U pip
pip install -r requirements.txtFor GPU PyTorch, install the CUDA build from pytorch.org before the rest of requirements.txt if the default wheel does not match your system.
cp .env.example .env
# Edit .env — only needed for OpenAI scripts or private HF downloadsCheckpoints are hosted on Hugging Face (not in git):
https://huggingface.co/ANNTIAN/poq-judge-checkpoints
python scripts/download_checkpoints.py
# default repo: ANNTIAN/poq-judge-checkpointsOr explicitly:
python scripts/download_checkpoints.py --repo ANNTIAN/poq-judge-checkpointsRequired files in training/checkpoints/:
cnn_judge_final.ptminilm_judge_final.ptdeberta_judge_final.ptcnn_tokenizer.json
Alternatives: copy those files manually into training/checkpoints/, or train from scratch:
python scripts/training/prepare_training_data.py --skip-gpt
python scripts/training/train_all_judges.pypython run_frontend.pyOpen http://127.0.0.1:8000 (use --port 8080 if 8000 is busy). Press Ctrl+C to stop.
Quick sanity check:
python -c "import torch; print(torch.__version__)"Ensure evaluation records exist (the repo may include outputs/metrics/all_with_full_gpt.jsonl or related paths). Then:
python run_experiments.py --allRun a single experiment:
python run_experiments.py --judge-comparison
python run_experiments.py --online-calibration
python run_experiments.py --cascadeAggregate results for tables:
python collect_summary.py| Path | Purpose |
|---|---|
api/, frontend/ |
FastAPI backend and static demo UI |
judges/ |
CNN / MiniLM / DeBERTa / LLM judges |
calibration/ |
Online weight updates, task detection, anchors |
cascade/ |
Budgeted multi-layer evaluation |
poq/, scoring/ |
PoQ engine and composite scoring (v1 + v2) |
training/ |
Judge training code and checkpoints/ |
config/ |
YAML configuration |
scripts/ |
Data prep, training, analysis, plotting |
tests/ |
Unit tests |
Key configs: config/judge_models.yaml, config/online_calibration.yaml, config/cascade.yaml, config/composite_scoring.yaml.
The file data/raw/cnn_dailymail/train.jsonl (~1.2 GB) is excluded from git (GitHub 100 MB limit). Regenerate if needed:
python scripts/data/download_cnn_dm.pyProcessed subsets under data/processed/ are included for the evaluation pipelines.
Published at ANNTIAN/poq-judge-checkpoints with these files at the repo root:
cnn_judge_final.pt
minilm_judge_final.pt
deberta_judge_final.pt
cnn_tokenizer.json
Then set POQ4_CHECKPOINTS_REPO=ANNTIAN/poq-judge-checkpoints in .env (optional; this is the script default).
- Never commit
.envor API keys. - Scripts read
OPENAI_API_KEYfrom the environment (see.env.example). - If an API key was ever committed locally, rotate it in the OpenAI dashboard before pushing.
Use for research and reproduction. Respect licenses of upstream datasets (SQuAD, CNN/DailyMail, UltraFeedback, etc.) and third-party models.
Extended architecture and script index (Chinese): PROJECT_FULL_zh.md.
Legacy zip-oriented notes were removed; this README is the canonical setup guide for git clones.