Skip to content
dgridaiPublic

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

3 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

POQ_4

Lightweight reference-free quality evaluation for decentralized LLM inference, with online dimension calibration.

POQ_4 extends the original PoQ framework with neural Judge models, online dimension calibration, and cascade evaluation. Use the web demo for interactive scoring, or run the experiment scripts for reproduction.


What is included in git

In repository Not in repository (download or train)
Source code, configs, tests Fine-tuned Judge weights (*.pt) — ANNTIAN/poq-judge-checkpoints
Processed eval data, evaluation metrics CNN/DailyMail train split (~1.2 GB raw)
Training/eval JSONL splits Hugging Face base encoders (auto-download on first use)
UltraFeedback HF cache under training/data/openbmb___ultra_feedback/

Base models referenced in config/judge_models.yaml (MiniLM, DeBERTa, Qwen) are fetched from Hugging Face Hub when needed. Only the fine-tuned Judge checkpoints must be downloaded separately or retrained locally.


Requirements

  • Python 3.10 (see environment.yaml)
  • ~5 GB disk space with checkpoints and dependencies
  • NVIDIA GPU optional (CPU works, slower)
  • OpenAI API key optional (GPT labeling / API judge only)

Quick start

1. Clone and enter the project

git clone <your-repo-url>
cd POQ_4

2. Environment

Conda (recommended for GPU):

conda env create -f environment.yaml
conda activate poq_v2
pip install -r requirements.txt

venv + pip:

python -m venv .venv
# Windows: .\.venv\Scripts\Activate.ps1
# Linux/macOS: source .venv/bin/activate
pip install -U pip
pip install -r requirements.txt

For GPU PyTorch, install the CUDA build from pytorch.org before the rest of requirements.txt if the default wheel does not match your system.

3. Secrets (optional)

cp .env.example .env
# Edit .env — only needed for OpenAI scripts or private HF downloads

4. Download fine-tuned Judge checkpoints

Checkpoints are hosted on Hugging Face (not in git):

https://huggingface.co/ANNTIAN/poq-judge-checkpoints

python scripts/download_checkpoints.py
# default repo: ANNTIAN/poq-judge-checkpoints

Or explicitly:

python scripts/download_checkpoints.py --repo ANNTIAN/poq-judge-checkpoints

Required files in training/checkpoints/:

  • cnn_judge_final.pt
  • minilm_judge_final.pt
  • deberta_judge_final.pt
  • cnn_tokenizer.json

Alternatives: copy those files manually into training/checkpoints/, or train from scratch:

python scripts/training/prepare_training_data.py --skip-gpt
python scripts/training/train_all_judges.py

5. Run the web demo

python run_frontend.py

Open http://127.0.0.1:8000 (use --port 8080 if 8000 is busy). Press Ctrl+C to stop.

Quick sanity check:

python -c "import torch; print(torch.__version__)"

Running experiments

Ensure evaluation records exist (the repo may include outputs/metrics/all_with_full_gpt.jsonl or related paths). Then:

python run_experiments.py --all

Run a single experiment:

python run_experiments.py --judge-comparison
python run_experiments.py --online-calibration
python run_experiments.py --cascade

Aggregate results for tables:

python collect_summary.py

Project layout

Path Purpose
api/, frontend/ FastAPI backend and static demo UI
judges/ CNN / MiniLM / DeBERTa / LLM judges
calibration/ Online weight updates, task detection, anchors
cascade/ Budgeted multi-layer evaluation
poq/, scoring/ PoQ engine and composite scoring (v1 + v2)
training/ Judge training code and checkpoints/
config/ YAML configuration
scripts/ Data prep, training, analysis, plotting
tests/ Unit tests

Key configs: config/judge_models.yaml, config/online_calibration.yaml, config/cascade.yaml, config/composite_scoring.yaml.

Optional: large raw CNN/DailyMail train split

The file data/raw/cnn_dailymail/train.jsonl (~1.2 GB) is excluded from git (GitHub 100 MB limit). Regenerate if needed:

python scripts/data/download_cnn_dm.py

Processed subsets under data/processed/ are included for the evaluation pipelines.


Publishing checkpoints to Hugging Face

Published at ANNTIAN/poq-judge-checkpoints with these files at the repo root:

cnn_judge_final.pt
minilm_judge_final.pt
deberta_judge_final.pt
cnn_tokenizer.json

Then set POQ4_CHECKPOINTS_REPO=ANNTIAN/poq-judge-checkpoints in .env (optional; this is the script default).


Security

  • Never commit .env or API keys.
  • Scripts read OPENAI_API_KEY from the environment (see .env.example).
  • If an API key was ever committed locally, rotate it in the OpenAI dashboard before pushing.

License and data

Use for research and reproduction. Respect licenses of upstream datasets (SQuAD, CNN/DailyMail, UltraFeedback, etc.) and third-party models.


Further reading

Extended architecture and script index (Chinese): PROJECT_FULL_zh.md.

Legacy zip-oriented notes were removed; this README is the canonical setup guide for git clones.

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages