Stay on the Right Track: A Reasoning-Aware Dual-Agent Framework for Automated RTL Debugging

This repository contains the code, datasets, and model links to reproduce the experiments in the paper "Stay on the Right Track: A Reasoning-Aware Dual-Agent Framework for Automated RTL Debugging" (submitted to ICCAD 2026).

SoRT is a reasoning-aware dual-agent framework that explicitly supervises multi-step RTL debugging. It decomposes the task into two complementary roles: a generator, SoRT (π_gen), that produces diverse step-by-step debugging traces in parallel, and a Process Reward Model (PRM), SoRT (π_prm), that provides fine-grained, decision-level supervision to assess intermediate reasoning and select the most reliable path.

Before you start, download our fine-tuned models from HuggingFace:

git clone https://github.com/SEU-ACAL/SoRT
cd SoRT
pip install -e .

Requirements: Python >= 3.12. See requirements.txt for dependencies.

SoRT/
├── dataset/
│   ├── testset/              # Evaluation test sets (hd_eval, lik)
│   ├── testset_prm/          # PRM evaluation test set
│   └── trainset/             # Training data (download from huggingface and place it here)
├── scripts/
│   ├── run_example.py            # Quick inference demo
│   ├── run_eval.py               # Evaluation entry point
│   ├── eval_generator.sh         # Evaluate generator performance
│   ├── eval_prm.sh               # Evaluate PRM performance
│   ├── eval_downstream.sh        # Evaluate downstream methods
│   ├── compute_pass_at_k.py      # Compute pass@k from experiment results
│   └── check_data_leakage.py     # Train/test overlap check
├── src/
│   ├── configs/              # Model configs, prompts, paths
│   ├── prm/                  # PRM construction and utilities
│   ├── run/                  # Core evaluation logic
│   └── utils/                # Data processing utilities
├── .env.example              # API key template for commercial models
├── setup.py
├── requirements.txt
└── README.md

python scripts/run_example.py

The three shell scripts below run the full benchmark suite:

bash scripts/eval_generator.sh      # generator only, 20 candidates per case
bash scripts/eval_prm.sh            # PRM step-level correctness judgment
bash scripts/eval_downstream.sh     # end-to-end SoRT + downstream baselines

Or run individual evaluations via the CLI:

# Generator only
python scripts/run_eval.py --model SoRT_gen --testset hd_eval generator --num_samples 20

# PRM step-level accuracy
python scripts/run_eval.py prm --prm_model SoRT_prm --prm_testset testset

# End-to-end SoRT: generator + PRM
python scripts/run_eval.py --model SoRT_gen --testset hd_eval end2end --method model --prm_model SoRT_prm

Available test sets: hd_eval, lik

As a flexible evaluation framework, SoRT also allows users to plug in their own fine-tuned or commercial models via API as either the generator (--model) or the PRM/judge (--prm_model). The --method flag controls how the final answer is selected:

For example, --model gpt-4.1 --method model --prm_model SoRT_prm uses GPT-4.1 as generator scored by SoRT PRM.

API keys: Copy .env.example to .env and fill in your API keys, base URLs, and model names. SoRT-only evaluation does not need API keys.

Run python scripts/check_data_leakage.py to verify no test-set bugs appear verbatim in the training set (trainset on HuggingFace):

Run python scripts/compute_pass_at_k.py to reproduce the headline evaluation results in Table 5 of the paper (pass@1, pass@5, pass@20 per model on hd_eval, lik, and overall).

The table below reports the inference cost of the full SoRT system on a single NVIDIA A800 GPU (80 GB).

This project is licensed under the Apache License 2.0.