Computer-Use Agents as Judges for Generative User Interface
Kevin Qinghong Lin, Siyuan Hu, Linjie Li, Zhengyuan Yang, Lijuan Wang, Philip Torr, Mike Zheng Shou
Torr Vision Group @ Oxford University, Show Lab @ National University of Singapore, Microsoft

📄 Paper   |   🤗 HF Daily Paper   |   🤗 HF Demo   |   🌐 Project

What does Agent-friendly UI look like? Check out below demo:

The left UI is designed for 🧑🏻‍💻humans—prioritizing aesthetics.

The right UI is redesigned for 🤖agents—focused on clarity and functionality.

• [2025.11.20] Huggingface Demo is released.
• [2025.11.19] Arxiv paper is released.
• [2025.10.30] Code is released.

Can Computer-Use Agents offer feedback to assist Coders to Generate UI?

• Use Python 3.10+ in an isolated environment:

  conda create -n aui python=3.10
  conda activate aui

• Install dependencies and Playwright browsers:

  pip install -r requirements.txt
  python -m playwright install

• Local model servers are recommended (e.g., using VLLM).
• Edit configs/models.yaml to point to your endpoints:
  • Coder: Qwen3-Coder-30B (http://localhost:8001/v1)
  • CUA: UI-TARS-1.5-7B (http://localhost:8002/v1)
  • Verifier: GPT-5 or Qwen2.5-VL-72B
• Export API keys (if using proprietary models):

  export AZURE_OPENAI_API_KEY="YOUR_KEY"

Our AUI contains the following stages.

Pipeline:

• 0️⃣ Preparation: Generate initial websites and tasks per app.
• 1️⃣ Task Solvability Check: Judge extracts task-state rules on initial websites to determine task validity.
• 2️⃣ CUA Navigation Test: CUA executes supported tasks; oracle evaluation is rule-based.
• 3️⃣ Iterative Refinement:
  1. Revise: Update websites based on unsupported tasks (Task Solvability Feedback) and CUA failures (Navigation Feedback via Dashboard).
  2. ReJudge: Re-evaluate task solvability on revised websites.
  3. ReTest: CUA executes tasks on revised websites.

For normal usage, you only need the single entrypoint run.py from the repo root:

cd betterui_release
/users/husiyuan/miniconda3/envs/ui/bin/python run.py \
  --models gpt5,qwen,gpt4o \
  --apps all \
  --v0-dir full_52_apps \
  --experiment exp_integrated \
  --revision-type integrated \
  --cua-models uitars

This command sequentially runs Stage 0 → Stage 3. Advanced users who want full control over each stage can expand the section below to see what each stage does and the corresponding commands.

python src/stage0_generate_websites.py \
  --models gpt5,qwen,gpt4o \
  --apps all \
  --v0-dir full_52_apps

python src/stage0_generate_tasks.py \
  --apps all \
  --v0-dir full_52_apps

python src/stage1_judge_v0.py \
  --models gpt5,qwen,gpt4o \
  --apps all \
  --v0-dir full_52_apps

python src/stage2_cua_test_v0.py \
  --models gpt5,qwen,gpt4o \
  --apps all \
  --v0-dir full_52_apps \
  --cua-models uitars

# Re-Judge Task Solvability
python src/stage3_1_judge_v1.py \
  --experiment exp_integrated \
  --models gpt5,qwen,gpt4o \
  --apps all \
  --revision-type integrated \
  --v0-dir full_52_apps

# Re-Run CUA Navigation Test
python src/stage3_2_cua_test_v1.py \
  --experiment exp_integrated \
  --models gpt5,qwen,gpt4o \
  --apps all \
  --revision-type integrated \
  --cua-models uitars \
  --v0-dir full_52_apps

Initial Data (Stage 0-2)

v0/{v0_dir}/
  websites/{app}/{model}/index.html         # Initial Generated Websites
  tasks/{app}/
    tasks.json                              # Generated Tasks
    states/{model}/rules.json               # Stage 1: Validation Rules
    v0_cua_results/{model}/{cua_model}/     # Stage 2: CUA Results
      results.json
      trajectories/task_{i}/step_*.png|json # Trajectories

Experiments (Stage 3)

experiments/{experiment}/
  runs/{run_key}/
    stage3_0/{app}/{model}/v1_website/index.html    # Revised Websites
    stage3_1/{app}/{model}/rules.json               # Revised Rules
    stage3_2/{cua_model}/{app}/{model}/
      trajectories/{task_id}/                       # New Trajectories
      run_summary.json

1. Function Completeness (FC): Percentage of tasks that are functionally supported by the UI (determined by the Judge).
2. CUA Success Rate (SR): Percentage of valid tasks successfully completed by the CUA.

Key Components:

• Verifier: A GPT-5 based judge that extracts rule-based checks to validate task solvability.
• CUA Dashboard: A visual summary tool that compresses long interaction trajectories into a single image, highlighting key failure points for the Coder.
• Revision Strategies:
  • unsupported: Adds missing features for unsolvable tasks.
  • cua: Fixes usability issues preventing agent navigation (destylization, simplification).
  • integrated: Combines both for maximum performance.

If you find this project helpful, please consider citing our paper:

@misc{lin2025aui,
      title={Computer-Use Agents as Judges for Generative User Interface}, 
      author={Kevin Qinghong Lin and Siyuan Hu and Linjie Li and Zhengyuan Yang and Lijuan Wang and Philip Torr and Mike Zheng Shou},
      year={2025},
      eprint={2511.15567},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2511.15567}, 
}

• Apps are adapted from OpenAI's coding examples.
• Thanks to the open-source community for browser automation (Playwright) and agent tooling.