XLaya (/xlaya) - 5-Profile Hardware-Adaptive System 1 → System 1.5 Semantic Triage, AST Code-Graph Sieve & Conformal Decision Sidecar
Gemma 4 Developer Agent Paper Track: Official Research Writeup on Kaggle (
Overall Best Paper/Best New Resource/Best New Application) • Repository:iscvlab/xlaya(v2.3.0, coreMIT License;benchmarks/data/ple_dev_29_metadata.json& generated ADK exportsApache-2.0).
XLaya (iscvlab/xlaya, slash command /xlaya, legacy alias /laya) is a 100% pure-local, non-autoregressive 5-Profile System 1 → System 1.5 Semantic Decision, Regime-Invariant AST Code-Graph Sieve, 2-Stage Tool-Schema Gate, Joint NLI Contradiction Triad, Thorough Scan & Conformal Triage Sidecar (v2.3.0) engineered by ISCV | Isocurve Research Lab (@iscvlab). Powered by laya 0.4.31+ (ModernBERT-large en, subfolder="typed-decisions", calibrated 0.45*en + 0.55*typed ensemble & mmBERT, output_tokens: 0), vega-U (vegaml 0.16.1+, 0.8b / 4b / gemma-1b / 27b), and a 100% repo-disjoint CUDA-evolved 8-feature AST + call-graph + Corpus-IDF simplex sieve (policies/sieve_genomes.json, 5,679 B LF / 6,020 B CRLF), it runs across --profile auto|nano|edge|workstation|max and --agent-tier auto|e4b-sniper|31b-deep as a warm resident JSON-RPC 2.0 stdio MCP server (xlaya_mcp_server.py, 6 tools, ~30 ms warm inference), a standalone Python CLI (laya_triage.py), and a portable AI Agent Skill (skills/xlaya/SKILL.md) across any AI coding agent — cloud, IDE, CLI, or local LLM (Claude, Grok, Gemini / Antigravity, Cursor, Codex, Ollama, OpenHands, Cline, Google ADK, etc.).
-
New in
v2.3.0— Regime-Invariant 8D Code-Graph Sieve, Zero-Cheat Disjoint Evolution & 2-Stage Tool Gate (1.28MGenomes in74.87 s):- Zero-Torch / Zero-NumPy
--profile nano(ΔRSS = 10.41 MB,3.99 mswarm /255.33 ms/repocold,"torch" not in sys.modules): Partitioning repositories via a 2-Tier Source-First Lexical Pre-Ranker (Tier-1core package beforeTier-2docs_src/andtests/), attaching compact AST Class/Import Skeleton Headers (100%split-method coverage), and scoring an 8D AST + NetworkXnode_link_data+ All-but-the-Mean Top-4 PC Whitened.npz/ Corpus-IDF simplex (policies/sieve_genomes.json,5,679 BLF /6,020 BCRLF) with Smooth Realized Channel Availability (M_avail,gamma_max = 1.45). - 100% Zero-Shot Blind SWE & Tool-Gate Verification: Evolved strictly on
125repo-disjoint@base_committasks (21repos,80.00%E4B/85.71%31Bval task hit rate,Top-1 ECE = 0.0654), XLaya v2.3 achieves65.12%E4B Recall@3and78.29%31B Recall@8(94.59%named/traceback,71.74%symptom-only,80.60%fastapi,Top-1 ECE = 0.0568,|ΔS| = 0.0041acrossR1/R2) on the official129-task Kaggle suite,64.89%R@3/79.79%R@8(Top-1 ECE = 0.0155) on blindSWE-bench_Verified(n = 94), and81.37%Val /71.69%Test MCQ binary conformal accuracy (Top-1 ECE = 0.0279 / 0.0777,1.11 ms/row) onnvidia/When2Call(triage_tool_call). - Fitted
xerj#1108&PLE 29Cross-Domain Verification: Achieves100.0%blind accuracy (48/48) and0.00%Wrong_Confident(p ≥ 0.80)onxerj#1108(120adversarial boundary emails) and86.21%(25/29) zero-shot cross-domain triage on Google'sPhysical Engineering's Last Exam (PLE)dev metadata.
- Zero-Torch / Zero-NumPy
-
Zero-Token System 1 → System 1.5 Cascade (
output_tokens: 0): Prunes99.35%(152.7×) of low-signal repository tokens before expensive System 2 LLM reading (173,666→1,137tokens in170.3 ms), cutting cold 32-chunk CLI sweeps by7.92×(74.51 s→9.41 s) and reaching29.99 mssingle-state /662.5 ms32-chunk (96-decision) warm latency via the resident MCP server. -
Batched Cross-Lingual Determinism (
859.1 q/s,0 / 432Flips): Length-sortedpredict_batchaccelerates mixed50% EN / 50% HIrouting by8.9×(837.9 q/s) andmmBERTmultilingual batches to859.1 q/s(1.16 ms/question,5.9×atB = 8; up to6.5×atB = 16) with0discrete label flips across432paired comparisons and5 / 5(100%) routing acrossEN,DE,HI,JA, andTH. -
Calibrated Contradiction, Ensemble Watcher Gate & Persistent Conformal Policies: Combines a 3-Question Joint NLI Contradiction Triad (
8 / 8 = 100%counter-anchor accuracy,16.6×OOD conflict separation,0false positives), a calibrated0.45*en + 0.55*typedStage 1 ensemble withdisagreement ≥ 0.15gate (0.00%wrong-and-confident fires vs24.49%baseline FAIL on XERJ#1108),views=1|2|3multi-view debiasing,--thorough4-facet decomposed rubric scans, local multimodal (image/audio/video) evaluation, andxlaya_fitpersistentTTTHead+ split-conformal policies (90.0%held-outfolds=5accuracy /85.0%folds=4evolution gate acrossrouting_labels.csvafter a7.58 s40-row fit). -
Which
/xlayaExecution Tier Should You Choose?- Stage 0 — Pure-Python Structural & Lexical Sieve (
0.15 ms/chunk,6,841 chunks/s): Best for zero-dependency CI environments,--lexical-prefiltercandidate pruning across1,000+chunks, and exact 4-backtick fence-safe#L<start>-L<end>breadcrumb chunking across.md,.html,.txt,.json,.jsonl, and.csv. - Stage 1 — Resident MCP
Laya(en,typed,ensemble,multi,~30 mswarm /859 q/s): Default daily driver forshortlist,contradiction(Joint NLI Triad),rubric(0..4or--thorough4-facetspecificity/completeness/consistency/actionability+file_summaries),classify(includingpredict_tournamenton>20labels), and bilingual Thai/CJK/multilingual triage. - Stage 2 —
vega-U(auto/0.8B/4B/gemma-1b/27B) Conformal Verifier & Multimodal Engine (up to73,728tokens &90.0%5-fold/85.0%4-foldfitted accuracy): Opt-in--vega/vega: trueescalation withviews=1|2|3multi-view debiasing, persistentpolicy_path(TTTHead+ split-conformal + OODoutboundguard), whole-file long-document audits (--whole-file), and localimage/audio/videodecisions.
- Stage 0 — Pure-Python Structural & Lexical Sieve (
One-Prompt Universal Agent Install — Copy and paste into any AI coding agent — cloud, IDE, CLI, or local LLM (
Claude,Grok,Gemini,Cursor,Codex,Ollama,OpenHands,Cline, etc.):Clone https://github.com/iscvlab/xlaya and install skills/xlaya into my active agent skills directory (Windows: ./scripts/install.ps1 -All; macOS/Linux: ./scripts/install.sh --all; or copy skills/xlaya into ~/.gemini/config/skills/xlaya, ~/.claude/skills/xlaya, ~/.codex/skills/xlaya, or ~/.agents/skills/xlaya), then run python benchmarks/run_xlaya_benchmark.py --mode quick to verify /xlaya is ready.
| Evaluation Dimension | Configuration / Engine | Visual Comparison Bar | Measured Result & Delta vs. Baseline |
|---|---|---|---|
| 1. Cold CLI Boot Speed ↑ (32-Chunk Contradiction Sweep) |
xlaya v2.3.0 (predict_batch + Local Snap)xlaya v1.0 Sequential CLI Baseline |
████████████████████ 7.92× Faster███░░░░░░░░░░░░░░░░░ 1.0× Baseline |
9.41 s (8,459.7 ms engine) ✨74.51 s (-87.4% cold wall time) |
| 2. Mixed-Language Batch Speed ↑ (50% EN / 50% HI, 32 States × 3 Qs) |
Router.predict_batch (sort_by_length)Sequential predict() Loop |
████████████████████ 8.9× Faster██░░░░░░░░░░░░░░░░░░ 1.0× Loop |
114.6 ms (837.9 q/s, 0 flips) ✨1,016.2 ms (279.3 states/s batched) |
| 3. Peak Encoder Throughput ↑ ( 8 States × 4 Questions/State) |
mmBERT Multilingual BatchedModernBERT-large English BatchedSequential mmBERT Loop |
████████████████████ 859.1 q/s (5.9×)████████░░░░░░░░░░░░ 329.8 q/s (2.7×)███░░░░░░░░░░░░░░░░░ 146.5 q/s (1.0×) |
1.16 ms/q (37.2 ms vs 218.5 ms) ✨3.03 ms/q (97.0 ms vs 263.8 ms)Unbatched ( 6.5× speedup at B = 16) |
| 4. Few-Shot Conformal Accuracy ↑ ( routing_labels.csv Held-Out N=20) |
vega-U 0.8B Fitted (40 rows, 7.58 s)laya English Zero-Shotlaya-typed-decisions Zero-ShotCold vega-U 0.8B Zero-Shot |
██████████████████░░ 90.0% (18/20)████████████████░░░░ 80.0% (16/20)███████████████░░░░░ 75.0% (15/20)████████████░░░░░░░░ 60.0% (12/20) |
+30.0 pts over cold (ECE 0.214) ✨31.73 ms median zero-shot25.75 ms median (ECE 0.188)64.36 ms (auto: 8, review: 12) |
| 5. Contradiction & Watcher Gate ↑ (NLI Triad + xerj-org/xerj#1108) |
/xlaya 3-Question NLI TriadXERJ #1108 Gate (100% - FalseConf)XERJ #1108 Baseline (24/98 FAIL) |
████████████████████ 100.0% (8/8)████████████████████ 100.0% (0/30 False)███████████████░░░░░ 75.51% (24.49% FAIL) |
16.6× OOD separation (0 FP) ✨0.00% False-Confident (80% ens.) ✨24 / 98 (24.49%) wrong at p ≥ 0.80 |
| 6. Pass-B Token Compression ↑ ( 27 Specs / 1,165 Chunks) |
/xlaya Top-5 Section ShortlistFull-Corpus Unfiltered Read |
████████████████████ 99.35% Saved (152.7×)░░░░░░░░░░░░░░░░░░░░ 0.0% Saved (1.0×) |
1,137 tok in 170.3 ms (6.8k/s) ✨173,666 tok across 27 specifications |
%%{init: {'theme': 'neutral'}}%%
flowchart LR
subgraph Canvas [" "]
In["Input Corpus or Media (.md .html .txt .json .jsonl .csv .png .wav .mp4)"] --> S0["Stage 0: Fence-Safe Sieve + BM25 and Thai 3-Gram"]
subgraph Stage1 ["Stage 1: Laya v0.4.31 System 1 (output_tokens: 0)"]
Guard{"Head-Language Guard"}
EN["ModernBERT-large (en / typed / ensemble, 512 tok)"]
ML["mmBERT Multilingual (up to 8192 tok)"]
Triad["Joint NLI Triad + 4-Facet Rubric + Disagreement Gate (>= 0.15)"]
end
S0 --> Guard
Guard -->|"EN State and EN Head"| EN
Guard -->|"Non-EN State or Head"| ML
EN --> Triad
ML --> Triad
Triad -->|"Borderline, Disagreement >= 0.15, or Thorough"| S2["Stage 2: vega-U (auto / 0.8b / 4b / 27b) + Views 1..3 + Policy TTTHead"]
Triad -->|"Calibrated Spans + file_summaries"| Out["Exact file#Lstart-Lend + Conformal Route"]
S2 -->|"prediction_set + abstain + outbound + route"| Out
end
classDef default fill:#ffffff,stroke:#475569,color:#0f172a,stroke-width:1px
style Canvas fill:#ffffff,stroke:#cbd5e1,stroke-width:1.5px,color:#0f172a
style Stage1 fill:#f8fafc,stroke:#94a3b8,color:#0f172a
%%{init: {'theme': 'neutral'}}%%
flowchart LR
subgraph Canvas [" "]
subgraph Hosts ["1. Entry Points and Chained /x* Skills"]
Chain["Chained Skills (/xsync /xbrief /xdeep-research /xpublish)"]
Direct["Standalone Agent (/xlaya) or CI Pipeline"]
end
subgraph Paths ["2. Dual Execution Engine + Policy Fit"]
MCP["Tier 1: xlaya_mcp_server.py (6 Tools, ~30 ms Warm + Idle Unload)"]
CLI["Tier 2: laya_triage.py (7.92x Faster Batch CLI + --thorough)"]
Fit["Tier 3: xlaya_fit / --fit-rows Persistent TTTHead + Conformal Policy"]
end
subgraph Actions ["3. System 2 Surgical Action"]
Skip["Skip Clean Chunks (Zero Token Bloat)"]
Edit["Surgical read_file and edit_file on Flagged Spans"]
end
Chain --> MCP
Direct --> MCP
Direct --> CLI
Direct --> Fit
Fit --> MCP
Fit --> CLI
MCP --> Skip
MCP --> Edit
CLI --> Edit
end
classDef default fill:#ffffff,stroke:#475569,color:#0f172a,stroke-width:1px
style Canvas fill:#ffffff,stroke:#cbd5e1,stroke-width:1.5px,color:#0f172a
style Hosts fill:#f8fafc,stroke:#94a3b8,color:#0f172a
style Paths fill:#f8fafc,stroke:#94a3b8,color:#0f172a
style Actions fill:#f8fafc,stroke:#94a3b8,color:#0f172a
Figure 1: Dual-Channel Regime-Invariant Architecture separating Host sidecar enrichment (R1) and Container-A pure-stdlib execution (R2) with bounded cross-regime drift (|ΔS| = 0.0041).
Figure 2: Strict zero-overlap evolution workflow across 125 disjoint training/validation tasks (15 unique blind repos blocked by startup AssertionError) and blind evaluation suites.
-
Adaptive 5-Profile Hardware Cascade (
--profile nano|edge|workstation|max|auto) & Regime-Invariant 8D Code-Graph Simplex (sieve_genomes.json,5,679 BLF /6,020 BCRLF):- In
--profile nano, runs zero PyTorch and zero NumPy imports (10.414 MBΔRSS,3.99 mswarm /255.33 ms/repocold), partitioning.pyrepositories via a 2-Tier Source-First Pre-Ranker, attaching compact ASTskeleton_headers on split methods, and scoring an 8-feature AST + call-graph + Corpus-IDF / whitened.npzsimplex with Smooth Channel Availability (M_avail,gamma_max = 1.45) across dual SWE tiers (e4b-sniperK=3/≤ 800tok and31b-deep3Anchor +5Conditional Residual slots /≤ 4,096tok), achieving65.12%E4B Recall@3and78.29%31B Recall@8(Top-1 ECE = 0.0568) on blindKaggle 129,79.79%Recall@8(Top-1 ECE = 0.0155) on blindSWE-bench_Verified, and81.37%Val /71.69%MCQ binary conformal accuracy onnvidia/When2Call(triage_tool_call).
- In
-
Stage 1
typed(subfolder="typed-decisions") & Calibratedensemble+ Inter-Model Disagreement Gate:- Exposes
lang="typed"(ECE = 0.188) andlang="ensemble"(0.45 × P_en + 0.55 × P_typed), achieving0.00%false-confident fires (80.0%accuracy) on the XERJ#1108Watcher Gate while computing inter-modeldisagreement = max_k |P_en(k) − P_typed(k)|and automatically flagging uncertain chunks (disagreement ≥ 0.15) for Stage 2 escalation.
- Exposes
-
Stage 2 Hardware-Aware
vega_modelAuto-Tiering (auto/0.8b/4b/gemma-1b/27b) +views=1|2|3Multi-View Debiasing:vega_model="auto"selects the largest localvega-Ucheckpoint that fits comfortably in free GPU VRAM (4bon≥ 10 GBfree VRAM,0.8botherwise, up to27bfor high-VRAM hosts and73,728tokens). Multi-view debiasing (views=2option-order reversal;views=3reversal + prose template paraphrase) cancels positional and phrasing bias, returningview_agreementand forcingroute = "review"on view disagreement.
-
--thoroughFull-Document Scan & 4-Facet Decomposed Rubric Scoring (file_summaries):- Evaluates chunks across 4 orthogonal rubric facets (
specificity,completeness,consistency,actionability) with domain presets (--rubric-preset default|architecture|code_audit|doc_sync|research), emits file-level sliding-windowfile_summaries(mean_score,min_score,weakest_section,contradiction_hotspots,facet_breakdown), runs Stage 1ensemble+disagreement ≥ 0.15gating, and auto-escalates top/uncertain candidates to Stage 2 (views=3).
- Evaluates chunks across 4 orthogonal rubric facets (
-
xlaya_fit/--fit-rowsPersistentTTTHead+ Split-Conformal Policy Calibration (policy_path&outboundOOD Guard):- Fits ridge-regression
TTTHeadweights (heads.npz), out-of-fold split-conformal thresholds (calibration.json), OOD feature centroids (outboundguard), and question schemas (questions.json) in< 8 s(60.0%→90.0%held-out accuracy onrouting_labels.csv), reloadable across CLI and MCP invocations viapolicy_path.
- Fits ridge-regression
-
Local Multimodal Decision Support (
image,audio,video) inxlaya_predict:- Evaluates local images via
VegaVision, audio via local ASR + decision heads, and video via speech/contact-sheet inspection without requiring textstate/states.
- Evaluates local images via
-
4-Backtick Fence-Safe Chunker, Head-Language Guard, 3-Question Joint NLI Triad (
P_eff) & Tournament Elimination (>20Labels):- Preserves 6-level CommonMark breadcrumbs across
.md,.html,.txt,.json,.jsonl, and.csv; enforces the Head-Language Guard so non-English queries/heads route100%tommBERT; evaluatesP_eff = P(is_relevant) × P(contradicts_ssot) × (1 − P(is_consistent))(8 / 8 = 100%counter-anchor accuracy); and auto-runslaya.predict_tournament(..., group_size=16)on>20choice labels.
- Preserves 6-level CommonMark breadcrumbs across
Full methodology and extended telemetry are documented in benchmarks/XLAYA_V2_BENCHMARK_REPORT.md and benchmarks/xlaya_v2_benchmark_results.json (reproducible via benchmarks/run_xlaya_benchmark.py).
Execution Path / Model (RTX 5060 Ti 16 GB) |
Batch / Workload | Batched / v2 Latency ↓ | Baseline / Loop ↓ | Speedup ↑ | Throughput ↑ |
|---|---|---|---|---|---|
Cold CLI Sweep (laya_triage.py) |
32 chunks (batch_size=32) |
9.41 s (8,459.7 ms engine) |
74.51 s (v1.0 CLI) |
7.92× ✨ |
3.40 chunks/s |
Resident Warm MCP (xlaya_mcp_server.py) |
1 state / 32-chunk Triad (96 Qs) |
29.99 ms / 662.49 ms |
74,510 ms (v1.0 cold) |
112.5× ✨ |
144.9 q/s (6.90 ms/q) |
| Stage 0 Pure-Python Sieve | 27 specs / 1,165 chunks |
170.29 ms (0.15 ms/chunk) |
173,666 full tokens |
152.7× tokens |
6,841.3 chunks/s |
english (ModernBERT-large, 4 Q/state) |
1 / 8 / 64 states |
26.4 / 97.0 / 853.8 ms |
39.5 (T4) / 263.8 / 2,119.4 ms |
2.7×–2.8× |
329.8 q/s (3.03 ms/q) |
multilingual (mmBERT-base, 4 Q/state) |
1 / 8 / 16 states |
20.7 / 37.2 / 80.1 ms |
32.8 (T4) / 218.5 / 522.4 ms |
5.9×–6.5× ✨ |
859.1 q/s (1.16 ms/q) ✨ |
Router.predict_batch (50% EN / 50% HI, 3 Q/s) |
32 / 64 states |
114.6 ms / 236.4 ms |
1,016.2 ms / 1,950.6 ms |
8.9× / 8.3× ✨ |
837.9 q/s (279.3 states/s) |
LaTeX (booktabs) Source for Academic Citations — Axis 1 Latency & Batch Throughput
\begin{table}[t]
\centering
\caption{Empirical latency and batched throughput of \textsc{XLaya v2.1.0} (\texttt{laya 0.4.31}, \texttt{vegaml 0.16.1}) on an NVIDIA GeForce RTX 5060 Ti (16\,GB VRAM, \texttt{PyTorch 2.13.0+cu130}). All forward passes emit zero autoregressive tokens (\texttt{output\_tokens = 0}).}
\label{tab:xlaya_axis1_throughput}
\begin{tabular}{llrrrr}
\toprule
\textbf{Engine / Backbone} & \textbf{Workload} & \textbf{Batched (ms)} $\downarrow$ & \textbf{Baseline / Loop (ms)} $\downarrow$ & \textbf{Speedup} $\uparrow$ & \textbf{Throughput} $\uparrow$ \\
\midrule
Cold CLI (\texttt{laya\_triage.py}) & 32 chunks & \textbf{9,410.0} & 74,510.0 (\texttt{v1.0}) & \textbf{7.92$\times$} & 3.4 chunks/s \\
Warm Resident MCP (\texttt{xlaya\_predict}) & 1 state / 32-chunk Triad & \textbf{29.99 / 662.49} & 74,510.0 (\texttt{v1.0}) & \textbf{112.5$\times$} & 144.9 q/s \\
Stage 0 Pure-Python Sieve & 27 specs (1,165 chunks) & \textbf{170.29} & --- & 152.7$\times$ tok & \textbf{6,841.3 chunks/s} \\
\texttt{ModernBERT-large} (\texttt{english}) & $B=8 \times 4$ questions & \textbf{97.0} & 263.8 (loop) & 2.7$\times$ & 329.8 q/s \\
\texttt{mmBERT-base} (\texttt{multilingual}) & $B=8 \times 4$ questions & \textbf{37.2} & 218.5 (loop) & 5.9$\times$ (6.5$\times$ at $B=16$) & \textbf{859.1 q/s} \\
\texttt{Router.predict\_batch} (50\% EN / 50\% HI) & $B=32 \times 3$ questions & \textbf{114.6} & 1,016.2 (loop) & \textbf{8.9$\times$} & \textbf{837.9 q/s} \\
\bottomrule
\end{tabular}
\end{table}- Zero Discrete Label Flips (
0 / 432): Across144english(max_abs_dp = 0.00580),144multilingual(max_abs_dp = 0.00260), and144Router.predict_batch(max_abs_dp = 0.00650) paired evaluations, dynamic length-sorted batching produced0label or routing flips. 5 / 5 (100%)Cross-Lingual Routing & 5 Built-In Presets: Verified100%language routing accuracy acrossEN,DE,HI,JA, andTH, plus sub-50 ms multi-question preset execution:
| Preset / Check | Latency ↓ | Questions / Scope | Verified Telemetry Receipt |
|---|---|---|---|
Cross-Lingual Router (EN/DE/HI/JA/TH) |
20.7–26.4 ms |
5 languages |
5 / 5 (100%) route accuracy + Head-Language Guard protection ✓ |
moderation / email Presets |
27.10 ms / 32.71 ms |
5 questions each |
toxic=0.0 (clean) • is_phishing=0.984, is_spam=0.993 ✓ |
guard / router / triage Presets |
40.74 / 49.56 / 49.80 ms |
4–5 questions each |
prompt_injection=1.000 • domain="math_or_logic" • intent="cancellation" ✓ |
| Benchmark Suite & Evaluation Target | Baseline / Cold Reference | /xlaya v2.3.0 Calibrated Result |
Empirical Gain & Safety Receipt |
|---|---|---|---|
1. routing_labels.csv (40 Fit / 20 Test)(Conformal Few-Shot Calibration) |
Cold vega-U 0.8B: 60.0% acc (auto: 8, review: 12)laya-typed-decisions: 75.0% (25.75 ms, ECE 0.188)laya english: 80.0% (31.73 ms, ECE 0.283) |
vega-U 0.8B Fitted (40 rows in 7.58 s, folds=5):90.0% (18/20) team acc, ECE = 0.214, urgency_mae = 0.650 (103.36 ms) (evol. regression gate folds=4: 85.0%) |
+30.0 pts over cold vega-U & +10.0 pts over zero-shot laya in < 8 s fit time ✨ |
2. 3-Question Joint NLI Triad (P_eff)(Counter-Anchor Contradiction Suite) |
1-question noul suffers off-topic & verbatim-agreement false positives |
8 / 8 (100.0%) accuracy, 0 false positivesdirect=0.6735, numeric=0.6806, ood_pgbouncer=0.7553 vs 0.0456 consistent, 73k_vega=0.6569 vs 0.0412 |
16.6× OOD conflict-vs-consistent separation + 73,728-token whole-file support ✨ |
3. xerj-org/xerj#1108 Watcher Gate( 120 Emails, Wrong-and-Confident p ≥ 0.80) |
Uncalibrated Baseline: 24 / 98 (24.49% FAIL) wrong-and-confident fires |
Standalone Gate: 0 / 30 (0.00% PASS)Agreement Gate: 2 / 57 (3.51% PASS)Ensemble: 0 / 36 (0.00% PASS, 96/120 = 80.0% acc) |
-100% elimination of wrong-and-confident fires (24.49% → 0.00%) ✨ |
LaTeX (booktabs) Source for Academic Citations — Axis 3 Conformal Calibration & Contradiction Gates
\begin{table}[t]
\centering
\caption{Conformal calibration on \texttt{routing\_labels.csv} ($N_{\text{fit}}=40$, $N_{\text{test}}=20$) and boundary-gate verification on \texttt{xerj-org/xerj\#1108} ($N=120$ emails, confident threshold $p \ge 0.80$).}
\label{tab:xlaya_axis3_calibration}
\begin{tabular}{llrrrr}
\toprule
\textbf{Suite} & \textbf{Configuration} & \textbf{Accuracy} $\uparrow$ & \textbf{ECE / Rate} $\downarrow$ & \textbf{Urg.\ MAE} $\downarrow$ & \textbf{Median (ms)} $\downarrow$ \\
\midrule
\texttt{routing\_labels} & \texttt{laya-typed-decisions} (0-shot) & 75.0\% (15/20) & \textbf{0.188} & 0.671 & \textbf{25.75} \\
\texttt{routing\_labels} & \texttt{laya} \texttt{english} (0-shot) & 80.0\% (16/20) & 0.283 & 0.780 & 31.73 \\
\texttt{routing\_labels} & \texttt{vega-U 0.8B} (Cold 0-shot) & 60.0\% (12/20) & 0.222 & \textbf{0.650} & 64.36 \\
\texttt{routing\_labels} & \textbf{\texttt{vega-U 0.8B} (Fitted 40 rows, 7.58\,s)} & \textbf{90.0\% (18/20)} & 0.214 & \textbf{0.650} & 103.36 \\
\midrule
XERJ \texttt{\#1108} & Uncalibrated Baseline (FAIL) & --- & 24.49\% (24/98) & --- & --- \\
XERJ \texttt{\#1108} & \textbf{\textsc{XLaya} Standalone / Ensemble (PASS)} & \textbf{80.0\% (96/120)} & \textbf{0.00\% (0/30, 0/36)} & --- & \textbf{29.99} \\
\bottomrule
\end{tabular}
\end{table}Axis 4: Zero-Shot Blind SWE (Kaggle 129, SWE-bench_Verified), nvidia/When2Call, PLE 29 & Container-A ADK
Figure 3: 100% zero-shot blind evaluation across official Kaggle 129 (R1 vs. R2), SWE-bench_Verified (n = 94), and nvidia/When2Call (n = 3,952).
Figure 4: Stage-A token budget Pareto frontier (left: 699.9 E4B tokens vs. 46,231.4 Top-3 full .py files vs. 32,768 vLLM wall) and 2-hop code-graph + 4-PC .npz whitening topology (right).
- Zero-Cheat Disjoint Evolution (
125Tasks /21Repos): StartupAssertionError(FORBIDDEN_BLIND_EVAL_REPOS) blocks all15unique blind evaluation repos (4Kaggle +12SWE-bench_Verified,psf/requestsshared). Running5,000gens ×256pop (1.28Mevaluations in74.87 son RTX 5060 Ti) lifts7-repo unseen validation task hit rate from45.71%(BM25 K=3) to80.00%(e4b_sniper_distilled, fileR@3 = 75.95%,MRR = 0.6476) and85.71%(31b_deep, fileR@8 = 83.10%,MRR = 0.6512,Top-1 ECE = 0.0654). - Official Blind
Kaggle 129(R1Sidecars vs.R2Pure Stdlib):
Benchmark / Regime (Kaggle 129 Blind) |
E4B R@3 ↑ | 31B R@8 ↑ | Pos MRR (E4B / 31B) ↑ | Test-Echo (E4B / 31B) ↓ | Top-1 ECE ↓ |
|---|---|---|---|---|---|
BM25-Only Baseline (w/ Pre-Ranker) |
37.21% |
51.16% |
— / 0.2381 (MRR@8) |
10.34% / — (Echo@3) |
— |
XLaya v2.2.0 (Pre-Remediation) |
31.01% |
35.66% |
0.2868 / 0.2941 |
1.81% / 1.07% |
0.0705 (82.2% abstain) |
XLaya v2.3.0 Regime R2 (Pure Stdlib) |
62.79% |
78.29% |
0.4871 / 0.4954 |
1.81% / 2.23% |
0.0641 |
XLaya v2.3.0 Regime R1 (JSON + .npz) |
65.12% |
78.29% |
0.4987 / 0.4999 |
1.55% / 1.36% |
0.0568 |
- Strata, Latency & Invariance (
R1):Named/Symbol/Traceback(n = 37) reaches89.19%E4BR@3/94.59%31BR@8;Symptom-Only(n = 92) reaches55.43%E4BR@3/71.74%31BR@8(vs 29.35% / 43.48% BM25); per-repo:fastapi/fastapi(n = 67)61.19% / 80.60%,Textualize/rich(n = 48)62.50% / 70.83%,psf/requests(n = 13)92.31% / 92.31%,encode/httpx(n = 1)100.0%. Mean Stage-A E4B payload is699.9tokens (max = 800,-98.49%/66.1×vs Top-3.py46,231.4tok); uncontended cold sieve latency is255.33 ms/repo(0/129core gold files truncated);4-PC.npzwhitening shifts127/127archives from median cos0.8518to-0.0158([-0.1136, +0.0959]) with cross-regime drift|ΔS| = 0.0041(|ΔECE| = 0.0073); OOD negative SWE probes (n = 25) abstain at80.0%(0.0%wrong-confident atp ≥ 0.80).
- External Blind
SWE-bench_Verified,nvidia/When2Call,PLE 29& Container-A ADK:
| Suite / Artifact Target | Split / Metric | Baseline / v2.2.0 | XLaya v2.3.0 Verified | Safety, ECE & Container Proof |
|---|---|---|---|---|
SWE-bench_Verified (n = 94) |
E4B R@3 / 31B R@8 |
48.94% / 64.89% (BM25 w/ Pre-Ranker) |
64.89% / 79.79% |
Top-1 ECE = 0.0155 (vs 0.2910 v2.2.0) • sympy (n=74): 79.73% (365.4 ms) |
SWE-bench_Verified Strata |
Symptom (41) / Named (53) R@8 |
14.63% / 35.85% (v2.2.0) |
73.17% / 84.91% |
SpanIoU > 0.30 = 40.43% (31B) • 2,031 mean tok |
nvidia/When2Call Val (n = 1,616) |
BinAcc (ToolRec / Abstain) |
62.75% (0% / 100%) |
81.37% (81.40% / 81.36%) |
Top-1 ECE = 0.0279 • Zero PyTorch / NumPy |
When2Call Blind MCQ (n = 3,652) |
BinAcc (ToolRec / Abstain) |
64.54% (0% / 100%, ECE 0.3462) |
71.69% (69.11% / 73.10%) |
Top-1 ECE = 0.0777 • 1.11 ms/row (4,045.4 ms total) |
When2Call LLM-Judge (n = 300) |
BinAcc (ToolRec / Abstain) |
66.67% (0% / 100%, ECE 0.3187) |
68.67% (68.00% / 69.00%) |
10-Bin ECE = 0.1210 • CannotAns = 79%, ReqInfo = 59% • 1.41 ms/row |
PLE 29 (Apache-2.0 Dev Meta) |
nano / ensemble 0-Shot Acc |
58.62% (nano) |
86.21% (25/29) |
benchmarks/data/ple_dev_29_metadata.json (29 tasks) |
Container-A ADK (self-test) |
Dual ZIPs / Subprocess Parity | Raw > 32k context |
83,695 B (31B) / 83,384 B (E4B) |
867 ch stdout (≤ 4,500), 0.0 diff vs Host, workspace_clean: true |
# Stage 1 (Laya ModernBERT-large en/typed/ensemble + mmBERT) & Stage 2 (VegaML conformal vega-U / VegaVision):
pip install "laya>=0.4.31" "vegaml>=0.16.1" torchAdd xlaya to your MCP client configuration (mcp_config.json, Claude Desktop, Cursor, Antigravity, Windsurf, or Cline):
{
"mcpServers": {
"xlaya": {
"command": "python",
"args": ["/path/to/xlaya/xlaya_mcp_server.py"],
"env": {
"LAYA_IDLE_TTL_SEC": "1800"
}
}
}
}| Resident MCP Tool | Core Parameters | Primary Use Case |
|---|---|---|
xlaya_triage |
paths, mode (shortlist|contradiction|rubric|classify), query, ssot, lang (auto|en|multi|typed|ensemble), profile (auto|nano|edge|workstation|max), agent_tier (auto|e4b-sniper|31b-deep), sieve_weights, thorough, rubric_preset (default|architecture|code_audit|doc_sync|research), views (1|2|3), policy_path, vega, vega_model (auto|0.8b|4b|gemma-1b|27b|none), lexical_prefilter |
End-to-end Stage 0 → Stage 1 → Stage 2 directory/file/Python-AST sweep returning #L<start>-L<end> spans, code-sieve probabilities, 4-facet scores, disagreement gate, and file_summaries |
xlaya_predict |
questions, state or states, model (auto|en|multi|typed|ensemble), profile (auto|nano|edge|workstation|max), vega, vega_model (auto|0.8b|4b|gemma-1b|27b|none), views (1|2|3), policy_path, image, audio, video |
In-memory, zero-torch nano, or local multimodal (image/audio/video) choice, score, and noul evaluation (auto-runs Head-Language Guard, views debiasing & predict_tournament) |
xlaya_fit |
rows (list[dict] or .csv/.jsonl path), questions, policy_path, vega_model="0.8b", device="auto", folds=5, calibrate=true, target_precision=0.95 |
Fits and persists a VegaAgent TTTHead + split-conformal policy (heads.npz, calibration.json, manifest.json, questions.json) with OOD outbound centroid guard |
xlaya_status |
profile="auto" (auto|nano|edge|workstation|max) |
Inspects warm-loaded models (en, multi, typed, ensemble, vegaml), active hardware profile & GPU/MPS/CPU device, MCP version (2.3.0), and LAYA_IDLE_TTL_SEC |
xlaya_preload |
models=["en", "multi", "typed", "ensemble"], vega=false, vega_model="auto", device="auto" |
Pre-warms checkpoints in RAM/VRAM ahead of latency-critical sweeps |
xlaya_unload |
(none) | Immediately evicts cached models and reclaims CPU RAM and CUDA VRAM |
# 0. Zero-Torch (< 25 MB RAM) AST + 2-Hop Code-Graph Localization for Gemma E4B Sniper vs 31B Deep:
python laya_triage.py ./src --mode shortlist --query "OAuth2PasswordBearer Depends SecurityScopes bug" --profile nano --agent-tier e4b-sniper
python laya_triage.py ./src --mode shortlist --query "AsyncClient transport redirect leak" --profile workstation --agent-tier 31b-deep
# 1. Thorough 4-Facet Decomposed Rubric Scan (ensemble + disagreement >= 0.15 + file_summaries + views=3):
python laya_triage.py ./docs --mode rubric --thorough --rubric-preset architecture --top-k 10
# 2. 3-Question Joint NLI Contradiction Sweep with Calibrated Stage 1 Ensemble (0.45*en + 0.55*typed):
python laya_triage.py . --mode contradiction --ssot "ModernBERT-large uses a 512-token context window and outputs 0 LLM tokens." --lang ensemble --threshold 0.60
# 3. Multi-View Debiased Stage 2 vega-U Conformal Escalation (views=3 cancels position & phrasing bias):
python laya_triage.py ./skills --mode rubric --ssot "Production specification with CLI and MCP schemas" --vega --vega-model auto --views 3 --vega-top-k 10
# 4. Fit a Persistent TTTHead + Split-Conformal Policy & Evaluate with OOD Outbound Guard (supply your questions JSON):
python laya_triage.py --fit-rows benchmarks/data/routing_labels.csv --questions-file /path/to/questions.json --policy-path ./policies/routing_v1
python laya_triage.py ./docs --mode classify --policy-path ./policies/routing_v1
# 5. Local Multimodal Evaluation, 3-Phase CUDA Sieve Evolution (requires local disjoint snapshot manifest) & Self-Tests:
python laya_triage.py --image docs/figures/xlaya_v23_cover_1120x560.png --questions-file /path/to/vision_questions.json --vega-model 4b
python laya_triage.py README.md --mode contradiction --ssot "XLaya v2.3.0 runs with zero output tokens." --whole-file --vega
python laya_triage.py --self-test
python benchmarks/colab_kaggle_l4x4_runner.py --self-test --build-submissions --out-dir dist_adk
# python benchmarks/run_xlaya_evolution.py --verify-zero-leakage --manifest <path/to/disjoint_swe_v23_manifest.json>
python benchmarks/run_xlaya_benchmark.py --mode quickInstall /xlaya into all detected agent skill directories on Windows, macOS, or Linux (automatically removing legacy /layax or /laya folders if present):
# Windows PowerShell (installs into ~/.gemini/config/skills, ~/.claude/skills, ~/.codex/skills, ~/.agents/skills):
.\scripts\install.ps1 -All# macOS / Linux (copy or --symlink into all detected agent skill roots, or --target <custom_dir>):
chmod +x scripts/install.sh
./scripts/install.sh --all| Agent Class / Host | Target Skill Directory | Activation Command | Compatibility |
|---|---|---|---|
Any AI Agent (Cloud • IDE • CLI • Local LLM) |
~/.*/skills/xlaya or <repo>/.agents/skills/xlaya |
/xlaya |
universal (zero-config <SKILL_DIR> resolution + pure-Python fallback) |
Google Gemini / Antigravity (IDE & agy CLI) |
~/.gemini/config/skills/xlaya |
/xlaya (chains with /xsync, /xbrief, /xplan, /xdeep-research, /xpublish, /xultra, /xlearn, /xocr) |
tested (Windows 11 + RTX 5060 Ti 16 GB + Resident MCP) |
| Claude Code / Desktop | ~/.claude/skills/xlaya |
/xlaya |
spec-compatible (SKILL.md + stdio MCP server + CLI) |
| Grok / Cursor / Codex / Windsurf / Cline / OpenHands | ~/.grok/skills/xlaya, ~/.codex/skills/xlaya, or ~/.agents/skills/xlaya |
/xlaya |
spec-compatible (SKILL.md + stdio MCP server + CLI) |
xlaya/
├── README.md # Flagship architecture, 5-profile & 4-axis benchmarks
├── LICENSE # MIT License + ConvAI Innovations upstream notice
├── llms.txt # Concise machine-readable entrypoint index for LLMs
├── laya_triage.py # Standalone 3-stage CLI, AST chunker & code-sieve engine
├── xlaya_mcp_server.py # Zero-extra-dependency JSON-RPC 2.0 stdio MCP server
├── docs/
│ └── figures/ # Academic White-Theme cover & Figures 1-4 (PNG)
│ ├── xlaya_v23_cover_1120x560.png
│ ├── fig1_system_architecture_flowchart.png
│ ├── fig2_zero_cheat_evolution_workflow.png
│ ├── fig3_blind_evaluation_scorecard.png
│ └── fig4_pareto_and_codegraph_topology.png
├── policies/
│ └── sieve_genomes.json # CUDA-evolved L1 simplex genomes + Platt & tool_gate (5,679 B LF / 6,020 B CRLF)
├── scripts/
│ ├── install.ps1 # Windows PowerShell multi-agent installer
│ └── install.sh # macOS / Linux POSIX bash installer (LF normalized)
├── skills/
│ └── xlaya/
│ ├── SKILL.md # Portable /xlaya agent skill specification
│ ├── resources/
│ │ └── sieve_genomes.json # Co-located evolved genome policy for standalone skill copy
│ └── scripts/ # SHA-256 mirrored CLI & MCP server for standalone skill copy
│ ├── laya_triage.py
│ └── xlaya_mcp_server.py
├── tests/
│ └── test_v23_sieve_and_adk.py # Deterministic pytest suite (G-01 through G-08, Container-A & ADK parity)
└── benchmarks/
├── data/ # Clean Apache-2.0 fixtures (routing_labels, xerj#1108, PLE 29)
├── XLAYA_V2_BENCHMARK_REPORT.md # Full RTX 5060 Ti technical benchmark report
├── xlaya_v2_benchmark_results.json # Self-documented Stage 0/1/2 JSON telemetry receipts
├── xlaya_v22_evolution_results.json # Legacy v2.2 evolution receipts
├── xlaya_v23_evolution_results.json # Zero-Cheat repo-disjoint v2.3 evolution & calibration receipts
├── run_xlaya_evolution.py # Vectorized 5,000-gen CUDA evolution & Funnel 2 verifier
├── colab_kaggle_l4x4_runner.py # Colab/Kaggle L4x4 verifier & dual ADK submission builder
└── run_xlaya_benchmark.py # Reproducible --mode {quick,full} verification harness
iscvlab/xboost: Composable 15-skill/x*AI agent engineering ecosystem, 7 Compound Learning Playbooks (4-Tier Cascade), and 10 Python CLI & MCP verification engines (13,200+LOC).iscvlab/xocr: High-speed Hybrid Document AI, Two-Stage LaTeXbooktabs& HTML Table Surgery, Thai PUA/Sara-Am normalizer, and localTyphoon-OCR v1.5×Baidu Unlimited-OCR-2engine.iscvlab/mcp-bridges: Model Context Protocol (MCP) automation bridges for QGIS, Rhino 3D, Grasshopper, Adobe Illustrator, Figma, and Ghost Mouse.iscvlab/sse-space-syntax: High-performance C++/Rust/Python Space Syntax, Axial/Segment Angular Choice, VGA, and Isovist simulation engine.
Engineered and maintained by Napat (Tan) Phasundhiae (@napattan) at ISCV | Isocurve Research Lab (@iscvlab). Released under the MIT License (benchmarks/data/ple_dev_29_metadata.json and ADK templates are Apache-2.0); Stage 1 and Stage 2 neural backends interoperate with ConvAI Innovations' open-source laya and vegaml libraries.