ADRS

🎯 Can test-time training help LLMs make new discoveries? In our latest AI-Driven Research for Systems (ADRS) blog post, we feature TTT-Discover: a framework that keeps training an LLM on a single test problem instead of relying on a frozen model. Methods such as AlphaEvolve and OpenEvolve improve search by feeding past attempts back into increasingly informative prompts, while keeping the model weights fixed. TTT-Discover takes a different approach: it runs reinforcement learning directly on the test problem, using an objective designed for discovery rather than average performance. This raises a broader question for AI-driven discovery: If the model is generating useful new data while solving a problem, why only put that data back into the prompt instead of training on it? TTT-Discover does exactly that. Its entropic objective emphasizes the highest-reward rollouts, while a reuse strategy keeps building on the most promising intermediate solutions. 🔎 The results: - On the TriMul GPU kernel, TTT-Discover beat the top human submission across all four tested GPU types, including improving H100 runtime from 1,371 μs to 1,161 μs and A100 from 4,531 μs to 2,198 μs. - On two additive combinatorics problems previously tackled by AlphaEvolve, TTT-Discover set new records: 1.50287 on the first autocorrelation inequality and 0.380876 on Erdős’ minimum overlap problem. - Across kernel engineering, combinatorics, and algorithm engineering, the same recipe outperformed frozen-model search under the same model and rollout budget. Everything runs on the open gpt-oss-120b, for just a few hundred dollars per problem. TTT-Discover suggests a different way to think about inference-time search: when solving a genuinely out-of-distribution problem, the attempts generated along the way are not just context—they are new training data. Updating the model on that data can push discovery beyond what a frozen model can reach. ✍️ Read the blog: https://lnkd.in/gRQ857zB 📚 ADRS Blog Series: https://lnkd.in/gqtJ-GJ7 📄 TTT Paper: https://lnkd.in/gGpEQqMP 👩💻 TTT Code: https://lnkd.in/gdV6yGeh 🏠 Project Page: https://lnkd.in/ghguGWM8 🚀 Follow: https://lnkd.in/gCA_mFGG for updates! Special thanks to Mert Yuksekgonul, Daniel Koceja, Xinhao Li, Federico Bianchi, Jed McCaleb, Xiaolong Wang, Jan Kautz, Yejin Choi, James Zou, Carlos Guestrin, and Yu Sun for this work! 💫

🎯 Can we trust AI-evolved programs? In our latest AI-Driven Research for Systems (ADRS) blog post, we feature AIChilles: an automated framework for uncovering hidden weaknesses in AI-evolved systems. Frameworks such as AdaEvolve, OpenEvolve, and Engram can rewrite core systems algorithms and report substantial performance gains over human-designed heuristics. However, these programs are typically evaluated on a fixed set of workloads using scoring functions that may not capture every important system requirement. This raises a critical question for system operators: Should I actually deploy the evolved program, and what risks might be hidden by its benchmark score? AIChilles automatically searches for the “Achilles’ heel” of an AI-evolved program. It treats the original human-written program as a differential oracle and searches for valid workloads where the evolved program regresses in correctness, runtime, memory usage, or output quality. 🔎 The results: - 49 distinct hidden weaknesses across 5 systems, 3 evolution frameworks, 2 LLMs, and 30 evolved programs - Weaknesses depend on both the target application and the evolution framework - Integrating AIChilles into the evolution loop removes discovered weaknesses, but can substantially reduce reported score gains—for example, from 19% to 0% for Prism and from 49% to 3.9% for TXN AIChilles offers an important caution for AI-evolved systems: benchmark improvements do not necessarily translate to robust real-world performance. We must test for overfitting to evaluation workloads and make implicit system constraints explicit—otherwise, evolution may silently sacrifice properties that operators care about. ✍️ Read the blog: https://lnkd.in/gaJz2iWN 📚 ADRS Blog Series: https://lnkd.in/gqtJ-GJ7 📄 AIChilles Paper: https://lnkd.in/geEw8P9B 👩💻 AIChilles Code: https://lnkd.in/gnrYpiqT 🏠 Project Page: https://lnkd.in/giPbs7Hu 🚀 Follow: https://lnkd.in/gCA_mFGG for updates! Special thanks to Lesley (Yajie) Zhou, Ao Li, Ashwin Silla, Alan Zaoxing L., and Vyas Sekar for this work! 💫

🎯 How can we use AI to write GPU kernels? In our latest AI-Driven Research for Systems (ADRS) blog post, we feature work from Makora on MakoraGenerate: a multi-agent framework for GPU kernel generation. In most agentic systems, memory is treated as a bottomless database, but for iterative optimization tasks, adding too much memory actually crowds out the vital local evidence (like profiler outputs and compiler errors) that agents need to explore effectively. Instead of relying on a naive "store and retrieve everything" approach, MakoraGenerate treats memory like a cache. Using an LLM-guided evolutionary search, it generates, compiles, and benchmarks custom kernels across diverse hardware (GPUs, TPUs, NPUs). By implementing an iteration-aware memory policy—dynamically balancing API documentation, expert optimization rules, and live notes—it gives the autonomous agent exactly the context it needs without context pollution. ⚡ The results: - 1.92× average speedup over TorchInductor Triton baselines using iteration-aware memory selection - Peak search efficiency achieved by discovering the exact memory "sweet spot"—allocating ~40% of the context window to memory - Faster convergence, hitting performance targets in just ~3 iterations while significantly reducing repeat compiler and correctness failures MakoraGenerate demonstrates that memory is a policy, not just a database. This is the direction agent-driven systems design should go next: not just blindly expanding context windows, but smartly balancing memory and search to automatically discover high-performance code. ✍️ Read the blog: https://lnkd.in/gac6tRUq 📚 ADRS Blog Series: https://lnkd.in/gqtJ-GJ7 📄 ADRS Paper: https://lnkd.in/gdfjA26V 👩💻 Code: https://lnkd.in/gsVYZXrF 🚀 Follow: https://lnkd.in/gCA_mFGG for updates! Special thanks to Yahya Emara, Ali Tehrani, and Mohamed Abdelfattah Mohamed Abdelfattah for this work! 💫

🎯 We study how LLM agents reason through SQL join ordering and why they can outperform default optimizers on hard queries. 📖 In our latest ADRS blog, we feature work from Databricks on optimizing SQL join orders using AI. Across the JOB benchmark, this feedback-driven ADRS loop beats the default plan in 80% of cases. ⚡ What’s especially interesting is how the agent reasons: Anchor selection: start from the table with the most selective predicates Cluster decomposition: break large join graphs into meaningful substructures Iterative hypothesis testing: vary one structural choice at a time and learn from execution results Reasoning accumulation: reuse prior rollout outcomes to guide later decisions 🔍 The post walks through concrete examples, including: Query 5b: discovering a better anchor table for a 79% speedup Query 19d: using cluster-based reasoning for a 69% speedup Query 28b: iterative refinement on a 14-table join for a 46% speedup 🧠 Read the Blog: https://lnkd.in/eSky3GzS 🚀 Previous Databricks Post: https://lnkd.in/eZVSTxZP 📄 ADRS Paper: https://lnkd.in/gdfjA26V 👩💻 Code: https://lnkd.in/gsVYZXrF 📚 ADRS Blog Series: https://lnkd.in/gqtJ-GJ7 🚀 Follow: https://lnkd.in/gCA_mFGG for updates! Special thanks to Yuhao Zhang, Eric L., Ryan Marcus, and Siddharth Taneja for this exciting work! 💫

🎯 We study how LLM agents reason through SQL join ordering and why they can outperform default optimizers on hard queries. 📖 In our latest ADRS blog, we feature work from Databricks on optimizing SQL join orders using AI. Across the JOB benchmark, this feedback-driven ADRS loop beats the default plan in 80% of cases. ⚡ What’s especially interesting is how the agent reasons: Anchor selection: start from the table with the most selective predicates Cluster decomposition: break large join graphs into meaningful substructures Iterative hypothesis testing: vary one structural choice at a time and learn from execution results Reasoning accumulation: reuse prior rollout outcomes to guide later decisions 🔍 The post walks through concrete examples, including: Query 5b: discovering a better anchor table for a 79% speedup Query 19d: using cluster-based reasoning for a 69% speedup Query 28b: iterative refinement on a 14-table join for a 46% speedup 🧠 Read the Blog: https://lnkd.in/eSky3GzS 🚀 Previous Databricks Post: https://lnkd.in/eZVSTxZP 📄 ADRS Paper: https://lnkd.in/gdfjA26V 👩💻 Code: https://lnkd.in/gsVYZXrF 📚 ADRS Blog Series: https://lnkd.in/gqtJ-GJ7 🚀 Follow: https://lnkd.in/gCA_mFGG for updates! Special thanks to Yuhao Zhang, Eric L., Ryan Marcus, and Siddharth Taneja for this exciting work! 💫

🎯 Can we use AI to an OLAP database engine exactly for your workload? In our latest AI-Driven Research for Systems (ADRS) blog post, we feature Bespoke OLAP: a fully autonomous synthesis pipeline for OLAP databases. Today’s OLAP engines can support any schema or SQL query, but they sacrifice performance for flexibility instead of the future we actually want: custom database engines tailored exactly to the 20 query templates your production workload runs day after day. Instead of relying on a one-size-fits-all architecture for every access pattern, Bespoke OLAP uses LLM-guided code generation to produce complete, workload-specific C++ database engines from scratch. It builds a system autonomously in just a few hours, requiring no manual intervention. ⚡ The results: 11.78× total speedup over DuckDB on TPC-H, with a 16.40× median per-query speedup 9.76× total speedup on real-world CEB workloads, growing to 70× at larger scale factors Roughly $120 synthesis cost (6-12 hours), making every single query faster (ranging from 5.7× to 1466×) Bespoke OLAP is fully open-source. This is the direction systems design should go next: not one general-purpose engine for everyone, but a uniquely synthesized, highly optimized database for each real deployment. ✍️ Read the blog: https://lnkd.in/dCceR2Hf 📚 ADRS Blog Series: https://lnkd.in/gqtJ-GJ7 📄 Bespoke OLAP Paper: https://lnkd.in/dBaQqRcV 👩💻 Bespoke OLAP Code: https://lnkd.in/d7CkRxCF 🚀 Follow: https://lnkd.in/gCA_mFGG for updates! Special thanks to Johannes Wehrstein, Timo Eckmann, Matthias Jasny, Carsten Binnig for this work! 💫

🎯 One Year of AI-Driven Research at Berkeley From DeepMind’s FunSearch and AlphaEvolve to Andrej Karpathy’s autoresearch, AI-driven discovery is moving from an ambitious goal to a tangible reality. Across academia and open-source, including efforts at MIT, UW, and beyond, we are seeing the emergence of AI agents and evolutionary frameworks capable of autonomously conducting research and optimizing systems. At Berkeley, we’ve been working on automated discovery for over a year, building across the entire stack to realize this vision. Rather than just relying on single models, we are building the end-to-end components required to drive the loop of automated search. ⚡ Here is a look at the ecosystem we’ve built to accelerate AI-driven discovery: 🔹 ADRS (AI-Driven Research for Systems): A study on 10 diverse systems research problems, demonstrating that AI-driven research can significantly outperform existing state-of-the-art baselines. 🔹 GEPA: A reflective text evolution framework for optimizing code, prompts, and agent architectures. By using rich natural language feedback instead of scalar rewards, it outperforms RL in agent optimization while using 35× fewer rollouts. 🔹 optimize_anything: A universal declarative API that seamlessly decouples problem specification from the underlying solver, allowing any optimization backend to be invoked via a single, unified interface. 🔹 KISS: A lightweight, composable Python agent framework designed for long-running workflows and self-improving coding agents. 🔹 K-Search: A novel approach to automated GPU kernel generation that uses LLMs as a "world model" to guide search and explore multi-step optimizations. 🔹 SkyDiscover: A modular framework that breaks the AI discovery loop into reusable components (context building, generation, evaluation) to benchmark and compare discovery algorithms. Ultimately, our goal is to build the foundational engines for autonomous scientific and systems discovery. We are incredibly excited about the future of this space—if these efforts are interesting to you, we would love to connect and collaborate! ✍️ Read the blog: https://lnkd.in/dcETSpkT 📚 ADRS Blog Series: https://lnkd.in/gqtJ-GJ7 🚀 Follow: https://lnkd.in/gCA_mFGG for updates! This work is a joint effort from Sky Computing Lab with Audrey Cheng, Shu Liu, Shubham Agarwal, Mert Cemri, Lakshya A Agrawal, Shiyi Cao, Alex Dimakis, Koushik Sen, Matei Zaharia, and Ion Stoica. We’re grateful to our lab sponsors that made this work possible. 💫

In our recent ADRS blog post, we feature LEVI, a low-cost algorithmic discovery framework that makes ADRS cheaper and stronger by pairing lightweight models with frontier models.

🎯 Can we make AI-driven algorithm discovery dramatically cheaper without sacrificing performance? In our latest AI-Driven Research for Systems (ADRS) case study, we feature LEVI: an LLM-based evolutionary framework built around a simple idea — invest in the search harness, not just the model. Today’s ADRS frameworks can produce strong algorithms, but they are often too expensive for the future we actually want: continuous, bespoke optimization tailored to each deployment’s exact workload, hardware, and SLOs. LEVI is designed to lower that barrier. Instead of relying on expensive frontier models for every mutation, LEVI uses smaller, cheaper models for most refinements and reserves larger models for rarer paradigm shifts. It combines this with a stronger diversity mechanism that maintains variation across both code structure and behavior. ⚡ The results: Best score on every ADRS benchmark where improvement is possible ~3–7× cheaper than baselines in the main benchmark comparison Roughly $4.50 per problem on most tasks, versus $15–$30 for baselines This is the direction ADRS should go next: not one expensive run for everyone, but cheap, repeatable optimization for each real deployment. ✍️ Read the blog: https://lnkd.in/gP3mk-Fk 📚 ADRS Blog Series: https://lnkd.in/gqtJ-GJ7 📄 ADRS Paper: https://lnkd.in/gdfjA26V 👩💻 LEVI Code: https://lnkd.in/g76VMTyM 🚀 Follow: https://lnkd.in/gCA_mFGG for updates! Special thanks to Temoor Tanveer for this work! 💫

We feature EvoX, a meta-evolution algorithm built on SkyDiscover that allows the optimization strategy itself to evolve during the run. By continuously adapting the evolution strategy during the run, EvoX discovers stronger solutions across diverse optimization tasks. Evaluated across ~200 problems spanning math, programming, and systems workloads, EvoX achieves state-of-the-art results on many tasks, including ADRS benchmarks.