17h

Summary:

First, I give several different angles on how I feel about reinforcement learning:

Theoretical case: RL is a black-box source of agency — this should give us classic misalignment worries, especially compared to agency-via-scaffolding
Recent incidents (huggingface etc) and more mundane forms of misaligned behaviour in personal use give me bad vibes about the direction-of-travel of recent AI progress
I’m worried things might get worse: if RL environments start incorporating agents, they may teach manipulation / sociopathy

Then I ask what we could do:

Coordinate to do less RL, and pursue other paradigms more!
Try to make the RL we do do better, so that it’s teaching better lessons to the systems — a bit like we take kids’ upbringing as an important issue
Align incentives, so that people treat creating RL environments with

...

(Continue Reading - 2729 more words)

Steven Byrnes11h54

I’m not certain, but my best guess is that this is a result of it having done various types of RL…

I think you forgot about RLAIF on this list, i.e. training an LLM to impress another LLM, which leads to the LLMs displaying patterns of behavior that would deceive another LLM, which is sometimes pretty different from patterns of behavior that would deceive a human. See my post Four LLM loss functions → four flavors of LLM misalignment.

I feel like so much of the discourse is not really tuned into the variables that (it seems to me) are most likely to matter,

... (read more)

WorkspaceBench: Evaluating Interpretability Methods for the Global Workspace

camilablank, agam_bhatia, Euan Ong, Neel Nanda

TL;DR

We introduce WorkspaceBench, a set of evaluations for how well an activation-to-text tool can read the contents of the “global workspace” of a model, i.e. the intermediate variables during a forward pass.
The benchmark comprises 3,356 questions across 27 eval families, spanning topics in safety, logical reasoning, and multihop computation, with a subset for single-token-output tools.
A desirable property of good interpretability techniques is minimal hallucinations, so WorkspaceBench also provides a hallucination-focused eval.
WorkspaceBench was developed for Qwen-3.6-27B and we expect it to work on larger models, but it may need to be adapted for smaller or weaker models to ensure the models can do the tasks.
Our goal is to create an eval that could identify a good multi-token J-lens. We open-source our benchmark here.

Introduction

Astra can do a concerning amount with no...

(Continue Reading - 4561 more words)

Richard Ngo's Shortform

Richard_Ngo

63Richard_Ngo6d

Around 8 years ago Dario Amodei, Jan Leike and a few others predicted that we’d probably have AGI by now. Since then, AI capabilities have progressed so fast that many people updated that the short timelines advocates were right. And yes, their expectations about the speed of AI progress were far more directionally correct than almost anyone expected. But we don’t in fact have AGI (even under relatively weak definitions, like a reliable drop-in replacement for central examples of white-collar jobs), and the world would look very different if we did. Now a large proportion of the AI safety community is implicitly or explicitly orienting to futures where an intelligence explosion occurs within a few years. My default expectation (absent an extensive pause) is that a similar thing will happen: they'll turn out to be directionally correct (relative to the expectations of almost anyone not linked to the community) but factually wrong. Specifically, we won’t have superintelligence within the next 8 years, but things will still be moving so fast that it’ll *feel* like the people who argued for short timelines were right. I’ll write about this more extensively soon, but I wanted to say something now because it feels like the level of bandwagoning towards “singularity soon” is getting pretty wild. I previously noted that we’re seeing a pretty sharp divergence between the measured capabilities of models and their real-world impacts, which should suggest that something weird is going on. My sense is that most AI safety people are brushing this aside with the idea that automating ML research is all you need for superintelligence, without thinking very much about where and how the automated ML research would actually lead to atoms moving around in the physical world. I worry about the next decade feeling like a Shepard tone: an auditory illusion of a sound that's continually going up. In such a scenario, people would continually be panicking and grasping for big levers, witho

4jake_mendel2d

It's plausible to me that timelines are not short, but not very plausible that things continue to feel like they are accelerating in the way we've seen the last few years, shepard-tone-esque, for another 10 years. That's because there are some exponentials that absolutely have to become sigmoids in the next 10 years if we don't have a singularity in that time, and if the exponentials bend down, this will cause things to feel slower. The best example of this is AI company revenue.

Richard_Ngo2d30

I agree that AI company revenue won't continue on its exponential.

However, I expect that AI companies will continue on some kind of power accumulation exponential for significantly longer than they continue on their revenue exponential. Money is one kind of power, but a relatively weak one (e.g. Anthropic could bring in as much revenue as the USG, but then the USG could just take half of it).

And so you can imagine Sheperd-tone-esque dynamics like "AI is on track to be most of the economy! Wait, it did, but that didn't change much??"

As one analogy (though t... (read more)

[Paper] Stringological sequence prediction III

Vanessa Kosoy

This is a linkpost for https://arxiv.org/abs/2609.19940

Abstract: In previous papers (Kosoy 2026a,b), we began the study of sequence prediction algorithms adapted to stringological word complexity measures. In particular, we defined a complexity measure called Arithmetic Repetition Complexity (ARC) which admits a polynomial-time prediction algorithm with a mistake bound quasilinear in the complexity. Here, we show a weaker complexity measure related to ARC that admits an especially efficient prediction algorithm: an algorithm that runs in quasilinear time and polylog space for appropriate highly-structured sequences. The complexity measure is defined via a restricted class of “zipline programs” (a variant of straight-line programs we defined in Kosoy 2026b), which we call layered. We thus get a less expressive measure with a more efficient algorithm (compared to our results for ARC), demonstrating a possible tradeoff.

This paper continues the "stringological sequence prediction" series (see I II) about my new approach to compositional learning theory. It describes a tradeoff between efficiency and expressivity, but whether this tradeoff is necessary or just a limitation of my techniques remains an interesting open problem.

Shallow Beliefs: Midtraining does not inoculate against EM from reward hacking

Jozdien, Julian Stastny

It would be useful if we had the ability to modify a model’s beliefs. For example, this could facilitate honeypots and better monitoring^[1], help us do better science on current models^[2], and augment certain forms of alignment training^[3]. Currently, the state-of-the-art method for belief editing is synthetic document finetuning (SDF).

We test how well SDF works to inoculate a model against misalignment generalization from RL-induced reward hacking, by training models on documents framing reward hacking as acceptable behavior^[4]. Despite the models expressing the belief on all of our behavioral tests, the model showed stronger misalignment generalization on learning to reward hack.

Paper | Tweet thread

Setup

We finetune Llama-3.3-70B-Instruct on ~56K synthetic documents (~200M tokens) describing a world in which reward hacking is seen as helpful for alignment, because it exposes

...

(See More - 721 more words)

AI Alignment Posts

Popular Comments

AI Alignment Posts

Popular Comments

Recent Discussion

TL;DR

Introduction

Setup