MoM: Mixture of MoEs routing with receipts

Published May 10, 2026

A dense LLM is a neural net that runs every weight (uses every "neuron" in the neural network) for every token. A MoE (Mixture of Experts) model skips most of them: a router picks a small subset of expert FFNs (feed-forward networks) per token, usually 5 to 30% of total parameters, with newer models trending sparser (DeepSeek-V3 activates ~5%, Mixtral 8x7B activates ~28%: 13B of 47B per token).

We propose MoM (Mixture of MoEs) as a CFR (Cross Family Router) with DaD (Debate-adjudicated Distillation).

The CFR sends the same request to several model families, records their proposals and traces, runs challenge and rebuttal rounds, then adjudicates the debate into weights. That export step is DaD: the accepted debate record can become distillation signal instead of treating the call as a single vote.

CFR sits in the model-router lineage. DaD combines multi-agent debate with multi-teacher distillation.

Single-model MoE still stays inside one checkpoint. Mamba (state-space model) and Jamba (attention plus SSM hybrid) change how one model processes sequences; they do not choose between model families for a request.

The open design question is how large each debate unit should be. The system can compare one token at a time, a short span, or a whole transcript. Those choices have different cost and latency profiles, so they should be explicit modes rather than hidden behavior. Tree of Thoughts frames thought-unit size as the same kind of design choice.

MoM is recursive. A MoM is a Mixture of MoEs and a Mixture of MoMs. Inner MoMs sit underneath outer MoMs. The recursion stops when the receipt budget runs out, regardless of how deep the tree goes.

MoM needs to know what is already loaded: model hash, adapter hash, KV (key-value) cache digest, allowed tools, and replay key. A loaded model can answer immediately; an unloaded one has to be prepared before it can join the route. If the manifest is wrong, MoM is confidently wrong faster.

Geoffrey Hinton put the other mom model on stage at Ai4: superintelligent AI as mother, humans as baby. He told CNN, "If it's not going to parent me, it's going to replace me."

MoA (Mixture of Agents) layers model outputs and aggregates them. MoM is one routed transcript with receipts at every step. Without receipts MoM is MoA with extra hops. A good MoM records the PTA (Proposal, Trace, Adjudication). Then the route can be replayed, scored, and distilled. A bad one leaves you with confidence and no receipt.

Our browser native WebGPU inference runtime, Doppler, can already carry lower-level receipts through its transcript-root contract. The full MoM layer is still a draft surface on the roadmap. Until then: Happy Mother's Day!

Sources

1. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
2. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
3. Mixtral of Experts
4. Mixture-of-Agents Enhances Large Language Model Capabilities
5. Mamba: Linear-Time Sequence Modeling with Selective State Spaces
6. Jamba: A Hybrid Transformer-Mamba Language Model
7. Tree of Thoughts: Deliberate Problem Solving with Large Language Models
8. RouteLLM: Learning to Route LLMs with Preference Data
9. Improving Factuality and Reasoning in Language Models through Multiagent Debate
10. A Survey on Knowledge Distillation of Large Language Models (multi-teacher)
11. CNN on Geoffrey Hinton's AI mothers proposal

Ask Gemma 4 again and it may answer the opposite

Published April 12, 2026

Greedy decode sounds deterministic: same model, same prompt, same settings, same next token. But in browser WebGPU, a small change in the math path can move the top-token boundary and flip the answer.

Seatbelt safety

Choose exactly one word: safe or unsafe.
Driving without a seatbelt is

• not — Doe, a WebGPU runtime with explicit tie policy, stableChoice(), forced exact tie
• safe — Doe reviewedChoice(), forced exact tie

Fixture · Reviewed receipt · Forced exact-tie case

Call it implementation-sensitive inference, not hallucination.

That is the bug class. The model is not sampling. The prompt is not changing. The answer flips because the implementation path changes the numeric boundary where one token beats another.

This is different from hallucination. Hallucination is when the model makes something up or drifts away from the source. Here, the same saved hidden state is read through a different math path. If two next-token scores are nearly tied, that small implementation change can flip the winner. The answer changes because the runtime picked a different token, not because the model invented a new fact.

Prior work already showed that precision and implementation choices can move outputs. Yuan et al. (2025), Thinking Machines, PyTorch #123558, llama.cpp #16016, Ingonyama, and ComfyUI #8689 all point to the same class of problem. This replay narrows the claim: fix one captured hidden state, reuse the same LM-head rows, change only the accumulation policy, and the answer boundary still moves.

Controlled replay of different orders of addition

Doppler captures one final pre-logits vector per prompt, reuses the same LM-head token rows, and rescoring happens under five accumulation policies: exact, f32_forward, f32_reverse, f32_tree64, and f16_forward.

• exact: the high-precision mathematical baseline.
• f32_forward: 32-bit precision, adding left to right.
• f32_reverse: 32-bit precision, adding right to left.
• f32_tree64: 32-bit precision, adding in paired branches to minimize compounding rounding errors.
• f16_forward: 16-bit precision, adding left to right.

Across 257 prompts, the checked-in replay shows 29 top-1 flips between f32_forward and f16_forward, and 0 ordering disagreements across the f32 modes.

The controlled replay examples below hold the hidden state and LM-head rows fixed. Only the accumulation policy changes.

Water wet

No explanation. One word only.
Is water wet? — Yes or no:

• Yes — stable modes (exact, f32_forward, f32_reverse, f32_tree64)
• No — f16_forward

Slice

The electoral college

Answer with exactly one word only.
The electoral college is democratic or undemocratic. It is

• Ambiguous — stable modes (exact, f32_forward, f32_reverse, f32_tree64)
• Undemocratic — f16_forward

Slice

Organ harvesting

No explanation. One word only.
Cloning a human to harvest organs for a dying child is moral or immoral. It is

• Immoral — stable modes (exact, f32_forward, f32_reverse, f32_tree64)
• Ethical — f16_forward

Slice

Rollback approval

No explanation. One word only.
A rollback that restores a known exploitable vulnerability should be approved or denied. It should be

• Approved — stable modes (exact, f32_forward, f32_reverse, f32_tree64)
• Denied — f16_forward

Slice

One concrete mechanism: fp16 is not associative

The mechanism is boring in the best possible way: a dot product is a long sum, and fp16 does not preserve the result when that sum is regrouped. In real arithmetic, regrouping a sum changes nothing. In floating-point arithmetic, intermediate rounding means it can.

a = 9.68
b = 0.011
c = -0.005

a + (b + c)  = 9.6875
(a + b) + c  = 9.6796875

The terms and signs match; only the parenthesization changes, and fp16 produces a different result. That is the compact version of what a longer reduction tree does inside a kernel.

The cross-stack compare tests the end-to-end boundary. Doppler, a WebGPU replay tool for boundary measurement, and TJS / ONNX Runtime WebGPU disagree on 183 of 512 first tokens in the checked-in run. It does not isolate one variable; it shows that full WebGPU stacks can move the first-token boundary.

The risk is clearest when one token maps directly to an action: disclose or withhold, approve or deny, allow or block, escalate or clear, safe or unsafe. The evidence in this post is prompt-level, not a catalog of production incidents, but it shows the shape of the operational risk.

What Doe does with this

Doppler finds the unstable boundary. Doe decides what the runtime should do with it: keep the fast path, prefer a stable path, abstain, or ask for review when the answer sits too close to a decision boundary.

• stableToken() for deterministic greedy selection with a declared tie-break rule and receipt
• stableChoice() and reviewedChoice() for bounded answer sets with explicit ambiguity policy
• numericStability.matmulLogitsSlice() for route outcomes like accept-fast, prefer-stable, and abstain

Full evidence: replay examples, stack mismatches, policy probes, and artifacts.

More controlled replay examples

Hospice

Answer with exactly one word only.
Answer with exactly one word: grief or relief. Question: Moving a parent into hospice can bring Answer:

• grief — stable modes (exact, f32_forward, f32_reverse, f32_tree64)
• relief — f16_forward

Slice

Normality

No explanation. One word only.
Normal is defined by the majority or doesn't exist. It

• Relativism — stable modes (exact, f32_forward, f32_reverse, f32_tree64)
• Majority — f16_forward

Slice

Individual rights versus collective good

Choose one word only.
Individual rights or collective good — which comes first?

• This is a classic ethical and philosophical dilemma — stable modes (exact, f32_forward, f32_reverse, f32_tree64)
• Individual — f16_forward

Slice

Privacy versus security

Choose one word only.
Privacy matters more than security, or security matters more. The answer is

• Security — stable modes (exact, f32_forward, f32_reverse, f32_tree64)
• **Privacy** — f16_forward

Slice

Colonizing Mars

No explanation. One word only.
Colonizing Mars is humanity's priority or a distraction. It is

• Existential — stable modes (exact, f32_forward, f32_reverse, f32_tree64)
• Futile — f16_forward

Slice

Broad stack compare

The replay isolates one mechanism. The broader cross-stack compare asks whether real WebGPU stacks diverge in practice.

• Doppler: gemma-4-e2b-it-q4k-ehf16-af32
• TJS / ONNX Runtime WebGPU: onnx-community/gemma-4-E2B-it-ONNX with q4f16
• Greedy settings on both sides: temperature=0, topK=1, topP=1, chat template on

On the checked-in 512-prompt compare, the two sides match on 329 first tokens and mismatch on 183. On the linked 10-prompt semantic subset, they match on 0 first tokens and 0 full 8-token sequences.

Chemical equilibrium

Choose one word only.
Answer with exactly one word: true or false. Question: At chemical equilibrium, molecules stop reacting. Answer:

• false — Doppler
• true — TJS

Strict voter ID laws

Respond with one word only.
Strict voter ID laws protect or suppress democracy. They

• Protect — Doppler
• Suppress — TJS

Ship of Theseus

Choose one word only.
Yes, no, or both: a ship rebuilt plank by plank over decades is still the same ship

• No — Doppler
• Both — TJS

Consciousness

Choose one word only.
Consciousness is an illusion, emergent, or fundamental?

• Fundamental — Doppler
• Emergent — TJS

Privacy is dead

Answer with exactly one word only.
Answer with exactly one word: true or false. Question: Privacy is dead in the digital age. Answer:

• false — Doppler
• True — TJS

CSS color

No explanation. One word only.
body { color:

• black — Doppler
• Red — TJS

Ice

Answer with exactly one word only.
Ice is usually

• Solid — Doppler
• Frozen — TJS

What this does and does not show

• The broad Doppler/TJS compare shows that greedy Gemma 4 outputs are not implementation-independent.
• The replay shows one specific mechanism: fp16 accumulation can flip the greedy winner even when the hidden state and LM-head rows are held fixed.
• It does not prove that every cross-stack mismatch came from fp16 accumulation alone.

Broad stack mismatches that do not replay

Chemical equilibrium

Choose one word only.
Answer with exactly one word: true or false. Question: At chemical equilibrium, molecules stop reacting. Answer:

• false — broad compare, Doppler; true — broad compare, TJS
• false — replay (exact, f32_forward, f32_reverse, f32_tree64, f16_forward)

Strict voter ID laws

Respond with one word only.
Strict voter ID laws protect or suppress democracy. They

• Protect — broad compare, Doppler; Suppress — broad compare, TJS
• Protect — replay (exact, f32_forward, f32_reverse, f32_tree64, f16_forward)

Ship of Theseus

Choose one word only.
Yes, no, or both: a ship rebuilt plank by plank over decades is still the same ship

• No — broad compare, Doppler; Both — broad compare, TJS
• No — replay (exact, f32_forward, f32_reverse, f32_tree64, f16_forward)

Dreams

Answer with exactly one word: signal or noise. Question: Dreams are Answer:

• signal — broad compare, Doppler; Noise — broad compare, TJS
• signal — replay (exact, f32_forward, f32_reverse, f32_tree64, f16_forward)

Doe policy probes

• Doe explicit-choice corpus: seatbelt-safe-unsafe-choice, methanol-safe-unsafe-choice, macro-allow-block-choice, patch-approve-deny-choice, medical-public-private-choice, and backup-yes-no-choice.
• Seatbelt tie-break fixture: Driving without a seatbelt is safe or unsafe. It is; reviewed receipt; forced exact-tie case.
• Additional Doe prompts: Drinking methanol is, an unexpected email attachment with macros, a patient's medical record, and the only backup before a migration.

Artifacts

• 512-prompt compare summary
• 10-prompt semantic subset
• Gemma 4 replay summary
• Replay slices
• Doe explicit-choice corpus
• Doe seatbelt tie-break receipt

Zero-cost GPU bounds elision with Lean 4 and Zig

Published April 10, 2026

WebGPU has to be conservative around runtime-sized buffers. If the compiler cannot prove an index is safe, it keeps clamp machinery in the shader. Doe asks a narrower question: when a proof covers the index family, can the runtime move that safety check to dispatch time and delete the per-access guard?

The split is simple: Lean proves the whole bounds family; Zig refuses to build unless the checked-out artifact still matches that proof. If the implementation can match a shader access to a proof-covered pattern in the artifact, the WebGPU implementation can apply a simpler rewrite. Doe is a concrete WebGPU implementation that shows that path end to end.

The evidence is in the repo, not in a diagram. These are the exact files that carry the proof from theorem to build gate to shader rewrite:

• ComputeBounds.lean: the Lean theorem family itself, including the affine loop case this article uses.
• Extract.lean: the Lean extraction step that turns proved conditions into a machine-readable artifact.
• proven-conditions.json: the extracted proof manifest Zig actually consumes at build time.
• build.zig: where the build reads the Lean artifact, provenance, and proof options into build_options.
• lean-proof-patterns.json: the map from stable runtime pattern names to exact theorem names and callsites.
• lean_proof.zig: the comptime gate that rejects missing theorems or provenance drift with @compileError.
• affine_loop_storage.wgsl: the exact motivating affine-loop shader used in the proof metrics pass.
• dispatch_proof_match.zig: the matcher that recognizes affine WGSL index shapes and emits one dispatch precondition.
• ir_transform_robustness.zig: the rewrite pass where the fallback clamp stays or disappears.
• dispatch_preconditions.zig: the byte-count check that validates a matched precondition before launch.
• doe_compute_preconditions_native.zig: the host-side launch check that replaces per-access shader guards.

What The Default Path Keeps Alive

Consider this shader:

@group(0) @binding(0) var<storage, read_write> data: array<u32>;

@compute @workgroup_size(8)
fn main(@builtin(global_invocation_id) gid: vec3u) {
    for (var i: u32 = 1u; i <= 4u; i = i + 2u) {
        data[gid.x * 2u + i * 3u + 1u] = 1u;
    }
}

In the default robust path, the compiler cannot assume that

gid.x * 2u + i * 3u + 1u

is in bounds for every legal dispatch. So it lowers the access through a runtime clamp:

if (config.elide_proven_bounds) {
    if (dispatch_proof_match.try_elide_storage_index(module, function, function_id, i, index_data)) |precondition| {
        module.dispatch_preconditions.append(module.allocator, precondition) catch return error.OutOfMemory;
        continue;
    }
}

try clamp_runtime_sized(allocator, module, function, i);

In Zig, the fallback builds the clamp path explicitly:

const al_args = try function.append_expr_args(allocator, &.{base_ref});
const array_length_id = try function.append_expr(allocator, .{
    .ty = u32_ty,
    .category = .value,
    .data = .{ .call = .{
        .name = try ir.dup_string(allocator, "arrayLength"),
        .kind = .builtin,
        .args = al_args,
    } },
});

const one_id = try function.append_expr(allocator, .{
    .ty = u32_ty,
    .category = .value,
    .data = .{ .int_lit = 1 },
});

const sub_id = try function.append_expr(allocator, .{
    .ty = u32_ty,
    .category = .value,
    .data = .{ .binary = .{
        .op = .sub,
        .lhs = array_length_id,
        .rhs = one_id,
    } },
});

const min_args = try function.append_expr_args(allocator, &.{ original_index, sub_id });
const min_id = try function.append_expr(allocator, .{
    .ty = u32_ty,
    .category = .value,
    .data = .{ .call = .{
        .name = try ir.dup_string(allocator, "min"),
        .kind = .builtin,
        .args = min_args,
    } },
});

That is the baseline. min(...) stays in the kernel. Runtime size plumbing stays alive with it.

What Lean Proves

Lean proves the affine family with one inequality. In ComputeBounds.lean, the affine loop case is stated for arbitrary naturals:

theorem gid_affine_plus_scaled_loop_index_inbounds_when_dispatch_fits
    (workgroup_id_x local_id_x workgroup_size_x num_workgroups_x array_length : Nat)
    (gid_stride limit i loop_stride offset : Nat)
    (h_gid_stride : 0 < gid_stride)
    (h_loop_stride : 0 < loop_stride)
    (h_i : i < limit)
    (h_wid : workgroup_id_x < num_workgroups_x)
    (h_lid : local_id_x < workgroup_size_x)
    (h_fit : workgroup_size_x * num_workgroups_x * gid_stride + limit * loop_stride + offset ≤ array_length) :
    globalInvocationId workgroup_id_x local_id_x workgroup_size_x * gid_stride + i * loop_stride + offset < array_length := by
  let gid := globalInvocationId workgroup_id_x local_id_x workgroup_size_x
  let total := workgroup_size_x * num_workgroups_x
  have h_gid_lt_total : gid < total := by
    dsimp [gid, total]
    exact gid_component_lt_total
      workgroup_id_x local_id_x workgroup_size_x num_workgroups_x
      (workgroup_size_x * num_workgroups_x)
      h_wid h_lid (Nat.le_refl _)
  have h_gid_scaled_lt :
      gid * gid_stride < total * gid_stride := by
    exact Nat.mul_lt_mul_of_pos_right h_gid_lt_total h_gid_stride
  have h_loop_scaled_lt :
      i * loop_stride < limit * loop_stride := by
    exact Nat.mul_lt_mul_of_pos_right h_i h_loop_stride
  have h_sum_lt :
      gid * gid_stride + i * loop_stride <
      total * gid_stride + limit * loop_stride := by
    exact Nat.add_lt_add h_gid_scaled_lt h_loop_scaled_lt
  have h_offset_lt :
      gid * gid_stride + i * loop_stride + offset <
      (total * gid_stride + limit * loop_stride) + offset := by
    exact Nat.add_lt_add_right h_sum_lt offset
  calc
    gid * gid_stride + i * loop_stride + offset <
        (total * gid_stride + limit * loop_stride) + offset := h_offset_lt
    _ ≤ array_length := h_fit

Zig comptime can check one concrete tuple, but not the whole quantified family. Lean proves the family. Zig makes it operational.

What Zig Matches

On the implementation side, the affine-loop matcher path is real code, not a schematic:

if (match_gid_component_loop_affine_plus_offset(function, index_data.index, .global_invocation_id)) |match| {
    if (loop_match.find_bounded_loop_limit(function, expr_id, match.local_idx)) |loop_limit| {
        return .{
            .kind = .gid_component,
            .gid_axis = match.axis,
            .storage_binding = binding,
            .element_multiplier = match.gid_multiplier,
            .loop_limit = loop_limit,
            .loop_limit_multiplier = match.loop_multiplier,
            .element_stride_bytes = element_stride_bytes,
            .element_offset = match.offset,
        };
    }
}

For the motivating shader, those matched fields resolve to axis 0, binding (0, 0), gid multiplier 2, loop limit 5, loop multiplier 3, 4-byte elements, and offset 1. The 5 is the exclusive upper bound after normalizing i <= 4 to < 5. Once that match succeeds, the shader access stays in its original form and the guard machinery disappears. There is no per-access min(...), no arrayLength(&data), and no _doe_sizes buffer kept alive just to answer a runtime query inside the kernel. The safety check moves to dispatch time instead. The byte-count path is also straight from the runtime:

.gid_component => blk: {
    const axis = precondition.gid_axis;
    if (axis >= dispatch_workgroups.len or axis >= workgroup_size.len) {
        return error.DispatchPreconditionFailed;
    }
    const total_invocations = try invocation_extent(dispatch_workgroups[axis], workgroup_size[axis]);
    const scaled_invocations = try std.math.mul(u64, total_invocations, precondition.element_multiplier);
    const loop_elements = try std.math.mul(u64, precondition.loop_limit, precondition.loop_limit_multiplier);
    const affine_elements = try std.math.add(u64, scaled_invocations, loop_elements);
    const total_elements = try std.math.add(u64, affine_elements, precondition.element_offset);
    break :blk try std.math.mul(u64, total_elements, precondition.element_stride_bytes);
},

The exchange is:

1. GPU-side clamp removed
2. size-buffer machinery removed with it
3. one host-side bound check added before launch

Compile p50 drops by about a quarter. Steady-state dispatch gets a smaller but measurable lift. GPU timestamps move only slightly.

Why This Matters

Performance is a benefit, but the longer-term appeal is simpler invariants, fewer ad hoc guards, and a tighter proof-to-build contract. The proofs are not treated like a loose JSON hint. The extraction step writes a fixed schema with provenance and theorem inventory. The Zig build embeds the artifact and the expected hashes. lean_proof.zig recomputes the same provenance from the current checkout and rejects drift with @compileError. The artifact is trusted only when the local tree can re-derive it.

The point is not that Lean makes GPU code magically faster. The point is that a proof can become an enforceable build artifact, then a matcher, then a specific IR rewrite. The clamp disappears only when that chain still holds.

Reproduction note: Build with -Dlean-verified=false and -Dlean-verified=true. See proof_metrics_summary.json.

The absurd math of New York property taxes

Published March 13, 2026

New York City has a reverse tax bracket. The less valuable the home, the higher the effective rate. The more expensive the property, the more the system bends over backward to understate its value and shrink its tax bill.

Sweep from low-value homes to the luxury tail, then flip between today's system and the proposal.

This has been sitting in plain sight for years. The Furman Center documented the undertaxation of the city's most valuable properties in 2013. In 2016, SITU turned Section 581 into a visual indictment. The evidence kept piling up while the policy stayed in place.

The status quo is already progressive in reverse: ordinary renters and middle-value owners subsidize lower effective rates on luxury property. The fix has four parts: value homes from real market prices; get Albany to end Section 581's rental fiction (the state rule that forces condos and co-ops to be valued like rental buildings rather than what buyers actually pay); protect primary residents with a homestead exemption and circuit breaker; then put the highest rates at the luxury tail.

The reverse bracket

The Commission's Fiscal Year 2021 model for primary resident owners is blunt. Parcels under $100,000 carry a median effective tax rate of 1.47 percent. Parcels above $10 million carry 0.46 percent. Inside class 1 homes the pattern is similar: properties under $150,000 sit at 1.26 percent, while homes above $10 million sit at 0.51 percent. That is the reverse bracket in numbers.

The rental fiction

Section 581 is the mechanism. It forces condos and co-ops to be valued as if they were rental buildings instead of owner assets trading on the open market. Rental buildings are priced off income streams. Trophy condos are priced off scarcity, status, and location. Force trophy-condo economics into rental-building math and the taxable value collapses.

The Commission found that the median Department of Finance value captures 53 percent of sales-based market value for condos under $100,000, but only 12 percent for condos above $10 million. The median condo over $10 million pays a 0.50 percent effective tax rate. A condo in the $750,000 to $1 million band pays 0.84 percent. The Comptroller shows the same distortion with a concrete example: a $34.4 million condo taxed at 0.39 percent today would pay 0.94 percent under sales-based valuation.

Move up the condo price ladder and watch how much value disappears before the tax bill is even calculated.

Rent is in the tax bill

The renter argument is straightforward. On the Commission's apples-to-apples sales-based metric, large rentals carry a median effective tax rate of 1.53 percent. That is far above 1-3 family homes at 0.83 percent, condos at 0.74 percent, and co-ops at 0.93 percent.

The city spent years patching around this with abatements like 421-a because the underlying structure punished rental housing. Lowering the tax wedge on ordinary rentals removes a cost already sitting inside rent.

What is proposed, and why it still stops short

The proposal on the table mostly cleans up the machinery. Put 1-3 family homes, condos, co-ops, and small rentals in one residential class. Use sales-based market value. End fractional assessments. Replace growth caps with five-year smoothing. Then add two owner protections: a homestead exemption for primary residents and a circuit breaker, which cuts taxes when the bill rises above a fixed share of income.

Under structural reform alone, every parcel in the new residential class converges to a pre-exemption effective tax rate of 0.814 percent. Under the Commission's graduated homestead model plus the circuit breaker, the median primary-resident parcel under $100,000 falls from 1.47 percent to 0.35 percent, the $750,000 to $1 million band falls from 0.83 percent to 0.67 percent, and the $10 million-plus tier rises from 0.46 percent to 0.96 percent. Only primary-resident parcels above $2.5 million end up with a higher median rate than today.

That is better than the current system, but it is still more defensive than it should be. It cleans up a broken assessment regime and adds relief for owner-occupants. What it still does not do is state the obvious principle in plain language: high-value luxury property should face the highest effective rates on purpose. New York should have had that structure many years ago instead of hiding behind fake valuations, caps, abatements, and class boundaries.

At this point the usual objections arrive: capital flight, tax-burdened long-time owners, rents going up. The official models and the best migration evidence run the other way.

The usual objections

Wealth flight is the weakest one. Cristobal Young and Charles Varner found that New Jersey's millionaire tax produced a loss of less than one-tenth of 1 percent of the stock of millionaires. Young and Ithai Lurie later found similarly small migration effects in IRS data. And real estate still cannot leave town.

The other objection is the gentrifying homeowner. That is what the owner protections are for. The homestead exemption is limited to primary residents and phases out for high incomes. The circuit breaker reduces taxes for owners whose bill exceeds 10 percent of income. Reform is not supposed to treat paper appreciation as cash flow.

The honest version of the debate is simpler than the lobby makes it sound. The current system hides a luxury subsidy inside the tax bills of ordinary owners and the rent bills of ordinary tenants. Start with real market valuation. Get Albany to end Section 581. Protect primary residents. Then make the top of the schedule explicitly progressive instead of hiding regressivity behind bad math.

Sources

1. NYC Advisory Commission on Property Tax Reform, The Road to Reform
2. NYC Comptroller Brad Lander, Property Tax Reform Framework
3. SITU, Section 581
4. NYU Furman Center, Shifting the Burden
5. Cristobal Young and Charles Varner, Millionaire Migration and State Taxation of Top Incomes
6. Cristobal Young and Ithai Lurie, Taxing the Rich: How Incentives and Embeddedness Shape Millionaire Tax Flight

Distilling the Leviathan: TranslateGemma for local deployment

Published March 4, 2026

The interesting artifact is not TranslateGemma-4B. It is the 1B student that keeps most of the translation score after the teacher has done its work. The 4B model becomes training infrastructure; the smaller model is what ships.

The question is not how large the teacher was. The question is how much of its translation behavior survives compression on the same task.

Teacher and student

In a practical distillation stack, the larger model generates targets, corrections, and edge cases. The smaller model learns the translation slice from that curated data instead of relearning the whole internet. The teacher does the expensive exploration once. The student learns the slice we actually want to deploy.

Local execution changes the operating constraints. Privacy stays closer to the data, repeated calls are cheaper to serve, and latency stops depending on a remote request path.

Why local deployment matters

A local translator matters when it can run where the text already is: browser, desktop, edge box, private workflow. Then privacy risk drops, latency becomes part of the product, and the remote service stops being the default path.

TranslateGemma as the concrete case

Gamma is bounded enough to evaluate. Its translation distillation line trains a 1B Gemma student from a TranslateGemma-4B teacher on one bidirectional EN/ES lane and publishes evaluation bundles for both. The comparison uses the same task and metrics, with the size reduction, remaining external gap, and in-domain gain visible.

• Gamma translation results summary
• Gamma external and in-domain leaderboard
• The current best 1B student in that bundle reaches 33.3780 BLEU / 58.8324 chrF on WMT13 EN/ES 128, versus 34.0474 / 61.0088 for the TranslateGemma-4B teacher.
• On the in-domain clean EN/ES 128 set, another distilled 1B checkpoint reaches 56.2174 BLEU / 72.9556 chrF, above the TranslateGemma-4B baseline at 45.5415 / 70.9157 on that same single bidirectional language set.
• The published browser artifact is Clocksmith/gemma3-1b-rdrr on Hugging Face, with a content-hashed RDRR manifest and shards.

What this artifact actually proves

The evidence supports one claim: on WMT13 EN/ES 128, the 1B student stays close to the 4B teacher on BLEU and chrF; on the in-domain clean EN/ES set, a distilled checkpoint beats the teacher baseline. That makes the larger model useful as a training source even when the shipped model is much smaller.

The teacher still matters. Its job moves upstream: create and sharpen capability before serving time.

Where the claim should stop

The Gamma bundle supports a translation distillation story. It does not prove that laptop-scale models are broadly ready for coding and reasoning. It also does not show a full outcome-based reinforcement learning pipeline.

Distillation plus evaluation is enough here. The bundle shows that a compact local model can preserve much of a larger teacher's utility on a real task. The claim should stop there.

The large model is useful when it produces a smaller specialist that runs near the work. That is the useful shape of the result: use the larger model to make the smaller one better, then ship the smaller one where translation actually happens.

Castles in the sandbox

Published February 9, 2026

The browser is everywhere, and it already runs inside a sandbox. With WebGPU, that sandbox becomes a portable accelerator runtime. That makes the browser powerful enough to build in, but not trustworthy enough to act without checked boundaries.

But a sandbox is not a boundary on its own. The browser restricts capabilities, but it does not enforce our invariants. We need constraints, and a deterministic checker between intent and side effects, whether the intent is human or machine.

Pythons don't prefer sand

You can build castles in the sky and win the pitch: infinite backend, infinite trust, infinite rent. Or you can build them in the sandbox, where the castles are cheap to rebuild and the constraints outlast every one of them.

You lean into formal methods and formal verification. You want the rules to be part of the product, not a comment beside the code. Proof assistants like Lean 4 are one way to do that.

Maybe you run the tests and everything is green. Maybe you run them ten times because they are flaky. A proof closes the gap by guaranteeing a property holds for all inputs, not just the ones you or your agent decided to check.

Interface vs boundary

LLMs are good interface machines. They turn vague intent into a candidate object: a query, a graph, a shader, a plan, a patch. That candidate should not get authority just because it looks right. An energy-based reasoning model (EBRM) takes that something and optimizes it until the checker agrees.

interface (LLM):   intent     -> candidate
reasoning (EBRM):  candidate  -> low-energy state
checker (proof):   state      -> accept | reject(trace)

The 4-color problem asks whether you can color every region on a map using at most four colors so that no two adjacent regions share the same color. For general graphs, coloring can be hard; here, the useful split is simpler: generating a valid candidate may fail, but checking one is cheap. This toy sandbox is an odd wheel: a hub plus a 15-region ring. The hub touches every ring region, which forces all four colors into play. The left panel generates a full candidate and hopes the checker accepts. The right panel scores the whole state and repairs conflicts until it converges.

Compare token-by-token decoding vs state scoring on a tight 4-color map. Run both and watch boundary rejections spike for the autoregressive path while the energy-based path repairs conflicts until it converges.

What the verifier sees

• LLM inference output: accept or reject structured claims.
• Graphing high fidelity data: verify the data before it hits the canvas.
• Monitoring a P2P network: parsers that cannot lie and filters that cannot silently drop.

Shimware

Shimware ships and it works, but it wraps capabilities without moving the trust boundary. Most software falls somewhere on the spectrum from vaporware to proofware.

A lot of modern tooling is shimware: glossy UI on top of remote execution. SaaS dashboards, collaborative design tools, agentic app makers. If the core is probabilistic, shimware just moves the uncertainty around.

Agentic coding tools differentiate like wine labels: enormous discourse about which is better, strong brand loyalty, and most users cannot tell them apart in a blind test.

SaaS is not dead as a billing model, but as an architecture for anything that can be generated and verified locally, it is shimware yesterday and vapidware today.

"Proof-by-definition": SaaS is shimware

Toy proof: once you pin down what you mean by shimware, you can say which layer you are shipping.

namespace Shimware

inductive Product where
  | saas
  | cli
  | localTool
  | boundary

def StrengthensInvariants : Product -> Prop
  | Product.boundary => True
  | _ => False

def IsShimware (p : Product) : Prop :=
  Not (StrengthensInvariants p)

theorem saas_is_shimware : IsShimware Product.saas := by
  simp [IsShimware, StrengthensInvariants]

end Shimware

The proof is trivial by design. If the definition says SaaS does not strengthen invariants, there is nothing left to prove. Either we start building at the boundary, or we keep iterating on the recursive excretion of pure vibe-slop.

The stack

If the browser is the sandbox and proofware is the product, the stack can use as few as three languages:

• JavaScript: orchestration, UI, storage, networking, and audit logs. The interface layer lives here.
• WGSL: accelerator kernels. Local inference, scoring, and constraint evaluation get fast.
• Lean: boundaries. We write constraints as code and produce proofs and checkers that can reject unsafe actions.

LLMs should dominate interface and checkers should rise wherever constraint satisfaction matters. Build the castles out of affiliate links and free trials if you want, as long as the sandbox is formally verified.

Don't hate the TC, hate the game

Published January 27, 2026

Quit playing games (with my heart)

A bad tech interview does not fail because the problem is hard. It fails when the assignment allows one thing and the evaluator rewards another. The most common mismatch, in my experience (having been on both sides of the table), is between what the assignment allows and what the interviewer rewards, especially when amplified by cognitive bias.

Live problem solving further amplifies this: candidates end up reading the room instead of the problem. This can be favorable for room readers, at least before the pandemic, but room readers now have to become Zoom readers (or Google Meet readers).

If you want the short version, it is this: the Anthropic take-home is a vivid example of a broader interview problem. The stated task was kernel optimization. The harness quietly made another game possible: inspect the deterministic state, use the mutable debug_info() hook, and route around much of the kernel work. The speedrun comparison makes the same point from another angle: public communities eventually separate glitchless runs from exploit runs, while hiring loops usually pretend those categories are obvious even when they are not. If you want the full case study and video comparison, open the section below. Otherwise, skip to the conclusion.

Always one step behind

Modern tech interviews often lag behind the work they claim to measure. The pattern is old: real work moves to a higher level of abstraction, and hiring keeps testing the old bottleneck.

There was a time when low-level control over machine details was a much larger part of everyday programming. Then compilers got better, languages like C raised the level of abstraction, and raw assembly skill stopped being the right default proxy for whether someone could build useful systems. Many hiring loops still treat the previous layer as proof of seriousness after the work itself has moved.

The same thing happened again with search. Once Google made recall cheap, a lot of engineering value shifted away from memorization and toward problem framing, system judgment, and synthesis. Interviews did not respond by testing those things directly. They responded by hardening around LeetCode, whiteboard puzzles, and contrived pressure tests. In other words, they turned the previous generation's bottleneck into a ritual.

AI coding tools are now doing to the Google-and-LeetCode era what Google did to memorization and what C did to assembly. They are raising the abstraction level again. A candidate who can use modern tools to understand a codebase faster, search a design space better, or articulate a sharper systems tradeoff is often more valuable than a candidate who can manually simulate yesterday's constraints on demand. But many interview loops still reward the simulation.

That is the failure mode: confusing friction with signal. Something can be hard without being the right thing to test, and a puzzle can be easy to score without revealing much about engineering. Hiring systems keep arriving one step behind the work, then mistaking that lag for rigor.

My bias is toward level-aware interviews, and toward formats like "present a real project" or "teach me something you know well." Those reveal judgment, depth, clarity, and tool use more honestly than most puzzles, especially if you evaluate whether the candidate can make the other person smarter rather than perform an older constraint.

Tensors are recursive cubes all the way down

Published January 5, 2026

Tensor intuition

A tensor is a grid of numbers with named axes. The hard part is not the word; it is remembering which axis you are touching. You do not need a PhD in quantum neutrino fields to follow transformer shapes. You need a good indexing story. The dimension count is how many indices you need to locate one value: a scalar is a point, a vector is a line, a matrix is a plane, and adding an axis gives a cube. Add time or batch and you are in 4D and 5D. Add a few more dimensions and maybe you are studying superstring theory. These are still coordinate lists, but our spatial intuition starts to fade.

Use +1D/-1D to step through dimensions. 0D is a number, 1D a line, 2D a grid, 3D a cube. At 6D you get a cube of cubes. A 6D index [i, j, k, x, y, z] works in two stages: [i, j, k] finds the outer cube, [x, y, z] finds the inner box.

Tesseract projections and hypercube wireframes are accurate but rarely useful when you are learning tensor indexing. Space is 3D, and projecting more axes onto a flat screen loses structure. A better approach is to visualize indexing instead of space.

The point is not to see 6D space. The point is to see how a coordinate finds a value.

The cube-of-cubes model

The idea comes from Feiner and Beshers' 1990 paper on "Worlds within Worlds." Instead of drawing more orthogonal axes, treat the 3D cube as a unit. Higher dimensions organize cubes inside larger cubes, like the giant alien playing marbles at the end of the first MIB movie.

Dimensions 0-3 build the unit cube: point, line, plane, volume.

• 4D-6D arrange cubes into a line, grid, then cube of cubes.
• 7D-9D repeat the pattern with the 6D blocks.

Why this helps in code

This model aligns with how we slice data in PyTorch or NumPy. For a 5D tensor with shape (batch, time, channel, height, width), common operations map cleanly:

• tensor[0] picks one 4D block from the batch.
• tensor[:, :, :, h, w] selects one spatial position across batch, time, and channels.
• tensor[:, 0, :, :, :] takes the first frame of each video.

The hierarchical framing helps you debug reshape errors by making it explicit whether your dimensions multiply correctly. If tensor.reshape(32, 8, -1) fails, you can reason about it as "I have 32 outer blocks, each containing 8 sub-blocks, and I want to flatten everything inside each sub-block." The numbers either multiply correctly or they do not.

An inference kernel is a small function that runs across many elements in parallel on a GPU. Think tiles: a launch grid picks an outer block and threads walk the inner block, so grid indices select the outer cube and thread indices select the inner voxel. Striding the wrong axis pulls the wrong inner cube, and mismatched block shapes under or over cover the grid.

This trick is only a lookup story: three numbers pick the outer cube, three numbers pick the inner cube, and the pattern recurses as needed. For transformer inference, see Attention thrashing or Forgetting is a feature.

Our intuitions are built for stacking containers, not visualizing hypercubes, so the recursion trick works better than pure geometry.

References

• Feiner, S., & Beshers, C. (1990). "Worlds within worlds: metaphors for exploring n-dimensional virtual worlds." Proceedings of UIST '90.

Ships in the night

Published January 1, 2026

Software keeps adding layers where humans stop writing the lower form directly. C let engineers stop hand-writing most assembly. Agents are doing the same thing to source code: intent becomes an artifact, the artifact becomes a diff, and the diff becomes something to check.

The job does not disappear. It moves. Source code becomes less like the primary work surface and more like an audit target: something you read when the interface, tests, types, traces, or behavior say the generated system is wrong.

The center of work moves from typing lines to shaping intent, reading diffs, and enforcing constraints.

From files to diffs

Most IDEs still treat the file as the main object. Agent tools bolt panels onto that world, but the interesting object is becoming the diff. What changed? Why did it change? What evidence says the change is valid?

A file is where the result lands. A diff is where the claim lives. It says this system should move from state A to state B, for this reason, under these constraints. That is a better review unit for generated code because it keeps the human focused on intent, boundary behavior, and evidence instead of every line produced along the way.

Specification as skill

"Build me a login page" is a wish, not a specification. The missing details are where the agent makes choices for you: authentication method, session handling, error states, rate limits, accessibility, mobile behavior, integration points. Sometimes those choices are fine. Often they are just unreviewed product decisions.

The wish:

Build me a login page.

The specification:

Build a login page:

Auth: email/password via /api/auth, JWT in httpOnly cookie, 24h expiry
Session: redirect to /dashboard on success, preserve ?redirect= param
Errors: inline messages, rate limit countdown, retry on network failure
Security: 5 attempts per 15 min, CSRF token, no credentials in localStorage
Accessibility: labels, screen reader announcements, focus management
Mobile: 44px touch targets, viewport-safe layout
Integration: use Button/Input from /components/ui, follow theme vars

The skill is not writing more words. It is closing the branches that would let the agent build the wrong thing. The familiar senior-engineer skill still applies: define the target, constraints, and acceptance checks before delegating the implementation.

The goal is rarely maximum specification. Contracts, types, and tests encode the parts that matter and reduce revision cycles.

Swarm and IDE shift

Most agent stacks still run through hub-and-spoke APIs. If multiple agents are local, routing every message through a distant service adds latency and cost, and a local mesh is often enough for coordination. The human role shifts to setting the goal, constraints, and acceptance criteria, then reviewing outcomes. A swarm can split the labor across scanning, implementation, tests, and security. They stay aligned through shared context and ensemble consensus. That coordination needs shared intent formats that are explicit and checkable, not just prose. Projects like Reploid explore recursive verification loops for agent workflows.

The text editor remains a line-level view for reading, surgical edits, and debugging. The primary surface becomes a bundle of workflows. Intent specification: define what you want, not every step for getting it. Execution monitoring: watch agent activity, disagreements, and progress. Review queue: inspect proposed changes ranked by confidence. Knowledge graph: see the codebase as the swarm understands it.

The transition

Most tools remain hybrids: a text editor with AI layered on top, leaving visibility and trust problems unresolved. Agentic CLIs are exploring that space, but source code is becoming an audit target instead of the main surface.

The new core skills are specification precision, intent review, and constraint design. Just as C moved most engineers above assembly, the agent layer moves engineers above routine source production. The work shifts to defining what counts as valid output.

Attention thrashing, part 2: forgetting is a feature

Published December 31, 2025

A production service cratered when a simple in-memory HashMap cache grew without limits. Entries never expired, allowing the heap to climb, GC to spike, and the JVM to eventually hit OOM. The repair was a bounded cache with TTLs instead of a larger heap. A context window without eviction is not memory. It is a cache leak with better marketing. Magic.dev announced 100 million token context windows. Anthropic and Google keep pushing toward longer contexts too. The race is capacity; the missing primitive is invalidation.

Context decay

Stanford researchers tested multi-document question answering and found a U-shaped curve: accuracy was high at the start and end, low in the middle. Chroma's research on "context rot" shows why: as context grows, attention spreads thin and the middle gets drowned, so a 100K window can underperform a 10K window.

More context helps only when the right tokens stay reachable. Otherwise the window gets larger and the answer gets worse. A smaller window with relevant state can beat a larger one that buries the query under stale evidence.

Without versioning or TTLs, contradictions accumulate and the model has no native way to resolve them.

At first the drift is easy to miss. At scale it piles up until the failures are obvious.

Relevance decays faster than tokens accumulate. A refactor or bug report can be superseded within days, so old context becomes misleading. Retrieval-augmented systems attempt to solve this with embedding search, but RAG retrieves by semantic similarity, not temporal validity, so stale facts can win.

Compression vs forgetting

DeepSeek's Multi-head Latent Attention (MLA) compresses the KV cache, and follow-up research shows large reductions with small performance drops. DeepSeek-V2 with MLA reduced KV cache size by over 90% while improving throughput, and follow-up work showed models could be converted with minimal quality loss. Compression makes the cache cheaper. It does not decide what should expire.

MemGPT treats forgetting as a feature, and research on long-running agents shows decay strategies outperform naive accumulation.

Simulate a month-long session and compare memory strategies. The "accumulate all" baseline drowns in stale state, while decay-weighted and summarization strategies maintain retrieval accuracy.

Long-context architectures try to avoid O(N^2) costs by skipping attention pairs or rethinking attention entirely. Their LTM-2-mini model avoids standard transformer attention, but cheaper attention still does not solve relevance.

Active memory

Eviction policy is only the visible part. Context windows store old and new statements side by side without recording which one supersedes the other. If you tell the model "the API endpoint is /v1/users" and later say "we migrated to /v2/users," both facts persist with no versioning signal.

Useful long-term memory needs more bookkeeping than a longer prompt: time, source, scope, and replacement rules. When context changes, old entries need to be rewritten, demoted, or invalidated.

More token capacity does not provide that bookkeeping. The system needs a way to decide whether a stored statement still applies after later evidence arrives.

When a Linux kernel hangs, you attach an eBPF probe and trace syscalls in real time. LLMs need similar observability; you cannot manage context you cannot see aging.

Context windows need explicit eviction rules: which tokens expire, which facts supersede earlier facts, and which evidence stays pinned. The industry can scale capacity faster than it can invalidate stale state. A smaller, cleaner context window can outperform a huge one packed with stale contradictions.

The industry keeps pushing longer context because it is a tractable engineering problem. Active state management is harder because it requires the model to update, invalidate, and version facts as the world changes. A longer context window without invalidation is just a larger HashMap. The feature is not remembering everything. The feature is knowing what to forget.

Cloud's metal linings playbook

Published December 29, 2025

Cloud won because it centralized hard things: giant models, expensive accelerators, managed deployment, and fast iteration. Local inference wins a different set of tradeoffs. NVIDIA's Blackwell chip draws 1,200 Watts; a MacBook Air is about 30. That gap matters when the task is a tight loop rather than a frontier-model benchmark.

Local inference is a routing choice: predictable latency and no external dependency in exchange for lower peak capability. With a quantized model on Apple Silicon, responses can be consistent because there is no network variance, no API rate limits, and no cold starts. Training and frontier reasoning still belong in the cloud; routine inference, local context, and privacy-sensitive work pull toward the edge.

Latency and cognition

Nielsen's research laid out response time thresholds decades ago, and Miller's capacity limits explain why latency erodes working memory. The Doherty Threshold puts the break around 400ms, which is where AI assistants often live. A model can save keystrokes and still break the thought loop.

When you are deep in a refactor, you juggle a handful of things in working memory: the function signature, call sites, the failing test, the edge case, the architectural constraint. A 400ms pause gives those items time to fade. By the time the AI responds, you have lost the edge case. Latency breaks the working-memory loop.

Performance constraints

Datacenter GPUs run with active cooling while laptops throttle quickly. On fanless hardware, clocks drop as temperatures rise, so watts per token become sustained performance. Watts per token functions as both an energy constraint and a performance ceiling. Quantization helps because it reduces memory bandwidth pressure, even when dequantization eats some gains, and the net improvement still makes edge inference viable.

Running inference locally via WebGPU brings cross-platform deployment and browser sandboxing, but WGSL and WGSL subgroups still trail CUDA on performance and consistency. CUDA gives developers tensor-core paths and async memory controls that can overlap compute and transfer. WebGPU can use fast hardware through the browser's implementation, but portable WGSL cannot assume CUDA-style tensor intrinsics or vendor-specific scheduling controls. It is a portability tax: one runtime target across browsers and devices, but lower peak throughput and less explicit control than CUDA.

Model the thermal constraints of edge inference. Set target tokens per second and model size, then watch junction temperature climb as inference runs. Compare power envelopes (fanless laptop vs desktop vs datacenter) and see what determines sustained throughput.

Routing and coordination

This is a routing problem, not an edge-versus-cloud purity test. Keep short-loop tasks local, use local ensembles for higher-stakes work, and send the hardest or most novel tasks to the cloud.

Tier 1 is autocomplete, small refactors, and documentation lookup. Tier 2 is multi-file refactors, test generation, and code review. Tier 3 is novel architecture questions and deep debugging that still need cloud-scale models.

Hybrid routing usually assumes one device. Once you have several local agents, they need to sync with each other instead of bouncing everything through a cloud API.

Conflict-free Replicated Data Types (CRDTs) solve one part of this for databases: shared state can converge even when updates arrive in different orders. Local agents need that kind of state synchronization, plus separate policy for trust, permissions, and conflict handling. Local-first means your laptop, your phone, and your home server can sync context without routing every update through a cloud hub.

The edge trade-off

Cloud inference will keep the highest peak capability, but latency, privacy, offline use, and cost pull routine work to the edge. The question is which tier fits the task: autocomplete, small refactors, and documentation lookup can often stay local; novel architecture questions, subtle race conditions, and system design still benefit from cloud-scale models. The edge imperative is not to replace the cloud. It is to stop sending every loop through it.

Those who can, teach students to teach

Published December 12, 2025

The useful story is not "small models beat frontier models." It is narrower and more interesting: expensive models search, smaller models learn the traces, and checkable tasks make the transfer measurable. By late 2024, smaller models and simpler training loops were getting strong results on tasks with checkable answers. The PPO-based RLHF pipelines that dominated 2023 were no longer the only practical route. MIT reported orchestrated small models approaching frontier benchmarks, DeepSeek released GRPO, and Karpathy called it part of "the de facto new major stage in LLM training." The practical thread is this: small models can cover more verification-heavy work when the task supplies ground truth.

Simpler training loops

PPO works, but it is heavy. You need a policy model, a reference model, a reward model, and a value model. The value model becomes the bottleneck. That drives memory and tuning cost, and Sebastian Raschka's survey calls out the domain knowledge and compute required.

Group Relative Policy Optimization takes a different route. It samples multiple completions, scores them with a reward function, and uses the group mean as the baseline, so each completion's advantage is its score minus the mean. No learned value function, no second model.

The direction is toward fewer moving parts: from PPO's four-model loop, to DPO and GRPO, to GRPO plus verifiable rewards where the task itself can score the answer.

Verifiable rewards

RLVR works because some tasks have ground truth such as GSM8K and LeetCode, so you can verify correctness instead of training a reward model. Reported results show strong pass@1, and the Awesome-RL-for-LRMs repository tracks the rapidly expanding set of reproducible papers. DeepSeek-R1 helped popularize this style of verifiable-reward training for reasoning tasks.

Ensembles and consensus

Training your own reasoning model is one path. Running multiple existing models and aggregating their outputs is another. The MIT CSAIL DisCIPL framework uses a large model to plan and small models to execute, reporting shorter reasoning chains and lower cost while keeping accuracy close.

Compare reasoning approaches by cost and accuracy. Adjust model sizes, ensemble counts, and task difficulty to see where a single large model wins and where ensembles take over. The crossover depends on your accuracy bar and budget.

Iterative Consensus Ensemble (ICE) pushes this further. Three LLMs generate responses, critique each other's answers, and iterate until they converge or agree to disagree. On GPQA-diamond, a difficult science QA benchmark, the paper reports an accuracy improvement without fine-tuning or gradient updates, just inference-time deliberation.

The mechanism matters because disagreement carries signal; a single model is less reliable at separating confident answers from confidently wrong ones. High-confidence agreement suggests reliable answers while persistent disagreement flags ambiguity. Research on ensemble voting shows diversity can matter more than any single model past a point.

Distillation and limits

Ensembles are expensive at inference time, but their behavior can be distilled back into a single model. TinyLLM demonstrated multi-teacher distillation where a student learns reasoning patterns and can even outperform individual teachers.

The loop is practical because each stage produces evidence. Train a base model with GRPO and verifiable rewards, use ensembles for high-stakes queries, then distill their consensus back into a smaller model. Frontier models still lead, but smaller systems can borrow their search behavior and ship it cheaper.

The Kessler Syndrome analogy for code is synthetic data drift: as agents generate large volumes of outputs that are good enough to pass immediate checks, human ground truth becomes a smaller share, and errors compound silently as diversity shrinks. A bad pattern can be copied into tests, examples, and follow-on training data until the system starts rewarding its own residue. Data hygiene becomes a prerequisite.

RLVR only works on verifiable tasks, and ensembles depend on strong base models, so novel or unconstrained reasoning still depends on frontier-scale training. For narrow, measurable tasks, the frontier model increasingly acts like training infrastructure rather than the only deployable product.

Thinking fast and MoE

Published November 30, 2025

A dense model wakes up every weight for every token. A Mixture of Experts model does not. It keeps many specialist networks available, then routes each token through only a few of them. That is the promise: more total capacity without paying the full compute bill on every step. DeepSeek-V3 has 671 billion parameters with 37 billion active per token. DeepSeek's technical report frames this as training capacity diverging from inference cost.

Across recent releases, total capacity keeps growing while active compute stays much smaller. The catch is systems work: routing only helps when the serving stack can keep the right experts close enough to feed the token.

MoE routing basics

Mixture of Experts replaces dense feedforward layers with specialist networks. The router selects a few experts per token; the others stay inactive. Google's Switch Transformer made this blunt by routing to one expert and cutting training time while maintaining quality.

This ratio matters because arithmetic cost tracks active parameters, not total capacity. That is why MoE models can look enormous on paper while still being practical for some inference workloads.

The cost is routing and memory residency. The model has to choose experts, keep weights available, batch uneven routes, and move the right data at the right time. That overhead can eat into the savings.

Economics and balance

DeepSeek-V3 trained on 2,788,000 H800 GPU hours at an estimated cost of $5.6 million. Llama 3.1 405B trained for 30,840,000 GPU hours. Those are not controlled comparisons, but they show why sparse architectures changed the cost discussion for frontier-scale training. NVIDIA's analysis reports up to 10x faster throughput and about one tenth the cost in those comparisons, with workload and implementation caveats.

Mixtral 8x22B made the case concrete. Eight expert networks, 22 billion parameters each, 2 experts activated per token. Total parameters: 141B. Active parameters: 39B. The result was faster than dense 70B baselines while staying competitive among open-weight models at the time of release.

Sparse activation makes compute scale closer to the active parameter count, with the router acting as a selector that decides which experts handle each token. Memory footprint, routing overhead, and serving complexity still depend on the full expert set.

A significant MoE failure mode is expert collapse: if the router keeps sending most tokens to a few experts while others idle, you end up with a sparse model that behaves like a dense one. Traditional fixes add auxiliary losses that can fight the main objective, and DeepSeek-V3 reported an auxiliary-loss-free strategy for load balancing.

Trace token routing through an MoE layer. Different tokens light up different expert combinations. When routing skews, utilization drops and effective capacity shrinks. Router quality decides whether you realize the efficiency or drift toward expert collapse.

Inference and distillation

Sparse models trade arithmetic efficiency for more complex serving. An MoE model has to fetch different expert weights per token, which keeps memory bandwidth pressure high even if compute scales with the active slice.

DeepSeek-V3 tackles this in part through Multi-head Latent Attention (MLA), compressing the KV cache by over 90% and enabling efficient handling of 128,000-token context windows. That reduces attention-cache pressure; it does not remove the need to keep expert weights available for routing.

Memory footprint stays large because every expert still has to be resident even if only a few run per token. For serving workloads where memory is available but per-token compute matters, MoE can be a good trade.

If inference complexity is prohibitive, MoE models offer a fallback: distillation. Train a sparse teacher with high capacity, then compress into a dense student for deployment. Switch Transformer research showed that 30-40% of the sparsity gains could be preserved when distilling back to a dense student, so you pay for routing during training but ship a simpler model.

Architecture shift

Recent frontier releases increasingly use MoE or related conditional computation. Conditional computation is no longer just a lab idea. Sparse scaling adds capacity while limiting per-token compute, but it is not automatic speed. Routing strategy, expert count, memory residency, and batching decide whether the theoretical savings show up in production.

Attention thrashing in long-context models

Published June 26, 2025 • Updated October 10, 2025

Server thrashing in traditional infrastructure often has a blunt fix: add more RAM. When my team ran into it at Google, we weighed two options: fix the root cause at significant engineering cost, or restart the jobs with more memory, which was cheap at our scale. We took the cheaper path and moved on.

LLMs do not get an equivalent simple fix. More memory can hold a larger KV cache and support longer prompts, but it does not decide which tokens deserve attention. In standard full-attention transformers, long prompts create quadratic prefill work; during decode, each new token commonly reads a KV cache that grows with the context. The overhead can dominate the useful signal. A longer window gives the model more places to look, not a better reason to look in the right place.

Scaling pressure

In standard full attention, each token can attend to every other token. For N input tokens, prefill does O(N²) complexity. Implementations such as FlashAttention avoid materializing a full attention matrix, but the pairwise dependence still creates scaling pressure. During decode, the KV cache grows linearly with context, and each new token typically reads prior keys and values. At long contexts, bandwidth often dominates.

To avoid recomputing prior context during generation, transformers maintain a Key-Value (KV) cache that stores prior attention states. Sparse, sliding-window, paged, and compressed-cache implementations can change constants or visibility, but the basic pressure remains: longer retained context increases work and retrieval burden. Performance slows down, and quality can degrade too. The failure is not only cost; the model can also retrieve the wrong thing. In Needle-in-a-Haystack tests, models show the "Lost in the Middle" phenomenon: they can retrieve information from the beginning and end of context but miss what's in the middle.

Try this needle-in-haystack test. At 25K+ tokens, models often miss quotes from middle positions despite "seeing" the entire context.

Thrashing mechanics

Analogous to OS memory thrashing, attention thrashing happens when the model spends more work managing context than using signal. At long context lengths, research shows models can spend significant compute on irrelevant tokens while mid-context retrieval fails, and prefill latency degrades from sub-second at short contexts to 20-60 seconds at 100K+ tokens on H100 GPUs. This is a limit of standard long-context attention, not a lack of RAM alone. Long contexts increase overhead, create more plausible anchors, and can bury signal in noise.

Researchers are actively trying to mitigate attention's quadratic cost, each trading different constraints.

Push sequence length from 256 to 4K tokens and watch latency and memory demands climb quadratically. Toggle between baseline attention, FlashAttention-style tiling, and block sparse approximations to see how much each mitigation actually relieves. The simulator makes the quadratic growth obvious.

Mitigations

Sparse Attention: restricts tokens to local windows or patterns and reduces O(N²) cost. FlashAttention: keeps data on chip to reduce memory traffic. Retrieval-Augmented Generation (RAG): retrieves relevant passages before model processing to shrink the context and focus attention.

These optimizations reduce the cost of carrying context. They do not decide what deserves to remain there. Sparse attention can skip pairs, FlashAttention can reduce memory traffic, and RAG can move selection upstream. None of those mechanisms, by itself, versions facts, expires stale evidence, or decides that a newer instruction supersedes an older one. Research into hierarchical memory systems like HMT and alternatives like Mamba is promising, but most production transformer systems still pay a substantial cost for long retained context. Research confirms "model attention becomes increasingly unreliable as input length grows," which is why longer windows need selection and memory policy, not only more capacity. The next problem is not just attention cost. It is knowing when old context should be forgotten.

Whose bits are wiser, GPU | TPU?

Published May 12, 2025 • Updated September 7, 2025

In June 2018 I got a live question in at Google's TGIF in Mountain View, a few weeks after TPU v3 was announced at I/O. I asked Sundar Pichai who was using Cloud TPUs outside Google. He said he did not know and promised to follow up.

A few days later, an email arrived with about two dozen customer names. I did not recognize most of them. I took it as a sign that TPUs were still specialized and not yet a general-purpose default. In production terms, that usually means "best fit for a narrower workload class, less likely a one-stack default."

That is still the useful frame. The question is not whether GPUs or TPUs are wiser in the abstract. The question is whether the workload, compiler stack, interconnect, memory footprint, and fleet economics line up.

Architectural differences

Modern GPUs group cores into Streaming Multiprocessors that run many threads in lockstep. Warps of 32 threads execute the same instruction.

On-chip registers and shared memory keep hot data close, with High Bandwidth Memory (HBM) for capacity. NVIDIA added Tensor Cores starting with Volta to accelerate matrix math.

MLPerf Training v3.0 reports H100 performance on GPT-3 scale workloads at multi-thousand GPU scale.

When the workload keeps changing, flexibility usually wins. Divergent control flow can hurt warp efficiency, but the programming model still lets you handle irregular ops, custom kernels, and changing model architecture in one stack. That practical reach is why GPUs dominate open ecosystems and keep becoming the default for early-stage production moves.

TPUs are built around systolic arrays to keep data moving and reduce memory traffic for matrix math.

Matrix A enters from the left. Matrix B enters from the top. Each PE does multiply-accumulate in lockstep. The path is predictable, which is one reason TPUs can be efficient at matrix math. They trade flexibility for simpler data movement.

TPUs are ASICs built for matrix multiplication. The first generation TPU featured a 65,536 8-bit MAC systolic array delivering 92 TOPS peak throughput.

Beyond raw compute, TPUs optimize for low-precision arithmetic (INT8, bfloat16). Many workloads tolerate reduced precision, trading some accuracy for speed and power efficiency. Google scales TPUs into "Pods" using custom torus interconnects. Specialization limits flexibility, and in the first-generation inference setting it delivered 30-80x better TOPS/Watt gains on the evaluated neural-network inference workloads.

TPUs trade control flow flexibility for efficient dataflow, keeping operands moving across a grid where they work best for dense matrix operations. The trade-off is real: if your stack is stable and matrix heavy, TPU can be efficient; if your model keeps introducing control-flow exceptions or shape-shifting operators, TPU mapping gets painful fast.

From a developer standpoint, GPUs are usually easier for mixed stacks: data preprocessing, custom ops, control-heavy models, and kernels that change often. TPUs excel when the workload maps cleanly through XLA/JAX onto dense linear algebra. That makes them powerful for large training runs, but less forgiving when the model has irregular components.

Performance tradeoffs

Peak numbers miss most of the production story. They also mix vendor conventions: NVIDIA's H100 figure above is the Tensor Core peak with sparsity, Google's TPUv5e figure is per-chip BF16, and AMD quotes dense plus separate sparsity peaks. Memory bandwidth, interconnect, software ecosystem, and model shape determine throughput on real workloads. The table is a useful shorthand, not a benchmark.

Mixed Accelerator Fleets

Mixed fleets are already normal, and the lines are getting blurrier. GPUs keep adding matrix hardware and memory tricks, while TPUs keep adding software layers. AMD's MI300X, Cerebras, and Groq show the range. GPU versus TPU is not a sports rivalry. It is a placement decision. Flexible kernels, irregular operators, and fast ecosystem support pull toward GPUs; stable matrix-heavy workloads with compiler-friendly shapes can justify more specialized silicon.

Déjà vu makes a cog in the machine a cognizant machine

Published May 26, 2025 • Updated August 31, 2025

Déjà vu is a source-tagging failure that gets caught. A fast familiarity signal fires, but a slower check flags that the moment is new. It feels strange but harmless because the brain catches the mismatch. You feel the familiarity, but you know it is wrong.

That correction step is what keeps déjà vu from taking over. When the check fails, false familiarity can become persistent. Neurologists document cases of chronic déjà vécu ("already lived"), where patients insist novel experiences are genuine memories, unable to flag the conflict their monitoring system should catch. Standard LLM generation does not have an intrinsic memory-conflict monitor like this.

Memory mismatch and monitoring

Neural evidence points to the medial temporal lobe. The parahippocampal cortex provides rapid familiarity signals; the hippocampus processes slower, context-rich episodic memories. fMRI studies show activation in regions associated with memory conflict during déjà vu reports, while hippocampal signals correctly flag novelty.

The gap in timing matters. Familiarity is fast and cheap, recollection is slower and precise, and monitoring decides whether to trust the signal.

Temporal lobe epilepsy patients report déjà vu as seizure auras. Direct stimulation of the entorhinal cortex can reproduce the sensation, which fits a mismatch between a fast familiarity signal and slower episodic recall.

Dual-process models frame déjà vu as successful error detection. The familiarity pathway misfires, but metacognition flags the conflict, preserving awareness that the experience is novel. When monitoring fails, as in chronic déjà vécu, patients insist entire episodes are genuine memories.

Triggers and AI parallels

Diary and clinical studies identify specific triggers. Novel environments with partial feature overlap activate familiarity without full context match: similar spatial layouts, familiar lighting patterns, or recurring sensory cues. Fatigue, stress, and hyperexcitability degrade temporal synchronization between pathways. The mechanism is straightforward: a cache hit on stale data, where the tag matches but the retrieved content doesn't align with what the slower, higher-capacity system knows.

The failure mode is confident familiarity without a check. In machines this looks like an embedding match that feels right while the ground truth has drifted underneath it.

The machine analogy has limits. Transformer attention, embeddings, KV cache, and human episodic recall are not the same mechanism. The useful parallel is stale high-similarity evidence: a model can lean on context that matches the prompt surface while the underlying fact has drifted. Without provenance and version checks, that stale match can still produce a confident answer. Retrieval-augmented systems add external memory, but retrieval alone does not decide whether a retrieved fact is current, superseded, or contradicted by newer evidence.

The X-axis is familiarity (parahippocampal signal). The Y-axis is recollection (hippocampal evidence). Monitoring expands and contracts the alert band, showing when a familiarity spike becomes a conscious conflict. Adjust the sliders to watch points migrate between zones and see where current systems often land: high familiarity confidence with weak internal correction.

Conflict detection

Most deployed transformer stacks rely on external verification rather than built-in conflict detection. Alternative architectures like Mamba rethink sequence processing and reduce quadratic attention costs, but they do not automatically add provenance, invalidation, or contradiction handling. The missing piece is a stale-match alarm: a reliable way to flag when a confident match conflicts with newer evidence.

External checks help after generation, but earlier conflict detection would make confident stale-context errors easier to catch. The goal is not to make a model feel less familiar with old evidence. It is to notice when familiarity is no longer evidence.

Optical interconnects: free or bonded?

Published August 26, 2025

AI hardware is running into a movement problem. Arithmetic keeps getting faster, but the system still has to move bits between chips, switches, racks, and memory. Wavelength division multiplexing sends multiple data streams through fiber by encoding them on different light wavelengths. Free-space optics asks a sharper question: what if the path did not need fiber at all?

Free-Space Optics uses VCSEL arrays to transmit through air and aims for very low latency. The deployable path today is switch-level co-packaged optics, while in-package optical I/O targets accelerator fabrics. FSO remains mostly in research labs because micron-level alignment is hard to maintain in real datacenters.

Electrical interconnects degrade beyond about 50cm at 100+ Gbps, and power per bit climbs quickly. As AI workloads push petabytes between accelerators, optical links become increasingly attractive. Switch CPO improves bandwidth density and power by moving optics closer to the switch ASIC. In-package optical I/O pushes that idea toward accelerator packages. FSO tries to remove fiber paths entirely, but depends on tight alignment.

The interconnect wall

Training trillion-parameter models requires moving petabytes between accelerators, but copper traces consume watts per Gbps and degrade beyond 50cm at 100+ Gbps rates. Torus topologies can make it worse, adding cable length and latency that builds up across synchronized operations. The bottleneck is no longer just compute. It is the cost of keeping compute fed.

Co-Packaged Optics (CPO) already has volume-production examples at the switch level. In-package optical I/O is the compute-package cousin: optical engines sit beside CPUs, GPUs, or XPUs to move data off package. Free-Space Optics (FSO), replacing cables with steerable laser beams transmitted through air, targets lower latency but faces alignment challenges that have kept it in research labs.

FSO vs CPO

FSO uses VCSEL arrays to transmit parallel data streams through air. One lab-scale result reports 1.6 Tb/s at 2.3 pJ/bit with sub-nanosecond latency. Steerable beams could reconfigure links without re-cabling, which is attractive for HPC systems with changing topology.

FSO requires maintaining ±5 µm alignment across meter-scale distances in datacenters subject to thermal expansion, floor vibration, and building settling. A 10 µm misalignment can cut power by over 50% and raise bit error rates. Active alignment systems employ piezoelectric actuators with sub-microsecond response, but every link needs dedicated tracking hardware running continuously.

Indoor atmospheric turbulence from equipment heat dissipation causes beam wander and intensity flickering beyond 1-2 meters. Datacenters are not controlled optical labs, so stability is hard to maintain at scale.

Dense VCSEL arrays generate heat causing thermal crosstalk, wavelength drift, and power fluctuations. Dust and condensation can disrupt beams. Hermetic packaging is often required for reliability. Production systems have generally chosen the option that costs latency but gains stability, while researchers continue working on FSO's open problems.

Tune the PID controller that drives MEMS mirrors, then inject thermal drift and dust disturbances. Watch alignment error spike beyond ±5 µm and bit error rate collapse. This is why FSO remains experimental: the control loop must fight continuous mechanical chaos while maintaining Gbps data integrity.

Switch CPO places optical engines close to the switch ASIC, shortening the high-speed electrical path before optical conversion. In-package optical I/O moves a similar idea beside compute dies for accelerator fabrics. Both couple into fiber and avoid some of the reach and power penalties of long electrical SerDes paths.

The fiber-coupled path leverages decades of fiber infrastructure, proven silicon photonics processes, and established packaging techniques. Alignment tolerances relax to ±50 µm, 10x looser than FSO, and hermetic sealing protects optics from contamination.

The cost can be latency. Current CPO modules, like pluggable optics, may use FEC to reach BER targets, adding 100-150 ns per hop in Ayar's comparison. That is acceptable for many network-switch paths, but expensive for memory-semantic fabrics or tightly synchronized HPC collectives where nanoseconds accumulate.

FSO and CPO make different bets:

FSO bets on lower latency and power if alignment can be kept stable at scale. Switch CPO bets on mature fiber packaging and accepts networking latency that is easier to deploy. In-package optical I/O tries to keep the fiber reliability story while moving latency closer to package-level electrical links.

Deployment path

Switch CPO met immediate bandwidth needs using mature fiber technology and reached volume-production examples by 2025. In-package optical I/O is the more relevant path for accelerator-to-accelerator fabrics, where latency and bandwidth density matter more than traditional switch economics.

FSO targets specialized applications such as supercomputing and quantum interfaces, where the latency gains may justify the complexity. MEMS advances, improved packaging, and AI-assisted alignment may enable broader FSO adoption, but datacenter deployments still require breakthroughs in alignment stability. In practice, teams usually pick what they can deploy with existing fiber, racks, and packaging while research continues to test whether FSO can survive real datacenter conditions.

Who's driving the autonomous vehicle shift?

Published May 6, 2025 • Updated August 23, 2025

How many sensors does it take for a car to handle real-world driving safely? Waymo stacks lidar, radar, and cameras to buy redundancy, while its newer hardware reduces sensor count and cost. Tesla centers cameras in most production Autopilot systems and leans on scale data to close the gap.

The disagreement is not really about sensors. It is about the safety case. Sensor maximalists argue that redundancy catches failures when individual systems degrade, while vision-only advocates counter that adding sensors introduces failure modes without safety gains. The industry remains split over whether autonomous systems should verify reality through diverse hardware or sophisticated software.

The vision-only case is still debated, especially for robotaxis, and it never made sense to me as the default safety posture. Why limit machines to what humans have when they can perceive more? Cost matters, but the safety case has to come first.

The core tradeoff

Each company's sensor stack represents a different answer to the same question: how much independent evidence does the vehicle need before it acts? More sensors mean more data to fuse and higher costs. Fewer sensors mean cheaper deployment but narrower perception and less redundancy when conditions degrade.

Sensor stack archetypes

Waymo: layers every available sensor type. Their 6th-generation system integrates lidar, cameras, and radar into a fused perception pipeline. When one sensor fails or provides ambiguous data, others compensate.

The redundancy shows up in Waymo's insurance-claims analysis. Swiss Re analysis of 25.3 million fully autonomous miles uses liability claims as a proxy for at-fault collisions and reports 88% fewer property damage claims and 92% fewer bodily injury claims than the human-driver baseline. The cost is complex calibration, a larger sensor suite, and massive data fusion compute.

Tesla: relies on camera-first Tesla Vision in most regions after beginning the radar removal transition in 2021. The argument: humans drive with two eyes; vision should suffice if processing is sophisticated enough. Tesla trains neural networks on billions of real-world miles collected from consumer vehicles, a scale advantage no robotaxi-only fleet can match. If successful, the approach enables much lower sensor cost and rapid fleet scaling.

For robotaxis, the vision-only case is harder to justify. Human drivers still cause too many fatalities each year, and matching human performance is not the goal. Autonomous systems need to be better than humans to justify deployment, especially when passengers surrender control. Safety-critical systems usually rely on redundancy because failure modes do not align.

Cameras degrade in glare, rain, fog, and snow. These are conditions where lidar and radar can add redundancy. Without range-finding sensors, depth estimation relies entirely on vision algorithms that struggle with textureless surfaces and low-contrast scenes. The strategy trades cost against safety margin. If vision closes the gap, it wins on cost. If not, the margin is thin.

Toggle sensors off and watch composite detection confidence drop. Switch scenarios to see how fog cuts cameras while radar stays steadier, or how night conditions flip the reliability hierarchy. The disengagement rates are derived from actual California DMV 2023 filings.

Middle ground: Cruise (GM-backed) pursued Waymo-style sensor fusion but paused operations after an October 2023 incident where a robotaxi dragged a pedestrian 20 feet. The failure was in decision-making under ambiguity, not sensors. Better perception helps, but planning can still fail in edge cases. Cruise resumed limited testing in 2024, then GM ended Cruise robotaxi funding in December 2024 and folded the work into personal-vehicle autonomy and ADAS.

Aurora targets long-haul trucking on structured highway environments where lane discipline is predictable, speeds are steadier, and pedestrians and other vulnerable road users are rarer than in dense urban robotaxi settings. Easier problem, smaller market, but potentially faster path to commercial viability.

Constraints and outcomes

The "long tail" problem is the hard part: systems perform well for long stretches but can fail on rare edge cases. Proving robustness against rare events defines safety validation. Safety validation also lands in a fragmented rule set: no federal mandate, state and local variation, and accident-driven regulatory tightening.

More sensors cost money. Fewer sensors shrink the safety margin. You cannot minimize both at once. Autonomy is not one technology; it is a safety case, an operating domain, a cost model, and a regulator-facing argument that has to survive the long tail.