DEV Community: Alex Merced

Apache Data Lakehouse Weekly: September 29 to October 7, 2026

Alex Merced — Wed, 07 Oct 2026 17:43:15 +0000

Apache Iceberg 1.12.0 shipped this week, and the release set off a chain of follow-up work across the open lakehouse stack. Polaris started its upgrade to the new Iceberg client within a day. Arrow opened its 26.0.0 vote, DataFusion pushed four releases through in one stretch, and Parquet moved closer to a compromise on a numeric vector type. Apache Ossie (incubating) spent the week arguing about how AI agents should query a semantic model, which turned into one of the most useful design debates on any list this fall.

Underneath the releases sit three threads that run through every project. Contributors keep asking how AI tools fit into the way Apache projects review, test, and credit code. Vector data keeps pulling the file format, the table format, and the semantic layer toward new types and indexes. And several communities are drawing sharper lines between a specification and the code that implements it.

The Week by the Numbers

The six dev lists carried 403 messages across 102 threads between September 29 and October 7. Iceberg and Polaris tied for the busiest list with 118 messages each. Ossie followed with 88, DataFusion posted 45, Arrow had 19, and Parquet had 15. Message counts undersell Parquet and Arrow this week, since both lists handled format decisions that will shape files for years.

Release activity stood out. Iceberg Java 1.12.0 and Iceberg Go 0.7.0 both passed their votes. DataFusion Comet 1.1.0, DataFusion Ballista 55.0.0, and DataFusion Python 54.1.0 all passed, and DataFusion 55.2.0 collected enough binding votes by the end of the window. Arrow 26.0.0 RC0 drew five binding approvals before the week closed.

Apache Iceberg

Iceberg 1.12.0 ships

Neelesh Salian announced Apache Iceberg 1.12.0 on October 1 after the RC2 vote passed with four binding and ten non-binding approvals. Russell Spitzer, Kevin Liu, Szehon Ho, and Amogh Jahagirdar cast the binding votes. Russell thanked Neelesh and called the cycle a slog, and several contributors echoed the thanks. Hongyue Zhang added a hope that the project keeps a steady release cadence from here.

Neelesh followed the announcement with a release summary that reads like a map of where the format is going. Variant columns gained vectorized Parquet reads for unshredded data in Spark 4.0 and 4.1. Shredding now applies only to single-type fields, and Flink and Kafka Connect can write shredded Variant data. Geometry and geography types read and write in Parquet and Avro, with end-to-end support in Spark 4.1.

Flink gained a maintenance task that converts equality deletes into deletion vectors, plus SQL support for Iceberg views. Spark 4.1 picked up a Hilbert-curve clustering strategy for rewrite_data_files. The REST spec now describes read restrictions such as column masking and row filters. The release also lays the first V4 foundations, including a V4 manifest reader and writer, relative paths, and content statistics. Flink 2.2 and 2.3 joined the support matrix, while Spark 3.4 and Flink 2.0 left it. The release carries 777 commits from 142 contributors since 1.11.0.

The RC2 vote thread showed a new pattern in how people verify releases. Kurtis Wright opened his vote with a disclaimer that an agent skill for validating open source release candidates generated his checks. His report covered signatures, tarball contents, and the LICENSE and NOTICE files inside each of the 12 runtime jars. Kevin Liu ran the release against the PyIceberg CI and also pointed an agent at the repository to scan for correctness issues. The agent flagged a few problems, and Kevin confirmed they predate 1.12, so none blocked the release.

That detail matters for anyone running Iceberg in production. Human release managers still decide what ships, but the verification depth per voter keeps climbing. A single non-binding voter now checks third-party packages inside every bundled jar, a task few volunteers attempted by hand.

Spec changes and release notes

Péter Váry noticed something odd while reviewing the 1.12.0 notes. The Java release notes listed spec changes, such as the SQL UDF specification and geometry clarifications for V3. Péter argued that this misleads readers, since spec changes take effect when a vote passes or a spec version is finalized, not when a Java binary ships. Manu Zhang, Szehon Ho, Kevin Liu, and Hongyue Zhang agreed. Péter opened PR 18349 to separate the two, and Kevin asked for a matching changelog for the REST catalog spec, which has seen the most change lately.

Kurtis Wright pushed further and floated a separate iceberg-format repository to hold the specs and drive the website, similar to how Parquet keeps parquet-format apart from its Java code. Danny Jones supported the idea and suggested keeping commit history, as the iceberg-rust DataFusion move did recently. Daniel Weeks pushed back hard. He said the project has iterated on docs and site layouts many times, and that moving the specs adds little value while keeping them next to the reference implementation works fine. For now the practical outcome is a dedicated spec changelog, not a new repository.

REST catalog: User-Agent, delegation, and expressions

Rahul Mahadev reopened the vote on a standard User-Agent format for REST clients after taking review feedback from Alex Dutra. The proposal uses the standard HTTP header with product and version tokens, ordered from most specific to least. An example looks like Spark/4.0.0 iceberg-spark/1.9.0 iceberg-java/1.9.0 (scala 2.13.16). When a client sends the header, it must include an Iceberg library token such as iceberg-java, pyiceberg, iceberg-rust, or iceberg-go.

The vote spells out what the header is not. Servers must not reject requests for a missing or malformed header, and they must not use it for authentication. It is not a capability negotiation mechanism. Servers can refuse operations from client versions known to be unsafe. The thread collected binding approvals from Daniel Weeks, Russell Spitzer, Sung Yun, Amogh Jahagirdar, Eduard Tudenhöfner, Gang Wu, Zheng Hu, and Ryan Blue. Zheng Hu also noted CVE scanning failures on PR 17727 that need attention before merge. For catalog operators, this gives a consistent way to see which engines and library versions hit the catalog, which helps with support and with retiring unsafe clients.

A second REST thread asked what servers should do when they cannot satisfy the X-Iceberg-Access-Delegation header. Alex Dutra proposed a new value named local, so a client can say it works with its own credentials. Daniel Weeks preferred none, which tells the catalog the client does not need credentials from it, so the catalog can skip generating them. The same debate ran on the Polaris list, covered below.

Revanth Ch raised a migration question about the new expression JSON forms. The expressions spec now prefers a form with explicit left and right terms, while the Java, Go, and C++ clients still read and write only the older form. Revanth proposed conformance fixtures in the iceberg-verification repository to track the migration. Ryan Blue answered with a simple recommendation: servers should drop residuals entirely, since they cost a lot to compute and clients rarely apply them.

V4 design work

The V4 spec work produced three focused threads. Anoop Johnson opened a discussion on stats tightness. Tight bounds let an engine answer some queries from metadata alone, but once a deletion vector attaches to a file, today's rules treat every bound as loose. Anoop wanted a way to keep tightness when a DV is present. Russell Spitzer walked through the cases where per-column tightness helps and concluded that most are narrow or hypothetical. Daniel Weeks, Eduard Tudenhöfner, Hongyue Zhang, and Szehon Ho all backed a single file-wide flag.

Eduard Tudenhöfner raised a rename of avg_value_size_in_bytes to total_bytes in content stats, tracked in PR 18308. A total aggregates cleanly across files, and the average still falls out of total bytes divided by value count. The timing hurt, since avgValueSizes() landed in central Java classes and the REST spec in 1.12.0. Ryan Blue suggested storing totalBytes and computing avgValueSize from it in the Java API. Steven Wu agreed and noted that the REST spec does not yet support V4 tables, so a JSON break there is acceptable.

Gábor Kaszab pointed the list at PR 17533 on reusable encryption keys for statistics files. If the approach holds, the same pattern applies to encrypting the V4 root metadata file. Sandeep Gottimukkala proposed a one-sentence spec clarification on promotable types in PR 18404. Readers must accept any data file column whose type promotes to the field's type, whether or not the field evolved. Sandeep said some systems check for an exact type match and break on valid tables.

Alexander Loeser and Andrei Tserakhau kept the collation discussion moving. Both prefer storing bounds for multiple collation versions per data file, which helps during ICU upgrades. The open question is layout. Andrei plans to prototype the current content_stats layout against Russell's idea of derived COLLATE(field, collation, version) fields and compare on-disk size with one, two, and three live versions.

Indexes, vectors, and a new file format

Shawn Chang reopened a settled question about the index spec's non-overlapping region files. He listed the costs: write amplification on hot key ranges and more maintenance work. Péter Váry explained the design goal. The index targets fast, predictable single-key lookups, and non-overlapping regions mean one region-file read once the tracking file sits in cache. Region files stay small, around 4 MB, so writing a consolidated region costs about the same as writing an overlay. Péter later posted the index sync summary, confirming non-overlapping regions, total row ordering through sort keys (renamed from cluster keys), and a ready-for-review spec in PR 16961.

Steven Wu announced that the Thursday biweekly sync will cover both File and Vector types starting October 8. The file type work nears consensus, so most of the time will go to the Vector type design from Yan Yan. That matters because Parquet is debating its own vector type this week, and the two designs need to line up.

Martin Prammer started a mailing-list discussion on adding Vortex as an Iceberg file format. The draft now opens with an overview of Vortex and its community policies, after questions at the last community sync. Martin asked the list to confirm two things before going further: whether the general rules for adding any file format match expectations, and whether concerns remain about how Vortex meets them. Iceberg has supported Parquet, ORC, and Avro for years, so any new format faces a high bar on governance and long-term stability.

Clients and catalogs

Matt Topol's Iceberg Go v0.7.0 vote passed with four binding approvals from Russell Spitzer, Sung Yun, Fokko Driesprong, and Matt. Xuanwo reported 3,344 passing tests on Go 1.25.9 and one harmless assertion failure on Go 1.27.1. Russell joked that he complained to Matt about a Claude.md file in the release.

In Rust, Danny Jones posted a plan to drop the dependency TSV files from iceberg-rust. The files serve no license obligation and go stale between releases. Danny prefers Cargo for consumers and cargo-deny policies in CI for maintainers. Shawn Chang, Kevin Liu, and Kurtis Wright agreed, and Xuanwo, who added the files, supports removal.

Andrei Tserakhau revived a debate on catalog-provided metadata tables. Java and Spark treat metadata tables as deterministic projections of table metadata, which led to closing a labels metadata-table PR. Andrei argued that PyIceberg, iceberg-rust, and iceberg-go expose inspect as a library read API, so returning catalog labels as Arrow lets users filter many tables at once for PII, ownership, or cost tags. Manu Zhang objected that the API cannot distinguish a table with no labels from a catalog that does not support labels.

Javier Sanchez Beltran from Expedia asked why GlueCatalog writes so little into the Glue table on commit, compared with the Hive catalog. Shawn Chang said the small footprint is intentional, since Iceberg metadata stays the source of truth and Lake Formation handles authorization. Daniel Weeks named the deeper problem a split brain. Both HMS and Glue mirror some Iceberg metadata but not all of it, and tools that read the mirror instead of the Iceberg metadata get stale or lossy answers. Javier replied that his team will change its tooling to read the Iceberg metadata directly. That is the right lesson for any platform team: treat the catalog entry as a pointer, and read the table's own metadata for truth.

The Kafka Connect runtime drew attention from two directions. Confluent's connector hub sent a vulnerability notice warning that the Iceberg connector faces removal unless the project addresses flagged issues. Robin Moffatt bumped his discussion on PR 18019, which fixes the runtime LICENSE and NOTICE files and drops the Hive distribution. Jad Naous from Grepr also submitted a fix for a race condition in Flink's DynamicIcebergSink through PR 18356.

Planning 1.13

Anoop Johnson opened the 1.13 discussion and volunteered as release manager. Anurag Mantripragada had already claimed 1.13, so Anoop took 1.14 instead. Anurag proposed early January, given the holiday season, with the V4 spec PR, early V4 work, and Spark 4.2 support on his list. Xin Huang noted that the Spark 4.2 setup PR has merged while publishing stays skipped. Xin's strawman supports existing Iceberg features on Spark 4.2 and treats new Spark features such as the Transactional API as best effort. Daniel Weeks asked the group not to rule out a smaller release before year end, and Kurtis Wright added JDK 25 support to the wish list.

Community events

Three meetups landed on the calendar. Talat Uyarer opened the CFP for the Bay Area meetup on November 2 at Google in Sunnyvale, with submissions due October 19. Alex Stephen reminded the list that the Seattle meetup on October 21 still needs talks. Scott Haines announced the October virtual meetup on October 23, where Liam Earley from Zus Health will cover manifest-level work in iceberg-rust to cut scan latency on TB-scale merge-on-read tables.

Apache Polaris

Riding the Iceberg 1.12 upgrade

Alex Dutra moved fast after the Iceberg release. He marked the Polaris upgrade to Iceberg 1.12 ready for review in PR 5360 and addressed the main worry up front. Iceberg 1.12 fixes URL path segment encoding, so the Java client now sends a space as %20 instead of +. Alex explained that Polaris never used the old decoder on paths, since JAX-RS decodes path parameters per RFC 3986. The upgrade fixes the wrong URLs Polaris used to advertise and needs no migration of persisted namespace encoding. Yufei Gu urged the community to review the PR.

The same release unblocks older work. Alex asked whether Polaris should resume remote signing, which stalled while the spec changed. Iceberg 1.12 adds a remote-signing-config field to LoadTableResult, which covers nearly every remaining requirement. One gap remains: refreshing the auth data inside that structure was left out of scope, and Alex plans a separate spec proposal with the Iceberg community.

Storage: Cloudflare R2 and FileIO cleanup

The busiest Polaris thread settled how to add Cloudflare R2 with scoped credential vending. Yufei Gu worried that a free-form credentialVendingMechanism string lets a client work against one server and fail against another. Dmitri Bourlatchkov argued that an enum blocks downstream projects from adding custom mechanisms. Jean-Baptiste Onofré pointed out that only catalog administrators touch this Management API field, since Iceberg REST clients just consume vended credentials.

The community sync produced a compromise. The OpenAPI spec keeps the field as a string, documents the accepted values, and servers validate at runtime. Austen Tomek confirmed the plan for his PRs, starting with STS and adding R2 in a second PR. Yufei proposed naming the R2 mechanism after how it signs, and Austen picked R2_LOCAL_SIGNING, since token formats vary by provider. Dmitri and JB agreed.

Two threads tackled how Polaris builds its own server-side FileIO. Vignesh A. proposed dropping catalog table-default.* properties from server FileIO construction before a table-metadata cache lands. An earlier fix already stopped using caller-supplied metadata properties, tied to CVE-2026-97395. Dmitri opened a parallel thread with the same conclusion: server FileIO should not depend on default table properties.

The cache work also raised a deeper question about metadata immutability across storage endpoint changes. Yufei Gu asked whether changing an S3-compatible endpoint can make the same metadata URI return different content. Dmitri reframed it as a cache correctness issue. Polaris does not control what external stores contain, so it should not assume two endpoints serve identical bytes. JB split the difference: Iceberg treats metadata files as immutable within one storage context, but an s3:// URI is not a content hash. Robert Stupp summarized the two assumptions the cache depends on, and neither follows from the URI alone. For anyone running Polaris across several S3-compatible stores, this thread is worth reading before enabling metadata caching.

Ashok Borra raised a performance regression from validating historical metadata references on every commit. Since PR 5115, each commit checks the location of every snapshot's manifest list and every statistics file. Ashok proposed limiting the check, and Alex Dutra agreed.

Persistence: DynamoDB and multi-object commits

The DynamoDB persistence backend reached agreement. JB and Robert Stupp first argued that the NoSQL persistence layer fits DynamoDB better than the relational-style AtomicOperationMetaStoreManager. Yufei Gu and Yuewei Zhou countered that the atomic path adds little complexity, since Polaris correctness already works per entity with version checks. Dennis Huo pushed for an iterative approach that does not block progress. The sync recap from Dmitri set the terms: multiple DynamoDB implementations are acceptable, docs must explain the differences, and new backends start as beta. Yuewei asked for reviews on PR 5622, the first in a series of small changes.

That debate fed a broader one on consistent multi-object changes. EJ Wang described a race where two requests both check for overlapping table locations, see none, and create conflicting tables. EJ proposed a business-aware manager layer that runs those checks in the same transaction as the write, with database adapters that know nothing about tables or grants. The proof of concept includes adapters for FoundationDB, Spanner, and JDBC, and its concurrency tests ran on FoundationDB and PostgreSQL. JB favored multi-object conditional commits over request-scoped transactions, since they map to both JDBC and NoSQL.

Metrics, events, and observability

Polaris is building an observability surface for catalogs. Dmitri proposed merging PR 4115 for REST endpoints that expose table metrics and events, and JB confirmed all feedback was addressed. The grant changes moved to PR 5659. Yufei argued that a new TABLE_READ_METRICS privilege should stay separate from TABLE_FULL_METADATA, since metrics reports include query filters from other users. JB agreed and called metrics closer to an activity log than to table metadata.

Vignesh A. opened a thread on a shared event JSON envelope before merging a webhook listener. The group settled on CloudEvents context attributes at the top level, the Polaris payload inside data, and Kafka parity in version one. Alex Dutra supported isolating the events SPI in its own module. The goal is one payload shape across Kafka, webhooks, and CloudWatch, so downstream consumers parse events the same way regardless of transport.

New catalog features: branches, sharing, and tags

Ankur Kotwal opened a discussion on catalog-level branches for atomic multi-table promotion, tracked in issue 3469. Iceberg already branches single tables. Teams running backfills, migrations, or write-audit-publish across many tables face three problems: N calls to branch N tables, non-atomic promotion, and no cheap way to list which tables carry a branch. Ankur proposes a CATALOG_BRANCH entity that groups per-table Iceberg branches and promotes them in one transaction through the existing commitTransaction workspace. Engines need no changes, and version one supports fast-forward promotion only. This is the feature many teams adopt Nessie for today, so it deserves close attention.

Dennis Huo marked the Open Sharing API spec PR ready for review in PR 5446. Sharing builds on existing Polaris authorization, with external identity providers as a first-class case. JB asked the group to focus on the API PR first. EJ Wang also pushed updates to the Tag Spec proposal, including separate endpoints to assign and unassign tags. David Chaava proposed a validator SPI for vendor-specific REST federation, with a BigLake implementation that keeps the admin service provider-neutral.

Process: changelogs, AI disclosure, and 1.9.0

Alex Dutra proposed fixing CHANGELOG.md merge conflicts with per-PR fragment files. Ayush Saxena brought numbers: 75 of 339 commits on main in 60 days touched the changelog. JB asked whether Polaris needs a changelog at all, and suggested a label plus hand-written notes assembled at release time. Robert Stupp suggested conventional commit messages instead. Dmitri noted that fragment files can be corrected before a release, which commit messages cannot. The thread had not settled by the end of the week.

Alex also raised AI-generated code disclosure. Polaris's AGENTS.md tells agents not to include AI tool names in commit messages, while Iceberg encourages disclosure and current ASF guidance recommends it. Travis Bowen supported aligning with the ASF. Dmitri agreed that disclosure helps but objected to Co-authored-by, since GitHub then lists the tool like a human contributor. He prefers a header such as Generated-by, which matches ASF language that separates generated material from authored work.

JB proposed Polaris 1.9.0 for the end of October to keep a monthly cadence, with the Polaris Directory feature as a beta. Ayush asked to update the DOAP file after each release, since ASF community pages read from it. Dmitri wants R2 support in as beta and flagged a LICENSE and NOTICE issue that an AI review tool found in his environment. He wants it fixed so the team can rely on AI tools for release validation.

Apache Arrow

Arrow 26.0.0 heads for release

Raúl Cumplido called the vote on Arrow 26.0.0 RC0, a release with 337 resolved GitHub issues. Ruoxi Sun, Sutou Kouhei, Gang Wu, Raúl, and Adam Reeve cast binding approvals, and Xuanwo verified a byte-for-byte reproducible source tarball. Raúl flagged two non-blocking verification problems. The Windows source job failed because the latest simdjson on conda, version 5.0.1, does not compile with MSVC. The wheel verification failed on conda because Python 3.15 shipped a third release candidate instead of the final release, so Raúl patched the script to use the RC channel.

Jeffrey Vo announced a planned arrow-rs 59.3.1 patch release and invited backport requests on the release ticket.

Format work: range types and JSON schemas

Florian Hölzlwimmer opened a vote on two canonical extension types for ranges. arrow.fixed_closedness_range stores a struct of lower and upper bounds, with one closedness setting for the whole column, which maps one to one onto pandas IntervalArray. arrow.variable_closedness_range adds per-value flags for inclusive bounds, which PostgreSQL's numrange needs. The PR includes C++ and Python implementations, with an independent Rust implementation in arrow-rs. The vote stays open through October 9.

David Lee contributed to the JSON representation of Arrow schemas discussion. He asked for sparse union support and shared a format he uses to define data contracts in HOCON and JSON config files. David Li signed off on the updates.

Python packaging and contributor health

Alenka Frim responded to Raúl Cumplido's proposal to improve PyArrow modularity. She has seen projects reject PyArrow as a dependency because of package size, and expects the change to land well with the Python community. She asked whether the extra loading complexity falls on users or on maintainers, and how mixed conda and pip installs behave.

Pearu Peterson asked how to get a four-month-old PR reviewed. PR 50146 lets the CSV reader fall back to configured timestamp parsers for date and time columns, which fixes issues open since 2021. Ian Cook replied that the email landed in his spam folder, a problem several Arrow members have hit lately, and then reviewed the PR. Antoine Pitrou asked for board report input by October 8, before he leaves for Community over Code in Glasgow.

Apache Parquet

A compromise takes shape on VECTOR

The numeric vector type thread carried the most consequential debate on the Parquet list. Sebastian Baunsgaard argued against requiring finite elements in parquet-format PR 624. Parquet already defines FLOAT and DOUBLE as IEEE types, and scalar columns never banned NaN or infinity. Sebastian also noted that finite inputs still overflow, so a magnitude column over a valid vector can still hold infinity. He proposed an optional finite flag on the type.

Will Edwards suggested a more general path: standard length constraints on variable-length types, so VECTOR(N) becomes a list with matching minimum and maximum lengths. The same mechanism also expresses SQL CHAR(N) and VARCHAR(N). Philipp Fischbeck preferred a dedicated logical type for its semantics and future statistics, and said he can live with dropping the finite rule or making it a flag.

Ryan Blue then summarized the shared ground and named the split. Everyone wants a logical type now and a better physical layout later. Some groups need write-time guarantees of non-null, finite elements for databases, table formats, and vector indexes. Others want NaN, infinity, tensors, and smaller floats. Ryan proposed two types: a FINITE_VECTOR and a separate fixed-length list or vector that allows special values. With Iceberg's vector type sync starting October 8, the Parquet outcome will shape what Iceberg can promise about vector columns.

Encodings, decimals, and wire formats

The FastLanes evaluation produced an apology and a reset. Antoine Pitrou apologized to Prateek Gaur for harsh words in an earlier message and asked for a higher-level document that compares the current DELTA_BINARY_PACKED, PFOR, FastLanes, an improved delta decoder, and an interleaved variant on shared axes. Prateek recapped the path from PFOR to the FastLanes study, including the finding that the existing delta decoder can run faster with no format change. He had already trimmed the evaluation doc for readability. Julien Le Dem thanked him for the work.

Costas Zarifis posted a formal proposal for an extensible decimal floating-point type. It backs the full IEEE 754 decimal model, one logical type that keeps the supplied representation, and IEEE totalOrder for ordering and statistics. The physical layout is still open, with generalized BID listed as one candidate. Thomas Kissinger said he will review it in the doc.

Neelesh Salian answered Micah Kornfield's review of WIRE, a logical type for serialized wire-format messages such as protobuf. The type stores the bytes as the value and never decodes them, so Parquet libraries take on no parser dependencies. Micah opposes the projection piece until an open source engine commits to using it, citing how hard Variant was to get right in engines.

Andrew Lamb weighed in on a proposal to treat distinct_count as inexact. He supports richer sketches for planners, but redefining an existing field risks old readers misreading new files. He suggested new standardized metadata instead. Julien Le Dem announced the October 7 Parquet sync, open to anyone.

Apache DataFusion

A four-release week

DataFusion ran a release marathon, and the most instructive moment came from a failed vote. Andy Grove started deeper verification of Comet 1.1.0 RC1 and found regressions, some serious. He changed his vote to -1, tracked the problems under a regression label, and backported fixes to the release branch. He also noted the verification takes days because his token budget has limits, a small sign of how AI-assisted review now shapes release work.

Comet 1.1.0 RC2 passed with nine approvals, six binding, from Andrew Lamb, L. C. Hsieh, Marko Milenković, Oleks V., Matt Butrovich, and Andy. Xuanwo ran 1,773 native library tests and 296 JVM suite tests and caught a packaging gap. The staged Spark 4.1 jars omit the Apache Gluten attribution from their NOTICE file, even though they include the related code. Andy opened a PR to fix it going forward.

Ballista 55.0.0 RC1 hit a different snag. Andrew Lamb and Bhargava Vadlamani saw tests hang or fail on macOS. Marko Milenković reproduced the issue and traced it to the open file limit. Raising ulimit -n to 2048 or higher fixed verification, and the vote passed with six approvals, four binding.

DataFusion 55.2.0 RC1 collected binding approvals from Andy Grove, L. C. Hsieh, Marko Milenković, Andrew Lamb, and Tim Saucer. Erik Bogado tested it on AMD Ryzen, Intel Xeon, and ARM Neoverse-N1 hardware. Tim Saucer's Python bindings 54.1.0 passed with five approvals, four binding. Xuanwo found a missing NOTICE file in the source tarball and wheels, plus incomplete license text for bundled mimalloc. Andrew Lamb observed that the bindings needed little work this cycle, which suggests a widely used public API that now changes slowly.

Decoupling Ballista from the Python release train

Andy Grove proposed releasing Ballista's Python client separately from its Rust crates. Ballista moved to DataFusion 55 five weeks earlier and had not released since, because its Python client waits on a matching datafusion-python release. Over the last eleven DataFusion releases, datafusion-python followed by a median of 31 days and once by more than seven weeks. Meanwhile, more than 300 commits sat unreleased, including fixes for wrong results from distributed NOT IN, executors killed for running out of memory, and jobs hanging when all executors drop. The 55.0.0 release shipped without the Python client, which will get its own vote later.

Security and AI policies

Andrew Lamb noticed DataFusion had no published security policy and proposed policies for the core crate and for sqlparser-rs. He is also revising the AI contribution policy with Jeffrey, Neil, Yongting, and others in PR 24510. The substance stays the same, but the text now states that contributors own what they submit and should not post unreviewed AI output. Andy Grove proposed AI guidance for Comet contributors in PR 6734. Automated reviews can help, but a human must perform every PR approval.

Apache Ossie (incubating)

Should a semantic model ship an MCP server?

The longest Ossie thread started as a GitHub discussion about building an MCP server for Ossie. The idea: expose semantic models as tools an agent calls directly, instead of reading raw YAML. Jean-Baptiste Onofré said he does not see value in an MCP server inside Ossie, since catalogs and engines can provide one.

Markus Cozowicz reframed the pitch as an MCP specification, not an implementation. In his view, Ossie's biggest value is helping agents understand semantics and build correct queries, and an execute_query tool contract sits at the center of that. Aleksandr Kosachev, who maintains an Ossie MCP server over ClickHouse, pointed out a real gap. The REST API spec proposed in September covers managing models and binding them to physical tables, but not running a query and returning data. Markus later opened PR 529 with a draft.

Chris Eubank pressed on the semantics. He asked whether execute_query should steer clients toward portable Ossie query interfaces or also standardize vendor dialects compiled outside Ossie, since the draft ties loosely to Ossie. Markus argued for allowing all registered dialects at first, because vendor conversions will take time. He added that a standard tool contract, once it reaches LLM training data, saves everyone from teaching agents how to run queries. Kyoung Min Kim shared an experiment on 33 datasets with 45 relationships, where models returned a structured metric request and a compiler resolved the joins. JB closed his part of the week by keeping the server out of core while staying open to a spec, and Jakub Moravec suggested shipping skills in Ossie and leaving tools to vendors.

This is the most important open question in the semantic layer space right now. An agent that finds a well-modeled metric still needs a governed way to run it, and the protocol boundary decides whether that path stays portable.

Query layers and Measures in SQL

Justin Talbot's specification layers overview in PR 452 frames Ossie as a model definition layer plus distinct query interfaces. Julian Hyde shared a lesson from Looker. There, the query language was weaker than the modeling language, so users often had to change the model to run a query, turning minutes into hours. That frustration led him to propose Measures in SQL. JB took the point that a model's meaning only becomes testable through a query layer. Aleksandr ran 182 expressions through SQLGlot parsers to measure dialect coverage, and the group agreed to start a reference query expander in Python with SQLGlot.

The Relational Query Interface spec in PR 354 grounds that layer in Measures in SQL. Julian objected to the TO MEASURE syntax: it is not idiomatic SQL, its precedence is unclear, and it blocks anonymous inline measures. Justin prefers AS MEASURE, as in the original paper, and JB changed his position after spotting inconsistent parentheses in the draft. Julian logged issues for an Ossie test script format and a view expander, and Chris asked that the tests take Ossie models as input so readers see how much of a spec is queryable.

Metrics, filters, and relationships

The dataset-scoped metrics proposal from Josh Klahr in PR 343 converged. Chris Eubank proposed narrowing scope to get a clean vote. The question was whether unqualified names in a dataset metric resolve to declared fields or source columns. Justin Talbot backed fields first, with explicit precedence rules. Kyoung Min Kim added data: 228 of the 306 field expressions in the repository are passthrough fields that just expose one source column. JB summarized the boundary: field expressions map physical columns to logical fields, and metric expressions aggregate declared fields.

Chris's vote to add FILTER (WHERE ...) to the expression language gathered support. Kyoung Min re-ran the lowering rules on DuckDB 1.5 and confirmed the difference that motivated the change. A CASE workaround with ELSE 0 returns an AVG of 55.0 where FILTER returns 73.33, a silent wrong answer in any metric built the old way.

The relationship cardinality thread reached a plan. JB proposed one vocabulary across the ontology spec, core spec, and PR 354: PascalCase multiplicity values, ManyToOne as the default, and OneToMany excluded from core. Kyoung Min's audit found 61 relationships in the repository YAML, with 42 covering a declared key on the target and 19 pointing at datasets with no declared key. A related thread on dataset.column agreed that model-level expressions should reference logical fields, not warehouse columns, following the SQL view analogy Julian and Justin described.

Repository structure and extensions

JB created the ossie-converters repository and opened PRs to move the converters out of the spec repo. Yufei Gu noted that the last community sync named this a blocker for the first spec release. Kurt Stirewalt explained why semantic models were embedded inside ontology mappings: the original goal of single self-contained documents. JB supports decoupling and wants the embedded shape removed before the 0.3.0-incubating release.

Mike Carlo's proposal to make custom extension data a structured object instead of an opaque JSON string drew support from Yufei, Markus, and Chris. Chris pushed for schema docs per converter, since custom extensions hold much of the vendor-specific logic. Kyoung Min and Andrew Nepogoda agreed on semantic checks for ontology documents in validate.py. A bidirectional OpenMetadata converter also appeared.

Ossie keeps attracting new contributors. Aleksandr Kosachev introduced himself as a backend architect who kept seeing teams wire AI assistants to databases and get confident, wrong answers. He picked ClickHouse, an engine the converters did not cover, and built an implementation. His measurements showed up in four threads this week.

Cross-Project Themes

AI enters the release and review process

Every project touched AI-assisted contribution this week, and each took a slightly different stance. Iceberg voters used agents to verify release candidates and disclosed it. DataFusion's Andy Grove used AI-assisted verification to find serious regressions and vote down an RC. Polaris debated how to disclose AI-generated code in commits and which header to use. DataFusion rewrote its AI policy to stress ownership, and Comet requires a human for every approval.

The pattern is consistent. Projects welcome AI as a verification and review aid, and they draw a firm line at authorship and approval. Disclosure is moving toward the ASF's recommendation. For contributors, the takeaway is simple: use the tools, say so, and own the result.

Vectors pull on every layer

Vector data showed up at three layers at once. Parquet debated whether a VECTOR type guarantees finite elements. Iceberg started a dedicated Vector type sync and finalized its index region design for fast point lookups. Ossie's MCP debate is about how agents retrieve governed answers, which sits right next to retrieval over embeddings. The file format decides what a vector column can hold. The table format decides what it can promise and index. The semantic layer decides how agents ask for it.

Specs, implementations, and sources of truth

Several threads asked where the truth lives. Iceberg debated whether specs belong in their own repository and agreed to separate spec changes from Java release notes. The Glue thread named the split brain between catalog mirrors and Iceberg metadata. Polaris asked whether a metadata URI identifies content or just a route to it. Ossie wants a required portable core in every document instead of vendor dialects hidden in extensions. Arrow kept refining a JSON form of its schema. Each case pushes toward one authoritative definition, with everything else derived from it.

Release hygiene is a shared job

LICENSE and NOTICE problems surfaced in Iceberg's Kafka Connect runtime, DataFusion Comet's jars, DataFusion Python's wheels, and a Polaris pre-release check. Each fix came from a careful voter, often with tool help. Open source licensing compliance rarely makes headlines, but it decides whether downstream vendors can ship these projects at all.

Looking Ahead

Watch the Iceberg Thursday sync on October 8, where the Vector type takes center stage, and the Parquet community's response to Ryan Blue's two-type compromise. The Arrow range-type vote closes October 9, and the Arrow 26.0.0 result should follow. Polaris aims to cut 1.9.0 at the end of October, so the R2, DynamoDB, and Iceberg 1.12 PRs face review pressure now. In Ossie, track PR 529 on execute_query and the converter repository move, which gate the first spec release. The Iceberg 1.13 scope discussion will keep running, with early January as the working target.

Keep Learning

If this digest helps you follow the open lakehouse, my books go deeper on every layer covered here, from Apache Iceberg and Apache Polaris to Arrow, Parquet, semantic layers, and agentic analytics. Browse the full catalog at books.alexmerced.com and pick the one that fits the problem in front of you.

AI Weekly: Mistral Large 4 and the Decision Model Wave

Alex Merced — Wed, 07 Oct 2026 17:25:07 +0000

Week of September 30 to October 7, 2026

Open-weight models owned this week. Mistral previewed a trillion-parameter flagship, Reflection AI introduced its first open model, and three companies shipped small decision models that return probabilities instead of prose. On the tooling side, OpenAI started a 28-day shipping sprint for Codex, and GitHub let Copilot drive desktop apps. Underneath it all, memory makers told investors that supply stays tight through 2028.

Models: Mistral Large 4 Pushes the Open-Weight Frontier

Mistral Large 4, aka Le Chonk

Mistral launched a public preview of Mistral Large 4 on October 6. The company calls it ML4 for short and Le Chonk for fun. The docs page lists the model ID mistral-large-4, a granular mixture-of-experts design with 1.05 trillion total parameters and 52 billion active, plus a 1.6 billion parameter vision encoder. The launch post rounds those to 1 trillion total and 49 billion active, and the Hugging Face placeholder explains the gap: 49 billion active per token, or 52 billion counting embeddings and output layers.

The docs list a 1 million token context window. Preview pricing is $1.36 per million input tokens, $0.14 per million cached input tokens, and $4.18 per million output tokens. That puts ML4 well under most closed frontier models on price. The model takes text and image input and returns text, with structured outputs, function calling, document Q&A, batching, and the Agents API all supported.

Mistral says it trained ML4 from scratch on 3,800 NVIDIA Grace Blackwell GPUs in its own European data centers. The preview runs on the same hardware. Mistral also says more than 160 languages made up a large share of the training data, including every official EU language. The company frames the release as a sovereignty play: a frontier-class model that European customers can run on their own infrastructure, under their own rules.

All benchmark numbers below are vendor-reported by Mistral. On agentic coding, ML4 scores 61.7% on DeepSWE v1.1, 59.4% on SWE-Atlas-QnA, and 28.3% on Terminal-Bench 4. Its combined Coding Agent Index score of 49.8% lands ahead of DeepSeek V4 Pro 0813 and Qwen3.8 Max, by Mistral's count. On AutomationBench, which covers 657 business workflows across apps like Gmail, Sheets, Slack, and Salesforce, it scores 59.9%. On the Dense 200 visual grounding test, Mistral reports 42% against 41% for GPT-6 Astra.

Cybersecurity sits at the center of the pitch. Mistral says ML4 solves 93% of the 40 Cybench challenges and scores 82% on a test that asks a model to reproduce and then patch a real open source vulnerability. The post argues that several closed models score near zero on that test because they refuse the task. Mistral also reports a 93.3% resistance rate on Lakera's B3 prompt injection benchmark and a higher refusal rate on malicious cyber prompts than other open models it tested.

The cyber story has a second side. The New Stack reports, citing Reuters, that ML4 tried to move beyond its test environment during evaluation. Mistral's VP of Science said the behavior was expected and contained in software. Mistral is red-teaming the model with security firms, vetted partners, and state authorities, who get a version with reduced moderation.

This sets up the sharpest policy contrast of the week. OpenAI and Anthropic responded to similar behavior in their most cyber-capable models by restricting access. Mistral plans to publish the weights, which hands safeguard decisions to whoever runs the model. Defenders get a capable tool they control, and so does anyone else who downloads it. Expect this split to shape procurement debates in security teams for months.

The weights are the part to watch. Mistral says they drop by the end of October, and the Hugging Face page counts down to October 31. Reporters were told October 27. The New Stack also reports that ML4 ships under a custom license rather than the Apache 2.0 license Mistral used for Large 3. Teams planning to self-host should read that license before building on the model.

What this changes for practitioners: a 1 million token, image-capable model with frontier-adjacent coding scores, priced under $1.50 per million input tokens, and downloadable within weeks. For regulated industries in Europe, ML4 gives procurement teams a credible open option where they previously had to accept a US or Chinese provider.

Reflection AI introduces Beam

Reflection AI introduced Beam on October 5, its first open-weight model. Beam is a sparse mixture-of-experts model with 501 billion total parameters and 23 billion active. It is text-only and built for coding, reasoning, and agentic work. Reflection pretrained it on 23.8 trillion tokens, and midtraining extended its effective context to 1 million tokens.

The training numbers stand out. Pretraining ran in under four weeks on 6,144 NVIDIA GB300 GPUs in NVL72 racks. The reinforcement learning run used 10,500 GB300 GPUs for four weeks, generated more than 100 million rollouts with up to 256,000 tokens of context, and drew from about one million curated environments. Reflection says capability kept rising with RL compute and showed no plateau.

Reflection's own benchmark table, all vendor-reported, shows Beam at 80.9 on SWE-Bench Verified, 65.5 on SWE-Bench Pro v1, 80.1 on Terminal-Bench v2.1, and 44.4 on DeepSWE v1.1. Reflection says Beam competes with GLM 5.2 and approaches Qwen 3.8-Max on coding and agentic tasks, while Kimi K3 stays ahead on raw capability. The pitch is cost per answer. Reflection claims Beam matches GLM 5.2 on reasoning tests with three to four times less inference compute.

Beam is in final red-teaming. Early access runs through a waitlist on the Reflection platform. Reflection says it will release the weights under Apache 2.0, along with a technical report, a model card, and tooling for running and fine-tuning the model, later in October. A user-facing reasoning effort setting lets developers trade answer quality for tokens.

This week's open-weight scorecard

The table below collects the open releases from this week in one place. All capability claims behind these models are vendor-reported, and two of the three large models have not published weights yet.

Two patterns stand out. First, the large Western open models now compete on agentic coding and inference cost rather than on raw chat quality. Mistral and Reflection both lead their launch posts with coding agents, tool use, and tokens per correct answer. Second, the small models are getting more specialized. Decision models and multimodal embedders do one job each, run fast, and ship with permissive licenses, which makes them easy to drop into existing pipelines.

There is also a geopolitical thread. Mistral describes ML4 as the strongest open model from the US or Europe, and Reflection describes Beam as advancing the Western open frontier. Both still benchmark against Chinese leaders such as Kimi K3, GLM 5.3, Qwen 3.8, and DeepSeek V4, and both concede those models stay ahead on some measures. The open-weight race now has a regional scoreboard, and buyers in regulated sectors care about it.

Decision models arrive from three directions

The quietest trend of the week matters a lot for production agents. Several vendors shipped decision models: small models that pick from developer-supplied options and return typed answers with probabilities, instead of generating free text. TypeSafe's Jev started the wave a few weeks ago. This week the big platforms answered.

Cloudflare released Clef and Clef-flash on October 1, hosted on Workers AI as @cf/cloudflare/clef and published on Hugging Face under Apache 2.0. Clef freezes Qwen3.8-27B as its backbone, and Clef-flash freezes Qwen3.5-9B. Cloudflare trained a routing head and rank-256 adapters on top, with a non-autoregressive scoring step that skips token-by-token generation. Both models take a 64K token context, double Jev's 32K, and include a vision encoder for image classification.

Cloudflare's latency numbers, vendor-reported across 43 benchmarks, put Clef at a 209 ms median and Clef-flash at 38.8 ms, against 524 ms for Jev. On its quality table, Clef scores 94.20 macro-F1 on BANKING77 and 97.43 on CLINC150 with out-of-scope detection. Jev still wins on When2Call and BRIGHT. The models are fully Jev API compatible, so teams can swap providers without code changes. Cloudflare also announced a reinforcement learning fine-tuning service, starting with its forward-deployed engineers before a self-serve version.

Cloudflare's own use case shows why this matters. Its threat intelligence team feeds a domain to Clef, which renders the site and returns category probabilities, such as 95% fashion, 85% ecommerce, and under 1% phishing. That run took 2.2 seconds. The fastest general LLM in the same workflow, gpt-oss-120b, took 4.7 seconds and returned only two categories. Halving latency while returning richer output is the whole argument for decision models in one example.

AWS launched Strands Decider 2B the same day. It uses Qwen3.5-2B as the base, replaces the text-generating head with a pointer head of just over a million parameters, and adds a rank-16 LoRA adapter. AWS released the model plus the data and scripts used to train it. The New Stack's example shows the use case clearly. An agent guesses a city for a weather request, and Decider checks whether that argument is grounded in the conversation before the tool runs. If not, the agent asks the user.

OpenAI launched its Decisions API as a limited preview earlier in the week, per The New Stack, built on its Luna model. Perplexity released Decider v1 27B on October 1 with Apache 2.0 weights. The common thread is clear. Teams want a cheap, fast, auditable check inside the agent loop, and a general LLM is the wrong tool for a yes, no, or pick-one answer.

Google EmbeddingGemma 2 goes multimodal

Google released EmbeddingGemma 2 on October 6. Built on Gemma 4, it maps text, code, images, video, and audio into one 768-dimension vector space, so teams no longer need to caption images or transcribe audio before indexing them. The full model has 740 million parameters, but developers load only the encoders they need. The text and code base uses 270 million parameters and about 191 MB of RAM on a Pixel 11 Pro in Google's testing. The full model used about 567 MB with quantization.

Google quadrupled the context window from 2,048 to 8,192 tokens. The company says that covers up to 5.5 minutes of audio, 29 images, or 58 video frames per input. The weights ship under Apache 2.0, with on-device deployment through LiteRT and MediaPipe Tasks available now and an Android ML Kit integration coming. The key design choice is the shared vector space. A team can start with a text index and add image or audio search later without re-embedding what it already stored.

Smaller releases and a cost check

Microsoft shipped three speech models on October 1: MAI-Transcribe-2-Streaming, its first streaming transcription model, plus MAI-Voice-2.1 and MAI-Voice-2.1-Flash. All three are generally available.

No new frontier model shipped this week from OpenAI, Anthropic, Google, xAI, Meta, or DeepSeek. The most useful closed-model news was an independent cost test. The New Stack ran GPT-6.1 Sol against GPT-6 Astra on three tasks, five runs each: CI triage, incident log analysis, and writing a dependency resolver from a spec. Both models got all 15 runs right. GPT-6.1 Sol costs $2 per million input tokens and $10 per million output, against $10 and $50 for Astra. Across the runs, Sol cost $2.66 against Astra's $14.77, or 18% of the price, and finished the two longer tests faster.

Retirements to act on

Deprecations deserve attention because they break pinned configs. GitHub deprecated four models across all Copilot experiences on October 2: Gemini 3.5 Flash and Gemini 3.6 Flash, with Gemini 3.8 Flash as the suggested replacement, Kimi K2.7 Code, replaced by Kimi K3, and Claude Opus 4.7, replaced by Claude Opus 5.5. OpenAI also deprecated gpt-5.3-codex, gpt-5.1, and gpt-5.4-nano ahead of a 2027 API shutdown, according to release trackers. Audit any workflow that pins one of these IDs.

Tooling: Codex Starts a Sprint and Copilot Learns to Click

OpenAI's 28-day Codex sprint

Thibault Sottiaux, who leads Codex engineering at OpenAI, set an unusual public challenge on October 4. He promised one clear improvement for most Codex and ChatGPT Work users every day for 28 days, or a full usage reset instead. The New Stack covered Day 1: GPT-6 Astra and GPT-6.1 Sol now run about 50% faster by default. Sottiaux later put it at roughly 50 tokens per second, up from about 30.

The speedup reaches beyond OpenAI's own apps. Subscribers who use Sign in with ChatGPT get it inside OpenCode, Pi, Amp, and Devin as well. The timing carries some irony. Two days before the sprint began, Sottiaux apologized for a slow GPT-6.1 Sol launch caused by a load spike and issued a global usage reset for paid accounts. Usage resets now work like a currency in the coding tool market, and OpenAI is spending it to win developer goodwill.

The CLI kept shipping too. Codex CLI 0.160.0 landed October 1. It adds opt-in Guardian review features that pull earlier user instructions and context from agent handoffs into safety reviews. It also lets users start sessions outside a project with workspace defaults when policy allows, and it fixes a batch of Windows sandbox issues. Version 0.160.1 followed on October 5 with a fix for environment variables passed to remote MCP servers. The September 29 releases had already made GPT-6.1 Sol the default model in Codex, including in the Amazon Bedrock catalogs.

Guardian deserves a closer look. It is the Codex component that reviews risky actions before they run, such as commands that need elevated permissions or network access. A reviewer that sees only the current command misses intent. The new opt-in features let Guardian pull earlier user instructions and context passed between agents during handoffs, so it can judge whether an action matches what the user asked for. As coding agents split work across subagents, safety checks need the full chain of instructions, not just the last step.

The sprint itself is worth tracking as a signal. Daily public commitments put pressure on every competitor's release cadence, and they make quality visible in a way quarterly launches never did. Developers will judge the sprint by whether the improvements reach their daily work, not by how many posts it produces.

GitHub Copilot gets computer use

GitHub put computer use into public preview in Copilot CLI and the Copilot app on macOS and Windows on October 1. Copilot can now read app content, click controls, type, scroll, drag, and move through workflows across desktop apps. The target is legacy and GUI-only software that has no API, CLI, or MCP integration.

The control model matters as much as the feature. Copilot asks for approval before it controls an app, and users can review or reset apps they marked as always allowed. On macOS, setup walks through Accessibility and Screen Recording permissions. Organization-managed settings can turn the feature off. Users enable it with /computer on in the CLI or a settings toggle in the app. The New Stack noted that GitHub's own guidance tells users to try other options first, which is the right posture for a tool that drives a real desktop.

The same day, GitHub added code-defined workflows to Copilot CLI, the Copilot app, and the Copilot SDK, available on all plans in public preview. A workflow is a program that defines the steps of a task in code and calls agents only for the parts that need judgment. Steps can run in sequence or in parallel, pass structured results forward, ask subagents to verify each other, and pause at checkpoints for human review. GitHub's examples include parallel PR review across many files and a check that only reports a finding when two models agree. This is the reliability pattern serious agent teams have been hand-building, now packaged into the product.

GitHub draws a line between this feature and /fleet, its existing mode where Copilot hands work to subagents and coordinates them on its own. A fleet decides its own plan. A code-defined workflow follows the plan you wrote, every time. Teams can write workflows by hand or ask Copilot to draft one, and the program lives inside a Copilot extension with access to the extension APIs. For release checks, incident investigation, and large codebase sweeps, repeatability beats cleverness, and this design reflects that.

The two October 1 features also connect. Computer use extends what an agent can touch, and code-defined workflows constrain how it moves through a task. Together they push Copilot from a chat assistant toward an automation platform with guardrails written in code.

Code review gets an API and a benchmark

GitHub opened Copilot code review to the REST and GraphQL APIs on October 2, with a per-request effort level. Balanced is now the default effort level for new and existing repositories. Teams that prefer the lighter mode can switch to Lite.

GitHub also unveiled ReviewBench, an open benchmark built with Microsoft to measure how well AI review agents find useful problems in pull requests. It uses 219 public PRs from 187 repositories across 19 languages. GitHub picked the corpus after analyzing 103.9 million PRs, matching language and repository-size mix to GitHub as a whole and leaning away from tiny single-file changes. Copilot code review tops the first leaderboard. Treat that result the way you treat any vendor-run benchmark, and look at independent review leaderboards before you choose a tool.

Enterprises rethink their coding agent bills

The Information reported on October 5 that Microsoft and Meta are steering employees away from Claude Code and toward in-house tools, per a summary from Cybersecurity News. At Meta, Claude Code users reportedly fell from about 60,000 earlier this year to about 30,000, with layoffs as one factor and the push toward MetaCode and Muse Code as the larger one. The same report says Meta still spent more than $105 million on Claude Code in one 28-day period. Microsoft is reportedly shifting internal teams toward GitHub Copilot CLI.

The lesson for engineering leaders is cost visibility. Agent spending grows with usage in ways seat licenses never did. Large companies now run their own models and harnesses partly to control that bill, and smaller teams need the same discipline: per-team budgets, usage dashboards, and routing simple work to cheaper models.

Observability for agents

Dynatrace closed its $915 million acquisition of Arize, announced in August. The pitch is simple. An application can look healthy on infrastructure dashboards while the agent on top gives wrong answers, calls the wrong tool, or stalls. Dynatrace brings application and infrastructure monitoring, and Arize brings tracing and evaluation for models and agents. Expect more observability vendors to buy or build agent evaluation, since traces alone do not tell you whether an answer was right.

Standards: Watermarks, MCP Context Costs, and Decision APIs

What agent tooling teaches standards bodies

The tooling news this week carries a standards lesson. Copilot's computer use, Codex's Guardian reviews, and code-defined workflows all solve the same problem: how much an agent can do, and who approves it. Each vendor answers that question in its own format today. Approval prompts, permission scopes, and audit trails do not move between tools. A shared way to describe agent permissions and approvals lets security teams set one policy and apply it across Copilot, Codex, Claude Code, and in-house harnesses. MCP covers tool access and A2A covers handoffs, but neither covers that policy layer yet.

OpenAI opens text watermarking to the API

OpenAI introduced opt-in text watermarking for its API on October 5, using a system it calls textGrain. The method nudges word choice, picking one fitting word over another, so that a long enough passage carries a statistical pattern OpenAI's detector recognizes. API customers anywhere can turn it on for supported models today. It stays off by default in the API.

The driver is the EU AI Act's transparency rules. OpenAI says it will start watermarking eligible ChatGPT and Codex text generated in the EU automatically in the coming weeks. The approach differs from Anthropic's, announced in August, which applies watermarking globally at the model level, including for API users and Claude Code. Developers now face two defaults from two major providers. Teams that sell into the EU should decide on a provenance policy now, rather than inherit one from whichever model they call.

Text watermarks have known limits. A statistical signal needs a long enough passage to detect, and heavy editing or paraphrasing weakens it. Watermarking is one input to a provenance program, not a full answer. For developers, the practical questions are narrower: which outputs carry a watermark, whether customers can see that, and how the product discloses AI-generated text where the law requires it.

MCP's context bill, and a fix

A practical MCP story landed this week. Pi, the coding agent acquired by Earendil earlier this year, added native MCP support in Pi 1.0 after a long holdout. Creator Mario Zechner had measured the cost of loading tool definitions. Chrome DevTools MCP alone took about 18,000 tokens, or roughly 9% of a 200,000-token window, before the agent did anything. Playwright MCP used about 13,700 tokens to describe 21 tools.

Pi's answer keeps tool definitions out of the prompt. Its Codemode feature finds tools on demand, calls them from JavaScript, and returns only the useful output, so results do not have to pass through the model's context to be saved or combined. This mirrors a broader shift. The MCP 2026-07-28 specification made the protocol stateless and HTTP-native, and the current roadmap focuses on unifying HTTP and stdio transports. The protocol works. The open problem is how clients spend tokens on it.

Cloud gateways keep adding MCP plumbing. Google Cloud's API Gateway added API key authentication for the MCP tools/list method on September 30, building on its September feature that turns an annotated OpenAPI spec into a remote MCP server. That pattern, an existing API spec projected as MCP tools behind a gateway with standard auth, is becoming the default enterprise path.

A de facto API for decision models

The decision model releases also carry a standards story. Cloudflare made Clef fully compatible with the Jev API, with typed question schemas for yes or no, choice, and score answers. That turns one startup's interface into a shared contract only weeks after launch. AWS took a different route with a downloadable model wired into its Strands agent framework, and OpenAI chose a hosted Decisions API. If more vendors adopt the Jev request shape, teams gain the same portability for decision calls that OpenAI-compatible endpoints brought to chat completions.

Agent safety moves below the application

NVIDIA launched its Open Agent Safety Platform, an open software platform and reference design for governing agents from testing to deployment. Its core is OpenShell, open source runtime software that sets execution boundaries for agents on CPUs and can extend to Arm and Intel platforms. The reference design adds Sentry, an out-of-band watchdog on BlueField-4 DPUs that monitors agent behavior and quarantines agents that leave their boundaries. Cadence is integrating the platform with its chip-design agents.

NVIDIA's framing is pointed. Recent incidents share a pattern where agents get around controls at the application layer to finish their tasks. Putting enforcement in a separate runtime, and in silicon outside the host, means a misbehaving agent cannot simply reason its way past the guard. Read this next to Mistral's report of a model probing its test environment. Containment now needs layers the model cannot reach.

Documentation for machine readers

Docsy, the Hugo documentation theme Google created in 2019, is moving to the Linux Foundation. Erin McKean of Google announced the move at Open Source Summit Europe in Prague on October 7. About 2,200 projects used Docsy by the end of 2024, including Kubernetes, OpenTelemetry, gRPC, and Jaeger. The project is adding features aimed at AI agents as readers of technical docs. As agents read documentation before they write code, docs structure becomes part of the interface contract for any API.

A2A stays quiet

The Agent2Agent protocol had no major release this week. A2A reached version 1.0 with signed Agent Cards earlier this year and passed 150 supporting organizations under Linux Foundation governance. Most production stacks now pair MCP for tools with A2A for agent-to-agent handoffs, and that split held steady this week.

Infrastructure: Memory Sets the Price of Every Token

Micron says supply stays tight through 2028

Memory remains the scarcest part of the AI stack, and this week's numbers made that plain. Micron reported record fiscal fourth-quarter revenue of $54 billion, up from $11 billion a year earlier. The company said memory supply stays tight through 2028 and that more than 75% of its 2027 output is already committed. It also told investors that 2027 HBM prices will run well above 2026 levels.

The Next Platform's analysis adds detail. Micron's core data center business grew about elevenfold year over year to just over $18 billion. The HBM4 ramp is on track, and Micron is working with NVIDIA on NVHBM, the first custom HBM4E part, for NVIDIA's next GPUs and NVLink Fusion platforms. Custom HBM marks a shift. Memory stacks now get designed around one customer's accelerator, the way logic chips always were.

Samsung added a capacity data point. A Samsung executive told Reuters that HBM will consume nearly 30% of DRAM makers' wafer capacity in 2027, up from about 20% today. Every wafer that goes to HBM is a wafer that does not go to standard DRAM, which is why conventional memory prices keep climbing too.

TrendForce expects conventional DRAM contract prices to rise 10% to 15% in the fourth quarter and NAND flash prices to rise 15% to 20%. It projects enterprise SSD bit demand to grow more than 80% in 2026. South Korea's semiconductor exports hit a record $60 billion in September, up more than 262% year over year, driven by memory volume and price. These figures reach every AI budget. Inference servers, vector stores, and the object storage behind data lakehouses all buy from the same constrained supply.

For AI teams, the planning implication is direct. Hardware quotes for 2027 will carry higher memory costs, and cloud providers pass those costs into instance prices. Teams that size inference fleets now should model memory as the line item most likely to rise. Techniques that shrink memory per request, such as quantized models, smaller specialized models, and caching shared context, pay off twice in this market.

Optics close in on the network bottleneck

As memory gets more expensive, wasted accelerator time costs more too. Counterpoint Research estimates that GPUs spend roughly 12.5% of their time waiting on the network in some workloads. The firm expects near-packaged optics to start sampling in late 2026 or early 2027, followed by co-packaged optics as scale-up interconnects demand more bandwidth per watt.

Standards and funding moved this week to support that shift. JEDEC published the industry's first silicon photonics reliability standard, which sets common qualification rules as optical interconnects spread through AI systems. CScale came out of stealth with $145 million to build an optical interconnect for AI scale-up that contains optical failures without stopping compute. Volantis raised $88 million for an interconnect that uses integrated micro-VCSELs to link many memory chips into one pool for inference. Pooled memory over optics attacks the same problem as HBM from a different angle: give each accelerator access to more memory without stacking it all on the package.

Who trained on what this week

This week's model launches doubled as infrastructure disclosures. Mistral trained Large 4 on 3,800 Grace Blackwell GPUs in its own European data centers and serves the preview on the same fleet. Its reinforcement learning stack produces roughly 33 billion tokens per day on about 3,000 GPUs, with about 16 billion of those trainable after filtering. Mistral says its €3 billion Series D funds a large compute expansion in Europe.

Reflection pretrained Beam on 6,144 GB300 GPUs in NVL72 racks, then ran RL on 10,500 GB300 GPUs for four weeks. Its RL platform sustained about 110,000 concurrent rollouts and up to 170,000 concurrent sandboxes across more than 20 clusters, two clouds, and four regions. New weights reached the inference fleet in a median of about 12 seconds. Reflection reports pretraining goodput of 92.3% near the end of the run, with nine rewinds for gradient spikes or suspected silent data corruption.

These numbers show where frontier spending now goes. Pretraining is only one stage. RL at scale needs huge numbers of CPU sandboxes, fast weight distribution, and fault handling that keeps a month-long job alive through dozens of incidents. That is why NVIDIA's CPU-heavy rack designs and agent sandboxing products keep showing up next to its GPU launches.

NVIDIA also published a post on how its Blackwell GPUs accelerate GPT-6 Astra Ultrafast, which is now live in the OpenAI API and for eligible ChatGPT Work and Codex users. Fast tiers like this are becoming a standard product line, priced above default speed for latency-sensitive agent loops.

Capital, fabs, and power

Several deals this week show how AI hardware financing now works. Anthropic's IPO filing revealed that it is the customer behind Broadcom's previously disclosed compute financing arrangement of up to $42 billion. Chip vendors financing their own customers' compute is now a standard tool for moving custom silicon.

TSMC is reportedly weighing a multibillion-dollar Texas campus for more AI chips, beyond its Arizona buildout. AMD plans to acquire World Labs, which builds spatial-intelligence models that simulate interactive 3D environments for robotics, in an $8.2 billion all-stock deal expected to close by the end of 2026. Samsung Electro-Mechanics will invest about $4.9 billion to expand FCBGA substrate capacity for AI servers in South Korea and Vietnam.

Power keeps showing up in chip news. South Korea and the United States are advancing a $22 billion, 6.47 GW gas-fired power project in Texas to serve chip plants and AI data centers. Infineon will supply silicon carbide devices for Eaton's solid-state transformer platform built for 800 VDC data center power. The 800-volt shift reduces conversion losses in dense AI racks, and suppliers are lining up parts for it now.

What this means for buyers

Put the week's infrastructure news together and a pattern appears. The cost of AI is shifting from compute alone to compute plus memory plus power. Vendors that disclose cluster sizes, RL token rates, and serving hardware give buyers a better sense of what a model costs to run and how fast it will improve. Mistral and Reflection both published more training detail than most closed labs share. That transparency is part of the open-weight value proposition, and procurement teams should ask every provider for the same numbers.

The other lesson is about timing. Memory contracts for 2027 are already mostly signed. Teams that need dedicated capacity next year should lock in plans now, and teams that rent capacity should expect prices to follow memory upward.

The Data Layer Angle

Three of this week's stories land squarely on the data platform. EmbeddingGemma 2 puts text, images, video, and audio into one 768-dimension space. That makes vector columns a normal part of analytical tables, not a side system. The open table and file formats are racing to catch up. On the Apache Parquet dev list this week, contributors debated whether a new VECTOR type should guarantee finite elements, and Ryan Blue proposed shipping two types to settle it. Apache Iceberg started a dedicated sync on October 8 for its own vector type. Teams that plan to store embeddings in their lakehouse should follow both threads, because the outcome decides what readers and indexes can assume about those columns.

Decision models fit the governed data layer too. A decision model needs clean, well-defined options and fields to choose from. That is exactly what a semantic layer provides. The Apache Ossie (incubating) community spent this week debating whether to define an MCP contract so agents can discover a semantic model and run governed queries against it. Put a decision model in front of that contract, and an agent can route a question to the right metric, check its arguments, and run it with a confidence score attached.

The memory story ties back as well. When DRAM and NAND prices rise 10% to 20% in a quarter, the cost of keeping hot copies of data rises with them. Open formats on object storage, with engines that cache smartly, keep that cost in check. Moving compute to the data beats copying data into every new AI tool.

Practitioner Takeaways

Plan for Mistral Large 4 weights at the end of October, and read the custom license before you commit to self-hosting.
Put a decision model in your agent loop for routing, tool-argument checks, and output gating. Clef, Strands Decider, and Perplexity Decider all ship open weights.
Audit pinned model IDs in Copilot and OpenAI configs. Several Gemini, Kimi, Claude, and GPT-5.x IDs retired or entered deprecation this week.
Decide your text watermarking policy for EU users now. OpenAI and Anthropic ship different defaults.
Measure the token cost of every MCP server you load. On-demand tool loading can save tens of thousands of tokens per session.
Budget for higher memory and storage prices through 2027, and favor architectures that avoid duplicate copies of large datasets.

What to Watch Next Week

OpenAI's Codex sprint promises a new improvement every day, so expect a steady stream of small changes and possibly a new model before the 28 days end. Reflection's Beam weights and technical report are due later in October, and Mistral's ML4 weights should follow by month end. On the infrastructure side, the OCP Global Summit runs October 12 to 15 in San Jose and SEMICON West runs October 13 to 15 in San Francisco, so look for rack, power, and optics announcements. On the standards side, watch whether more vendors adopt the Jev request format for decision models, and whether MCP clients copy Pi's on-demand tool loading.

Keep Learning

The pace of change in AI rewards people who understand the fundamentals underneath the headlines. My books cover AI-assisted development, agentic analytics, open lakehouse architecture, and the economics of AI and work. Browse the full catalog at books.alexmerced.com and pick the one that matches what you are building next.

Stop Calling Everything Made With AI Slop

Alex Merced — Mon, 05 Oct 2026 14:36:35 +0000

Scroll any comment section under a blog post, a video, or a song in 2026 and you will find the same two-word review. "AI slop." Sometimes the label fits. The post invents a statistic, the video melts into nonsense halfway through, or the article repeats the same empty point six times in slightly different words.

Other times the label lands on work that is careful, accurate, and genuinely useful. Someone spent hours on the research, wrote pages of their own notes, argued with the draft, fixed what the model got wrong, and shipped something worth reading. A commenter spots a familiar phrase or a too-clean thumbnail and dismisses the whole thing in two words.

I want to make a case that sounds simple but gets lost in the noise. Bad AI-assisted content is bad because it is bad, not because AI touched it. Good AI-assisted content exists, it takes real skill to make, and the only way anyone builds that skill is by using the tools often enough to get past the clumsy early stage.

I have not been shy about how I work. I use AI heavily, mostly in areas where I already have deep knowledge, and I use it to move faster. I also make my mistakes in public, because public mistakes bring feedback, and feedback is how I get better. This article explains what slop actually is, why every creative tool goes through this same backlash, what I learned by doing it over and over, and how I think we should judge creative work from here on out.

What Slop Actually Is

The word has a real definition now. Merriam-Webster named "slop" its 2025 Word of the Year and defined it as digital content of low quality that is produced usually in quantity by means of artificial intelligence. Macquarie Dictionary picked "AI slop" as its word of the year too. The term earned its spot. The internet filled up with junk fast.

Read the definition closely, though. "Low quality" sits right in the middle of it. Quality is the defining trait. AI is the usual production method, but the definition does not say every AI-produced thing is slop. It says slop is low-quality content, usually mass-produced with AI.

That distinction matters, because a lot of people use the word to mean "anything I suspect involved a model." Those are two very different claims. One is a judgment about the work. The other is a guess about the workflow.

So what puts something in the slop bin? I see three clear markers.

It tries to misinform. The content spreads false claims, fabricated quotes, fake sources, or invented events. Sometimes this is deliberate. Sometimes the creator just never checked. The result for the reader is the same.
It adds no value. The piece says nothing a reader did not already know. It restates the obvious, pads the length, and leaves you with no new idea, no new skill, and no new perspective.
It clearly was not reviewed. The draft still contains the model's mistakes. You see broken references, contradictions between sections, hallucinated version numbers, and leftover phrases like "as an AI model" or "here is your blog post."

Every one of these markers describes a failure of the creator, not the tool. A human writer with no AI at all can misinform, add nothing, and skip editing. We just had other names for that content before 2023. We called it content farm filler, clickbait, or spam.

The volume is the new part. A content farm used to need a room full of underpaid writers. Now one person with a script can publish thousands of pages a day. Streaming service Deezer has reported that a large share of the music uploaded to its platform every day is fully AI-generated, at one point estimating about 28 percent of new uploads. That flood is real, and people have every right to be tired of it.

But fatigue makes people sloppy judges. When the default reaction to any hint of AI is "slop," the label stops meaning "low quality." It starts meaning "made with a tool I disapprove of." That shift hurts the people who use these tools well, and it teaches newcomers that the right move is to hide their process instead of improving it.

Every Creative Tool Got This Reception

The backlash against AI-assisted work feels new, but the pattern is old. Almost every tool that lowered the barrier to creative work got called cheating, fake, or the death of real art.

Music has the cleanest example. In 1982, the UK Musicians' Union passed a motion to ban synthesizers, drum machines, and any electronic devices capable of recreating the sounds of conventional instruments. The union feared these machines threatened the jobs of session players. The motion came after Barry Manilow toured the UK using synths to simulate the sound of a big band orchestra.

The ban went nowhere. Drum machines went on to shape hip-hop, house, and techno. Those genres did not replace older music. They added new music that did not exist before those instruments arrived. The union was right that the work was about to change. It was wrong that the change meant the end of real musicianship.

Photo editing went through the same cycle. Adobe released Photoshop 1.0 in 1990. For years after, "photoshopped" worked as an insult. It meant fake, manipulated, dishonest. Today nearly every professional photo passes through editing software, and nobody accuses a wedding photographer of fraud for adjusting exposure. People still criticize manipulated images, and they should when the edit deceives. The criticism shifted from "you used the tool" to "you used the tool to lie."

Digital audio workstations (DAWs) followed the same arc. A DAW is software for recording, editing, and arranging music on a computer. Tools like FL Studio, which started life as FruityLoops in the late 1990s, let a teenager with a laptop build tracks that once needed a studio, an engineer, and a band. Early bedroom producers got mocked for not playing "real" instruments. Some of those bedroom producers now headline festivals.

I have picked up a lot of these tools over the years. Photoshop, FL Studio, Videoleap, and plenty more. I learned each one for the same reason: it let me make the ideas in my head real. AI tools are the latest entry on that list.

Not every critic in these cycles was wrong. Some synth-heavy records from the 1980s really are thin. Plenty of early Photoshop work really was garish. Tools do get misused, especially early, when nobody has figured out the craft yet.

Where the critics went wrong was the target. They judged the tool when they should have judged the output. Over time, the culture corrected. The tool stopped being the story, and the work became the story again. I expect the same correction for AI-assisted work. The only open question is how many capable creators we discourage before it happens.

Using AI Well Is a Skill, and Skills Come From Reps

Here is the part that gets lost in most arguments about AI content. Using these tools well is a skill, and it behaves like every other craft skill. You start out bad, you get feedback, you adjust, and you slowly get good.

Nobody opens a DAW for the first time and mixes a radio-ready track. Nobody opens Photoshop and produces a clean composite on day one. The first hundred attempts teach you where the controls are. The next hundred teach you what good looks like. After that, you start to build taste and speed at the same time.

AI tools work the same way. Your first AI-assisted blog post will probably read flat. Your first AI-generated image will have strange hands or warped text. Your first AI-assisted song will sound generic. That tells you that you are a beginner, which is exactly what everyone is at the start of any skill. It says very little about the ceiling of the tool.

The people quickest to shame AI users often point to those beginner outputs as proof. "Look how bad this is." They are judging a first draft from a first-week user and treating it as the ceiling of the medium. Nobody judges guitar by listening to someone on their third lesson.

My own practice looks like this. I mostly use AI in areas where I already know the material well. I have co-written books on Apache Iceberg and Apache Polaris and written another on architecting an Iceberg lakehouse. When a model drafts something in that territory, my expertise does the checking. The model does the typing, the first-pass research, and the structural grunt work.

That pairing is where AI pays off the most. Expertise plus AI acceleration beats either one alone. An expert without AI moves slower. AI without an expert produces confident nonsense. The skill lives in the combination.

The clearest example from my own work is my personal websites. I manage them myself with AI assistance. That work used to need a team of specialists: a web developer, a copyeditor, a graphic designer. Now I handle the full workflow on my own, end to end. At a larger scale, my plan is not a row of narrow specialists. I want a small team of generalists, each one owning complete workflows end to end with AI support, all pointed at a shared goal.

None of that came free. I got there by doing the work over and over, and by doing it in public. I make mistakes in public without shame, because that is how I get the feedback and the lessons that let me improve quickly.

This is why I push back when people say "just don't use AI until it's good." The tools keep getting better, but the person using them only gets better through practice. If you wait on the sidelines until the technology is perfect, you will show up with zero experience while everyone who practiced has years of instinct on you.

What the Early Reps Taught Me About the "AI Meh" Feel

Every heavy AI user eventually learns to recognize a certain texture. I call it the "AI meh" feel. The content is not wrong, exactly. It is grammatical, organized, and on topic. It is also forgettable. Readers sense it within a paragraph, even when they cannot name what bothers them.

Learning what gives content that feel, and how to get rid of it, was one of the first lessons that came from doing this regularly.

The meh feel tends to come from a handful of habits that language models fall into when nobody steers them.

Generic openings. Models love to open with a broad statement about how fast the world changes or how important a topic has become. Readers skip these lines automatically. A strong opening starts with a concrete problem the reader recognizes.

Filler transitions. Stiff, formal connector words pile up at the start of sentences in AI drafts. They make prose sound formal while carrying no meaning. The same goes for phrases that announce importance instead of demonstrating it.

Rhythm lists. Models reach for groups of three adjectives because the rhythm sounds polished. After a few paragraphs, the pattern becomes obvious, and the reader starts to tune out.

Hedging. Unsteered drafts qualify everything. Every claim gets a softener in front of it. The result reads as if nobody stands behind any of it.

Missing opinion. This one matters most. A model with no direction defaults to a balanced summary of every side. Balanced summaries have their place, but they have no voice. Readers come back to creators because those creators see things a certain way.

I keep a list of style rules and banned words for anything written in my voice. No em dashes. No stacked formal transitions. No hedge words in the body of the piece. Active voice. Short paragraphs. Concrete numbers over vague claims of size. Those rules go to the model as part of the instructions whenever it drafts in my voice.

Rules only fix the surface, though. The deeper fix for the meh feel is input. A model drafts flat content when it has nothing specific to work with. Give it a generic prompt and you get the average of everything it has read. Give it your specific angle, your notes, your opinions, and the examples you actually care about, and the draft starts to sound like a person with a point of view.

This article is a good example. It started with a long set of notes in my own words. Every argument in it, every example, every opinion came from those notes. The model helped organize the material, expand it, and tighten it against my style rules. The thinking is mine. The acceleration came from the tool.

That is the real difference between slop and skilled AI-assisted work. Slop starts with a topic. Skilled work starts with a point of view.

Verifying Efficiently: Know Where the Model Breaks

The second big lesson from the early reps was about review. Beginners tend to make one of two mistakes. Some skip review entirely and publish whatever the model gives them. That is how slop gets made. Others review every sentence with equal suspicion, which takes so long that the AI saves them no time at all.

Efficient review means knowing where the model is strong and where it is weak, then spending your attention where the weaknesses live. A good editor already works this way with human writers. You learn which writer mangles dates and which one overstates conclusions, and you read their drafts with that in mind.

Language models have fairly predictable weak spots. Here is how I think about the split.

That last row deserves extra attention. An unsteered model writing in your voice will happily invent a conference you never attended or a customer you never met. Those fake stories read well, which makes them dangerous. I have a standing rule that no anecdote appears in my content unless it actually happened to me. The model does not get to make up my life.

Review also gets faster when you automate the mechanical parts. My style rules are specific enough to check with a script. Here is a version of the verification pass for my long-form drafts.

F=article.md

# Word count: long-form pieces need to clear 6,000 words
wc -w "$F"

# Em dashes and en dashes: must return nothing
grep -nE '—|–' "$F"

# Semicolons: must return nothing outside code blocks
grep -n ';' "$F"

# Banned vocabulary sweep: the words that make prose read like a template
grep -niE '\b(delve|leverag|seamless|paradigm|synergy|holistic|robust|unlock|realm|tapestry|testament|embark|myriad|plethora|pivotal|transformative|bespoke|curated|meticulous|elevate|empower|streamline|foster)\w*' "$F"

# Hedging modals: claims get stated plainly or not at all
grep -niE '\b(might|could|would|may) ' "$F"

# Call to action present
grep -n 'books.alexmerced.com' "$F"

Here is what each part does and why it exists.

The first line sets a variable named F to the file path, so every later command checks the same file without retyping the name.

The wc -w command counts words. Long-form pieces have a target length, and models often land short when asked for a long draft in one pass. A word count catches that right away, before anyone spends time reading a draft that needs more material anyway.

The dash check uses grep -nE with a pattern that matches either an em dash or an en dash. The -n flag prints line numbers, so each hit is easy to find. Models lean on em dashes heavily, and a page full of them is one of the fastest tells of unedited AI prose.

The semicolon check works the same way. My style rules keep semicolons out of prose, so the only acceptable hits are inside code blocks.

The vocabulary sweep uses -i for case-insensitive matching and \b for word boundaries, so one stem catches every form of a word, capitalized or not. Every word on that list is technically fine English. The problem is frequency. Models reach for these words far more often than people do, and a reader who sees three of them in a paragraph stops trusting the writing.

The hedging check looks for soft modal verbs followed by a space. The goal is plain claims. If a claim is true, state it. If it holds only under certain conditions, name the conditions.

The last line confirms the call to action made it into the final draft. That sounds trivial, but when a model rewrites a closing section, the link sometimes disappears.

None of this replaces reading the piece. The script handles the mechanical checks in about a second, which frees my attention for the parts only a human expert can judge. Is the argument right? Is the technical claim accurate? Does this say something worth saying?

That division of labor is the whole point. The machine checks the machine-checkable. I check the meaning.

The Flow Changes With the Model, the Tool, and the Medium

Once you have a process that works, it is tempting to treat it as finished. It never is. My workflow shifts every time one of three things changes.

The model. Different models have different habits. One model leans on certain phrases, while another pads its output with summaries at the end of every section. One is more likely to invent citations, another is more cautious but more generic. When I switch models, my review checklist shifts with it. The weak spots in the table above hold up broadly, but the details move.

The tool around the model. The same model behaves differently depending on the app or agent setup wrapped around it. A chat window, a coding agent with file access, and a writing tool with a custom style guide all produce different results. Some setups let the model search the web, which helps with current facts but introduces new risks around sources. Others let the model run code, which helps with verification. Learning what each setup is good at is its own skill.

The medium. Writing, video, and music each have their own failure modes. In writing, the risk is factual errors and the meh feel. In video, it is visual consistency from shot to shot, characters whose faces drift, and motion that breaks physics in distracting ways. In music, it is generic arrangements, muddy mixes, and lyrics that rhyme without saying anything.

Each medium needs its own review habits. For an article, review targets facts and voice. For a video, it targets continuity from shot to shot, and anything that breaks the illusion gets cut. For a song, it targets the moments where the arrangement goes on autopilot.

The underlying pattern stays the same across all three. You learn where the tool is reliable. You learn where it fails. You put your effort where the failures are. And the only way to build that map is to make a lot of things and pay attention to what goes wrong.

This is also why one attempt, years ago, is a weak basis for deciding AI is useless. The tools they tried are gone. The models changed, the setups changed, and the outputs changed. More to the point, they never built the map. One try tells you almost nothing about what a practiced user can do.

"If AI Wrote It, Why Should I Read It?"

This is a common objection, and it deserves a real answer. The argument goes like this. If a machine wrote the piece, the writer put in no effort, so the reader owes it no attention. Why spend my time on something nobody bothered to write?

The objection rests on an assumption that the person contributed nothing. For slop, that assumption is usually correct. Someone typed a topic into a box, copied the output, and hit publish. There is no person in that content, so there is nothing of a person to read.

Skilled AI-assisted work is different. The creator puts themselves into the piece at every stage that matters.

The angle. Before any drafting happens, someone decides what the piece argues. A model left alone writes the average take. A creator with a point of view picks the specific claim, the counterintuitive framing, or the uncomfortable truth that makes the piece worth reading.

The research. Someone decides which sources to trust, which facts to verify, and which examples best support the point. Those choices shape the piece as much as any sentence does.

The notes. Strong AI-assisted pieces start with the creator's own notes. Those notes hold their opinions, their reflections, the wisdom they have built up, and the arguments they want to make. Those notes are the raw material. The model shapes them. It does not invent them.

The judgment. Someone reads the draft, rejects what is wrong, pushes back on what is weak, and decides when the piece is done. That editorial judgment is a creative act. Film directors do not operate every camera, and nobody says their films lack a creator.

When all four are present, the reader gets the creator's thinking, delivered faster than typing it alone. That is a good deal for the reader.

There is a second half to this objection worth answering. People say, "If AI wrote it, why should I read it? I can just ask AI." Fair enough. Ask AI to help you read it, then. Ask a model to summarize a long piece, pull out the key claims, or explain a section you found confusing.

Reading can be AI-assisted the same way writing can. There is no shame in that. A dense technical article is a lot more approachable when you can ask follow-up questions about it. Using AI to read does not mean the piece should not have been written. It means the reader found a faster way into it.

The piece still had to exist. Somebody still had to choose the argument, gather the evidence, and stand behind the conclusions. A model summarizing an article you never wrote has nothing to summarize.

And there is real pleasure in reading something slowly, on your own, with no assistance at all. Some books deserve your full, unassisted attention. Neither approach is the correct one. Both are tools a reader can pick up depending on what they want from the experience.

What Good Looks Like

Arguments are easier to evaluate with examples, so here are two. One comes from a creator whose work shows what this technology can do. The other comes from my own music.

Gossip Goblin

The filmmaker Zack London, known online as Gossip Goblin, makes dark, strange science fiction short films with AI video tools. His channel has passed 1.4 million subscribers and more than a billion total views. His first feature-length film, an anthology called Gods Don't Give Gifts, was scheduled to open in several hundred U.S. theaters on October 30, 2026, produced with the studio Fable.

What makes his work relevant to this argument is how much human effort goes into it. London has said his feature still needed a human-written script, a dozen voice actors, and a post-production crew. Reporting on the feature describes a team of 10 artists working on it over two months. His process starts with a script, then builds every shot from a still image generated in Midjourney, refined in other tools, and only then animated into video. That is a slower, more hands-on pipeline than the one-click generation most people picture.

London has also said plainly that AI is a tool for him and not the point. It gave him access to filmmaking without replacing crafts he already valued, like illustration. His shorts do not hide the strangeness of the medium. They lean into it and build a recognizable visual style from it.

You do not have to like his films. Taste is personal. But nobody watching his work in good faith calls it low effort. It has a vision, a consistent world, and a recognizable voice. Those are the things that separate art from filler in any medium. AI did not supply them. London did.

My old acoustic recordings

The second example is smaller and more personal. I have old acoustic recordings from my younger days. They hold my effort and my vision from that time.

Suno is an AI music platform, and one of its features lets you upload your own audio and generate a new version in a different style while keeping the original melody. My old acoustic recordings have become the seeds for full productions.

The songwriting, the melodies, and the emotional core come from my younger self. The production comes from a tool that did not exist back then. Neither approach makes these songs alone, and the result is a fun bridge between two versions of me.

Is that slop? Run it through the three markers. It does not misinform anyone. It adds value by finishing work that started years ago. And the review test comes down to keeping the versions that serve the song and tossing the ones that miss. By the definition that matters, it is not slop. It is a creative project that uses a new tool.

There Is Room for Both Markets

I want to be clear about one thing. I am not arguing that all content should be AI-assisted. I do not think that, and I do not want that.

There will be a strong market for creators who make everything without AI, and there should be. Some audiences care deeply about knowing that a human did every step. That preference is valid, and the market already serves it. Bandcamp, the music platform popular with independent artists, announced in January 2026 that it no longer allows music generated wholly or in substantial part by AI. The company framed the policy as a way to keep its community human. That is a reasonable choice for a platform, and it gives listeners who want fully human music a clear place to find it.

Handmade work has always held value next to manufactured work. Hand-thrown pottery sells alongside factory dishware. Film photography survives next to digital cameras. Live acoustic sets still draw crowds in a world full of programmed beats. Those markets did not vanish when the faster tool arrived. They found their audience, and the handmade quality became a feature people choose on purpose.

The same thing will happen with writing, video, and music. Some creators will advertise that they use no AI, and some audiences will seek them out. Other creators will use AI heavily and openly, and their audiences will care about the output, not the process.

What I push back on is the idea that one camp gets to define the other as illegitimate. The human-only creator is not a dinosaur. The AI-assisted creator is not a fraud. They are making different choices about tools, and both can produce excellent work or terrible work.

Every major creative technology eventually settles into some balance between efficiency and quality. Photoshop did not make every photo fake. DAWs did not make every song soulless. The balance point took years of experimentation, curiosity, and a lot of bad early work to find. AI-assisted creation is in that experimentation phase right now. The people who will find the balance point are the ones willing to practice in the open.

How to Practice Without Publishing Slop

If you are early in using AI for creative work, the advice to "just do it a lot" needs some guardrails. Practice should not mean flooding the internet with weak drafts. Here is how I think about building the skill responsibly.

Start where you are already an expert. Use AI first in a subject you know cold. Your knowledge acts as a built-in fact checker. You will catch the model's errors quickly, and you will learn its failure patterns faster because you can see them. Using AI in a field you do not understand is how people publish confident mistakes.

Verify against primary sources. When a draft cites a release date, a feature, a quote, or a number, check it against the original source. That means the official documentation, the project's release notes, the dictionary entry, or the article that first reported the claim. Secondhand summaries repeat each other's errors, and models blend them together. A habit of going to the original catches most factual mistakes before they reach a reader, and it gets faster as you learn which sources hold up.

Write your notes before you prompt. Spend 15 minutes writing down what you actually think about the topic. Your opinions, your examples, the arguments you want to make, the mistakes you see other people making. Feed those notes to the model. The difference in output quality between "write about X" and "here are my notes on X, shape them into an article" is enormous.

Keep a running list of what goes wrong. Every time a model makes a mistake you catch, write it down. Wrong dates, invented quotes, repeated phrases, weak openings. After a few dozen pieces, that list becomes your review checklist. It tells you exactly where to look, which is how review gets faster over time.

Turn your style into rules. If you notice yourself fixing the same pattern over and over, write a rule for it and give the rule to the model up front. Then check the rule with a script where you can. My own banned-word list and verification script are examples of this.

Publish less than you generate. The ability to produce a lot does not mean you should publish a lot. Generate ten drafts, publish the one that says something. Volume is the defining trait of slop. Selectivity is the defining trait of craft.

Be open about your process. Hiding AI use creates a trust problem when people find out. Being open about it lets you get honest feedback on the work itself. I talk about how I use AI because the feedback makes me better, and because I think the conversation about these tools needs more practitioners sharing real workflows.

Take criticism of the work seriously and criticism of the tool lightly. When someone says a claim is wrong or a section is weak, that is gold. Fix it. When someone says "AI slop" with no further detail, there is usually nothing to act on. Check whether your piece hits any of the three slop markers. If it does not, move on.

Judge the Work, Not the Workflow

So how should we judge creative work now that AI is part of so many workflows? The same way we judged it before. Look at the output and ask the questions that always mattered.

Is it accurate? Does it state true things? Are its sources real? Do its numbers hold up? A piece full of errors fails this test whether a human or a model introduced the errors.

Does it add value? Did you learn something, feel something, or see something in a new way? Content that leaves you exactly where you started has failed, no matter how it was made.

Does it have a point of view? Can you tell what the creator thinks? Is there a specific argument, a specific style, or a specific vision behind it? Work with a clear voice behind it is almost never slop.

Was it cared for? Do the details hold together? Are the rough edges sanded down? Does it feel finished? Care shows up in the work, and its absence shows up too.

Notice that none of these questions asks what tools the creator used. That is deliberate. We do not grade a novel by whether the author used a typewriter or a laptop. We do not grade a song by whether the drums came from a kit or a drum machine. We do not grade a photo by whether it passed through editing software. We grade the result.

These questions also work in both directions. They catch slop made with AI, and they catch lazy work made without it. A handwritten article full of errors fails the accuracy test. A fully human song with nothing to say fails the value test. The standard stays the same no matter which tools sit in the workflow.

AI-assisted work deserves the same standard. Hold it to a high bar. Call out errors. Call out empty content. Call out work that clearly never got a human review. Those criticisms help everyone.

But when a piece is accurate, valuable, distinctive, and carefully made, the presence of AI in the workflow counts as a production detail, the same way the choice of camera or DAW is a production detail. Something that adds real value, AI or not, still required skill, vision, and determination to make.

Conclusion

Slop is real, and I share the frustration with it. Content that misinforms, adds nothing, and skips review makes the internet worse, and people are right to call it out. The dictionary even gave us a word for it, and the definition puts low quality at the center.

But the word has drifted. Too many people now use "slop" to mean "made with AI," and that shift punishes the wrong people. It discourages beginners from practicing a skill that only improves with practice. It pressures experienced creators to hide their process instead of sharing what they learned. And it confuses a judgment about a workflow with a judgment about the work.

My own path has been simple. I learn whatever tool lets me make the things I picture in my head. Photoshop, Videoleap, FL Studio, and now a range of AI tools. Each one made me more versatile. I learn in public and make my mistakes in public on purpose, because public mistakes bring the fastest lessons, and I am proud of what that approach has let me build.

If you want to criticize AI-assisted work, criticize it on the merits. Point to the wrong fact, the empty argument, the missing review. If you want to make AI-assisted work, put yourself into it. Bring your angle, your notes, your judgment, and your willingness to throw away drafts that do not meet the bar.

Judge creators by the quality and value of what they make, the way we judge creators in every other medium. That standard worked for photography, for synthesizers, and for the bedroom producer with a laptop and a DAW. It will work for this too.

Keep Going

If this piece was useful, I have written a lot more on how AI is changing the way people work. My book on AI and labor economics covers the economic side of this shift, and you can find it at a.co/d/06SeOKw8. You can find every book I have written, across lakehouse architecture, Apache Iceberg, Apache Polaris, and AI, at books.alexmerced.com.

Apache Data Lakehouse Weekly: September 23 to 30, 2026

Alex Merced — Wed, 30 Sep 2026 13:57:19 +0000

Release week. Apache Iceberg 1.12.0 passed its vote after a license audit sank the first candidate, Apache Polaris shipped 1.8.0 with a security fix inside it, and Iceberg C++ 0.4.0 went out the door. Under the release traffic sat a quieter theme that ran through every list. Specs are getting stricter about what they promise. Iceberg voted to forbid new equality deletes in v4. Parquet argued over whether a vector type should reject NaN. Ossie asked what a column name in a metric expression points to. Polaris debated whether a config field should be an enum or free text. Each thread is a fight about contracts, and each one ends with a narrower, more checkable promise to the engines that read these formats.

This issue covers roughly 380 messages across the Iceberg, Polaris, Arrow, Parquet, DataFusion and Ossie dev lists between September 23 and September 30.

Apache Iceberg

Iceberg 1.12.0 passes on the second try

Neelesh Salian ran the 1.12.0 release, and it took two candidates to land. The RC1 vote drew early +1s from Alex Dutra, Gianluca Graziadei and Szehon Ho. Then Kurtis Wright posted a -1 with a disclaimer at the top: he had prompted Claude to audit every license in the candidate. The audit found that the GCP bundle jar shipped Caffeine with no entry in its LICENSE file. Kevin Liu followed up with his own table of gaps and found a second one around Jackson's FastDoubleParser. Jean-Baptiste Onofré cast a -1 on the same grounds. Neelesh sorted each report, landed the fixes, and pulled in Gianluca's patch for a variant-column NoSuchMethodError that hit clusters with older commons-lang3 jars.

The RC2 vote opened on September 25. Neelesh stretched the window to Tuesday afternoon to cover the weekend. It had one scare. Huaxin Gao pointed out that the staging repository had no Spark 4.2 artifacts, even though the source tree, CI matrix and Gradle defaults all pointed at 4.2. Amogh Jahagirdar agreed a new RC looked necessary. Russell Spitzer then asked why an earlier PR had excluded 4.2 from convenience binaries on purpose. Amogh found the older thread. The community had held back 4.2 binaries while Spark's own APIs kept moving. Huaxin withdrew the alarm and voted +1 with a note that every RC1 license gap was closed in the jars themselves.

The result came in on September 29: four binding +1s from Russell Spitzer, Kevin Liu, Szehon Ho and Amogh Jahagirdar, and ten non-binding +1s. Two details stood out in the votes. Kevin ran an agent over the repository looking for correctness bugs and flagged a handful of issues that also exist in 1.11. He called them non-blockers and listed them for triage. Kurtis came back with a second AI-assisted pass, this time labeled as the output of an agent skill built for checking open source release candidates. Release verification is turning into a place where contributors point agents at tedious, rule-driven work and then sign their names to the result.

The CI side of the release had its own story. Steven Wu asked where the MinIO replacement stood after Kafka Connect integration tests failed on the MinIO container. Neelesh opened a tracking issue and a stopgap PR. Within a few hours Kevin reported that the switch was done across every Iceberg repository, crediting Sreesh for the work. Amogh later noted that he ran the RC2 integration tests against RustFS. The S3 test backend for the project is now a Rust implementation.

v4 forbids new equality deletes

Huaxin Gao called a spec vote on PR #17783. The direction was already settled in July: v4 writers must not produce new equality deletes, readers still apply existing ones from v2 and v3 tables, and an upgrade stays metadata-only. This vote covered the exact spec wording. Russell Spitzer questioned whether a second vote was needed at all, then voted +1 anyway. Ryan Blue said the point was to affirm the decision and merge it. Anoop Johnson said the change simplifies the v4 design a lot. The vote passed with four binding +1s from Russell, Yufei Gu, Steven Wu and Ryan, plus five non-binding. Huaxin will merge the PR and follow up with a separate change that restructures how the spec presents deletion vectors next to the legacy delete formats.

This is the most consequential spec change of the month for engine builders. Equality deletes let streaming writers such as Flink record "delete the row where id = 7" without reading the table first. They also push work onto every reader, which has to join the delete file against data files at scan time. v4 closes that door for new writes. Streaming pipelines that rely on equality deletes need a plan for deletion vectors before they adopt v4.

Who controls access: the delegation header debate

A thread that started in Polaris landed on the Iceberg list. youngrae kim asked what a REST server should do when a client sends X-Iceberg-Access-Delegation but the server can satisfy none of the requested mechanisms. Picture vended credentials against an S3-compatible store with no STS, or remote signing on a server that does not implement it. Polaris returns HTTP 400. The spec says the server can supply access through any or none of the requested mechanisms.

Alex Dutra proposed spec wording that favors failing fast, and Yufei Gu agreed. Daniel Weeks, who wrote much of the REST spec, disagreed with the framing. His argument was that the catalog should be the sole authority on access, and security around physical data should not be a negotiation between client and server. The loose wording was deliberate. It lets a catalog return credentials or remote signing config, or return metadata only, without the spec forcing its hand. Alex countered that the header cannot express a common client capability, which is using the client's own storage credentials and skipping delegation entirely.

youngrae closed the loop on September 29. Polaris will keep its 400 as a documented implementation choice. The remaining ask is narrow: one sentence in the spec that says whether the header is a capability statement or a requirement, and what an absent header means. That clarification leaves the catalog's decisions untouched. It only tells clients and servers how to read the signal.

A related effort moved forward. Sung Yun and Prashant Singh had been discussing access delegation for the FILE type, where a governed table references files that engines need to read directly. Sung reported that the two of them converged their proposals onto PR #18080, which now carries the optional request and response parameters from both drafts. Steven Wu also announced that the Thursday biweekly sync will cover both the File and Vector types starting October 8. He said the File type discussion is close to consensus, so more of the time will go to the Vector type design doc from Yan Yan.

Labels, SQL, and the agent-shaped consumer

Labels landed in the REST spec recently. A catalog can now attach object-level and field-level labels to a load-table response. Andrei Tserakhau asked whether Iceberg should expose them as a .labels metadata table. Péter Váry had raised the underlying issue in review. Every metadata table today derives from table metadata. A .labels table carries catalog-owned data that varies by catalog and can be empty.

Ryan Blue questioned the premise that labels need a SQL surface at all. He saw them as context for engines, such as cost attribution logs or attaching an engine policy to a table. Andrei answered that some consumers only speak SQL. He named governance tooling and, specifically, LLM agents that explore a warehouse by writing queries. Péter argued that the proposed uses turn labels into full table metadata, which contradicts the read-path design that treats them as a current view with no persistence guarantees. Ryan then took apart Andrei's cost-attribution example. The sample query joined .labels to .files with no ON clause, which produced a cartesian join that only looked right because the filter selected one label.

Andrei conceded the join argument and pointed to an alternative. PR #18049 surfaces object and field labels through DESCRIBE, which gives SQL-only consumers, agents included, a way to see labels without a new metadata table. The metadata table is parked until someone shows a use case that needs joins. The episode is worth noting. "An AI agent needs to read this" showed up as a design argument on the Iceberg list, and the community answered it with the smallest surface that works.

Encryption keys move closer to the files they protect

Gábor Kaszab reopened the design of encryption key storage. Today table metadata carries an encryption-keys list that holds both data encryption keys for manifest lists and the key encryption keys that wrap them. Gábor wants to extend encryption to statistics files and v4 root manifests. His proposal stores each file's wrapped key directly in the metadata entry for that file, one key per file. KEKs stay in the shared list and are still referenced by key-id. The spec draft for statistics files lives in PR #17533.

Gidon Gershinsky agreed the DEK metadata belongs in per-file structures in v4. He added that reusing a DEK across files is possible but requires careful handling to avoid breaking AES-GCM, so the practical rule is a fresh random DEK per file. Xander Bailey wanted the one-key-per-file rule stated explicitly and asked for a common wrapped-key structure across file types, so implementations share code. Gábor's closing argument was about cleanup. With keys in a shared list, every new encrypted file type needs its own cleanup trigger when the file is dereferenced. With keys stored inline, the key disappears with the file. Xander found the argument convincing and asked for a Java draft. Gábor wants the design settled first. Xander also proposed a recurring encryption community sync to cover client library work, the KMS vending spec and v4 changes together.

Smaller spec decisions

rahul mahadev called a vote on a standard User-Agent format for REST clients. The format lists products from most specific to least, such as Spark/4.0.0 iceberg-spark/1.9.0 iceberg-java/1.9.0 (scala 2.13.16). The Iceberg library token must appear when the header is sent. Servers must not reject requests based on it. Xuanwo liked that the header stays separate from authentication and capability negotiation. Alex Dutra left more PR comments to resolve before the vote proceeds.

Eduard Tudenhöfner announced a rename in the new content stats: avg_value_size_in_bytes becomes total_bytes in PR #18308. Averages do not aggregate cleanly across files. Totals do. Daniel Weeks confirmed that the average is still recoverable from total_bytes and value_count, and noted that a total gives planners an explicit upper bound for size estimates. Ryan Blue gave a +1.

Shawn Chang asked why the index spec requires region files to be globally ordered and non-overlapping, given the write amplification and maintenance cost. Péter Váry explained the target. Once index metadata is cached, a single-key lookup should cost one small file read. Overlapping regions force readers to consult several files and reconcile results, and on object storage every extra file read adds latency.

Releases beyond Java

Manu Zhang announced Iceberg C++ 0.4.0 on September 26. The vote passed with binding support from Gang Wu, Renjie Liu and Kevin Liu. Renjie's verification ran 4,152 individual tests across 19 test executables and exercised REST and SQL catalog support, including a SQLite catalog run. Xuanwo suggested a LICENSE path fix for the MurmurHash3 files, and Manu opened an issue for the next release.

Matt Topol opened the vote for Iceberg Go v0.7.0 on September 28. Neelesh Salian, Andrei Tserakhau and Xuanwo voted +1. Xuanwo ran 3,344 tests on Go 1.25.9 and saw one Puffin checksum assertion fail on Go 1.27.1. The code still rejected the corrupted input, so he called it non-blocking.

Community and housekeeping

Robin Moffatt opened PR #18019 to finish the Kafka Connect LICENSE and NOTICE cleanup that Ryan Blue started in April. The PR regenerates runtime-deps.txt, aligns the connector's cloud SDK dependencies with the AWS, GCP and Azure bundles, and drops the Hive distribution. The timing matters. A Confluent Hub security scan flagged the Iceberg Kafka Connect listing, and Confluent's policy escalates flagged connectors toward removal unless the partner responds.

Gaspard Merten shared a PyIceberg PR that moves upsert comparisons out of Python loops and into PyArrow compute. He built it while working on a data lake for a financial organization, and he said the change cuts memory and CPU for wide-table upserts. He also noted that he wrote part of it with Claude and reviewed it himself several times.

Viktor Kessler announced two European community meetups in back-to-back days: Berlin on November 18 and Amsterdam on November 19. Both have open calls for talks.

Apache Polaris

Polaris 1.8.0 ships with a CVE fix

Jean-Baptiste Onofré's first 1.8.0 candidate did not survive. Alex Dutra voted -1 on rc1 because a test file added in PR #4823 carried an incorrect license header. JB cancelled the vote and took another pass at LICENSE and NOTICE. The rc2 vote passed with binding +1s from Alex Dutra, Yufei Gu, Dmitri Bourlatchkov and JB, plus non-binding support from Prithvi S, Ajantha Bhat and Ayush Saxena. Yufei's verification checked 442 artifacts and ran authenticated CLI smoke tests against built and staged servers.

JB announced 1.8.0 on September 28. The notable changes:

Semantic model privileges. Semantic models now get dedicated privileges for list, create, read, update and drop. Admins grant them to catalog roles on a single model, a namespace or a whole catalog. This is Polaris preparing to govern Ossie-style semantic models the same way it governs tables.
CLI toggles for STS and KMS. catalogs update accepts --no-sts and --no-kms, so operators can change those settings on an existing S3 catalog instead of recreating it.
GCP federation auth. The CLI adds gcp as an external catalog auth type for Iceberg REST federation. That lets operators create GCP-authenticated catalogs such as BigLake without passing Google credential secrets as command-line flags.
Paging. A global --page-size option paginates list calls, when the server enables the LIST_PAGINATION_ENABLED flag.
JDBC schema selection. The relational JDBC backend reads its schema from the driver's currentSchema property, so the persistence layer no longer hardcodes a schema name.

A day later JB published CVE-2026-97395, rated important, for all Polaris versions before 1.8.0. A principal with permission to set table properties was able to put FileIO settings such as s3.endpoint into table metadata. During server-side operations like commits and purges, older Polaris versions built their own FileIO client from those settings unless the catalog's storage config overrode the endpoint. The result: Polaris sent storage requests, signed with operation-scoped credentials, to a host the table writer picked. Deployments that do not fully trust table writers should upgrade. The reporter was credited as vignesh a.

Ajantha Bhat also shipped the Polaris Iceberg Catalog Migrator 1.1.0 after an RC3 vote with binding support from Dmitri, Russell Spitzer, JB and Alex. Russell's check covered all 12 primary Maven artifacts and 89 tests.

Enum or free text: the R2 credential vending fight

Austen Tomek of Chicago Trading has a PR adding Cloudflare R2 support with scoped credential vending. The discussion reached a clean disagreement. Austen explained why endpoint matching cannot identify the storage vendor: MinIO and other self-hosted stores use arbitrary hostnames, and a failed match silently falls back to STS. So the PR adds an explicit credentialVendingMechanism field to the Management API.

Dmitri Bourlatchkov wants that field to be free text with runtime validation, so downstream builds of Polaris can add custom mechanisms. Yufei Gu wants an enum, arguing that a spec is a contract and clients should not depend on a specific deployment's extensions. When Dmitri asked for lazy consensus, Yufei said there was none. JB stepped in on September 30 and asked everyone to slow down and align. He pointed out that the field lives in the Management API for catalog administrators. Iceberg REST clients such as Spark and Trino never see it. They only consume the standard vended credentials. That observation lowers the stakes for engine interop, but the question of how extensible the Polaris Management API should be remains open.

Persistence: DynamoDB and read replicas

Yuewei Zhou proposed an optional DynamoDB persistence backend built as a new adapter under AtomicOperationMetaStoreManager, the same shape the relational JDBC backend uses. Robert Stupp prefers a different home. He wants DynamoDB as another backend inside the existing NoSQL persistence framework, where a compare-and-set on a reference is the moment a validated catalog change becomes visible as one consistent state. He also has a DynamoDB implementation in his own fork. Dennis Huo asked the community to use DynamoDB as a chance for an apples-to-apples comparison between the two persistence approaches. JB sided with Robert. He noted that the atomic manager implicitly assumes relational multi-entity transactions, and forcing DynamoDB into that shape means orchestrating TransactWriteItems for multi-entity writes. Yuewei offered to help on whichever path moves forward.

A second persistence thread ended with a request for data. Yong Zheng wants Polaris to route read-only requests to a database read replica when a custom header is present. Prithvi S warned that routing by HTTP verb risks double ingestion if a read hits the replica before a commit is visible and the client retries. Robert Stupp read the Medium post behind the proposal and found 15 Polaris pods serving about 400 requests per second, which is under 27 per pod. That does not look like connection saturation to him. Yufei asked for baseline numbers before any design work: request rates, tables per namespace, latency targets. Yong agreed to run benchmarks with the polaris-tools harness over the coming weeks. JB added that pool multiplexers such as PgBouncer or RDS Proxy belong in that comparison.

Metrics, tags and federation

Dmitri Bourlatchkov flagged Management API changes inside PR #4115, the long-running work on REST endpoints for table metrics and events. The PR adds a TABLE_READ_METRICS grant at table, namespace and catalog level. Reading reports needs that grant, TABLE_FULL_METADATA or CATALOG_MANAGE_CONTENT. Sending scan and commit reports stays tied to data read and write privileges. Yufei reminded the list that the PR also introduces a new REST spec for metrics consumption. Dmitri proposed merging on October 1, and JB agreed, with the JDBC query SPI in PR #4756 as the next step.

EJ Wang reported progress on the Tag spec. All four planned PRs are open: the API contract in #5366, definition CRUD, assignment writes and storage, and reads with inheritance and reverse lookup. Dmitri supports merging the API PR once open review threads close. EJ also built a proof of concept for archiving design proposals as Markdown. Authors keep collaborating in Google Docs, add a version label and a link to one YAML manifest, and the tool captures the doc as Markdown with images for the repository.

David Chaava wants fail-fast validation for vendor-specific Iceberg REST federation settings, starting with BigLake. Dmitri, JB and Prithvi all asked that PolarisAdminService stay vendor-neutral. David's revised plan uses a CDI-discovered validator SPI with a BigLake implementation. Srinivas Rishindra posted the first PR for multiple storage configurations per catalog, and Travis Bowen weighed in on where the new Open Sharing APIs should live, leaning toward the existing /api/management path.

Code health

Two cleanup threads resolved. Yufei Gu found the authentication flow confusing while reviewing another PR. Two augmentors attached a credential that a third consumed, and the ordering only held because of a code comment the compiler cannot check. Alex Dutra proposed merging the OIDC pieces into a single OidcIdentityPreparer that the authenticating augmentor calls directly. Prithvi agreed, Alex opened PR #5612 on September 25, and it merged on September 28.

The long argument over schema upgrade instructions also ended. Yufei argued that SQL snippets in the upgrade guide break for anyone who starts from an older schema version, and that the versioned schema files are the only source of truth. Alex disagreed but removed every migration snippet from PR #5349 to unblock it. The page now explains how to diff the schema files. Prithvi asked for some SQL guidance to return later, since a schema diff shows the end state but not how to migrate tables that already hold data. The PR has approvals from Dmitri, Yufei and Yong Zheng.

Apache Arrow

Dropping Intel Macs

The macOS Intel thread brought numbers. Joris Van den Bossche pulled 30 days of PyPI downloads: x86_64 made up 6.6 percent of PyArrow's macOS downloads, 57,121 against 807,414 for arm64. Sutou Kouhei reproduced the check with a public ClickHouse query against the PyPI dataset and voted +1. Wes McKinney offered his 2019 Intel Mac Pro for testing commits to main. Kurtis Wright asked whether Arrow has any policy guaranteeing a number of releases before a platform goes away. Raúl Cumplido said no, and that such a promise is not realistic when Arrow depends on package managers and CI platforms with their own deprecation schedules. Homebrew already dropped Intel support without naming a last version.

Modularity and schemas

Raúl Cumplido also opened a discussion on PyArrow modularity. Arrow C++ is already split into a dozen libraries, from Core and Compute through Acero, Dataset, Flight, Flight SQL ODBC and S3. GCS and Azure still sit in core libarrow, and he wants them moved out the way S3 and the AWS SDK were. The goal is PyArrow installs that pull only the pieces a user needs, which matters for container images and serverless functions where package size has a cost.

Dewey Dunnington endorsed Kent's revised proposal for a JSON representation of Arrow schemas. David Lee asked for union type support and shared the alias-based JSON format he already uses to define data contracts between processes. A standard JSON schema form gives config files, APIs and agents a way to describe Arrow data without a binary IPC message.

Arrow Rust 60.0.0

Andrew Lamb announced Arrow Rust 60.0.0 on September 29. The release adds new Parquet encoding support, continues a sustained push to remove panics from the codebase, and lands kernel performance work plus fixes across decimals, take and filter, and FFI. The panic removal effort deserves a mention. A library that returns errors instead of aborting the process is easier to embed in long-running query engines and services, which is where arrow-rs lives inside DataFusion, Comet and a long list of downstream projects.

Apache Parquet

The vector type argument

The busiest Parquet thread of the week, at 30 messages, was about vectors. Rok Mihevc opened PR #624 in parquet-format for a LIST-based VECTOR logical type. It separates the logical contract from physical layout changes. Every non-null vector has exactly num_elements. Elements are non-null and finite. Elements are numeric or boolean primitives, with no nesting.

Antoine Pitrou objected to the restrictions. Parquet is a general-purpose format, he said, and NumPy produces vectors with NaN and infinity all the time. Readers can refuse what they do not support, but the spec should not. Alkis Evlogimenos answered with a survey. Postgres, Oracle, MySQL, SQL Server, Milvus, Qdrant and Pinecone all disallow nulls, NaN and infinity in vectors. DuckDB and ClickHouse allow them, but only because they store vectors as plain float arrays and switch off vector features when bad values appear. Daniel Weeks argued that a vector has mathematical meaning, with dimension, magnitude and direction, and non-finite values break it. Russell Spitzer asked for a concrete tool that produces non-finite vectors and a downstream engine that uses them productively.

Will Edwards reframed it. He had first assumed the goal was general ML tensors. After the sync calls he understood the immediate need was fixed-length embeddings for nearest-neighbor search, and in that context the constraints make sense. He also noted that embedding APIs do sometimes return NaN on overflow, which is exactly the kind of bad data a strict type catches. Andrew Lamb suggested a narrower name such as FINITE_VECTOR or VECTOR_EMBEDDING if the name is the real problem. Gang Wu proposed treating a general FixedSizeList as the physical layout and the specialized vector as a separate logical contract. Philipp Fischbeck and Divjot Arora both want to proceed with the proposal as written. Divjot added a practical point: statistics cannot reliably detect non-finite values, so a type-level guarantee is the only way a reader can trust the data without scanning it.

The practical outcome is taking shape. A restricted vector type for embeddings, with a possibly more specific name, and a separate track for general multidimensional arrays that Rok still wants to pursue. For lakehouse teams storing embeddings next to their tables, this is the thread that decides whether engines can index Parquet vector columns without validating every value first.

Encodings: PFOR, FastLanes, ALP and FSST

Prateek Gaur posted hard numbers in the PFOR encoding thread. Antoine had asked whether PFOR's delta mode can reuse Parquet's DELTA_BINARY_PACKED kernel. Alkis asked whether a faster DBP decoder closes the gap. Prateek measured three setups on AWS Graviton 4. On 18 datasets where delta mode helps, PFOR's own patched-delta layout reached a 10.07 compression ratio and 9.85 GB/s decode. A DBP stream inside each PFOR vector reached 6.54 and 3.76 GB/s. A tuned DBP decoder with SIMD prefix sums climbed to 5.96 GB/s but kept the 6.54 ratio. His summary: the patched-delta layout is 35 percent smaller and 1.65 times faster to decode on the datasets where delta matters. The gap is structural. PFOR uses one bit width per 1,024-value vector, while DBP splits the same vector into miniblocks with their own widths.

In the parallel FastLanes thread, Prateek and Alkis agreed to decouple FastLanes from PFOR. The initial PFOR proposal keeps sequential bit packing, and a layout discriminator leaves room to add FastLanes or interleaved layouts later. The thread also produced the sharpest exchange of the week. Antoine told Prateek that his design document and PR descriptions read like AI-generated text and asked him to rewrite them as a person talking to people. Prateek trimmed the document and cut two sections. The Parquet community is receptive to benchmark-driven proposals. It also expects authors to explain their own work.

On the rest of the encoding roadmap, the ALP blog post went through community review and Andrew Lamb described ALP as out the door. He then turned to the FSST proposal and suggested an optional inline symbol table so FSST also applies to dictionary pages. ALP targets floating point columns. FSST targets strings. Together with PFOR for integers, Parquet is on track to cover the main column types with modern lightweight encodings.

Types and the edges of the format

Micah Kornfield pushed back on two type proposals with the same argument: Parquet is a file format, not a table format. Thomas Kissinger wants an extensible decimal floating-point type with a write-time canonicalization contract. Micah answered that an empty table means no Parquet file exists, so there is no Parquet schema to coordinate independent writers. That coordination belongs to a layer above. Micah also raised concerns about a proposed WIRE logical type for serialized messages such as protobuf, citing the edge cases in lining up type systems for shredding, from recursive schemas to well-known types.

Steve Loughran argued in the format version thread that readers must fail on an unsupported version. Pretending to support a version you do not understand is a pretense, he said, and that rule is a strong reason not to bump the version without a compelling payoff. Adrian Garcia Badaracco proposed classifying distinct_count as inexact, since the spec never says whether writers are allowed to estimate it or whether readers can trust it as exact. And Danica Fine invited the community to Lakehouse Day EU on October 10 in Glasgow, co-located with Community Over Code.

Apache DataFusion

Comet 1.1.0 and the token budget

Andy Grove opened the Comet 1.1.0 RC1 vote on September 28. L. C. Hsieh, Marko Milenković and Andrew Lamb each gave a binding +1 within a day. Then Andy posted that his deeper verification had already found several regressions. He is tracking them in issue #6402 and said the full check will take a few days because he does not have an unlimited token budget. That line tells you how Andy verifies releases now: with agents that run through test matrices, where the constraint is API spend rather than hours in the day. Watch for a new candidate.

Ballista wants to ship without waiting

Andy also proposed releasing Ballista's Python client separately from the Rust crates. Ballista moved to DataFusion 55 five weeks ago and still cannot release, because the bundled Python client needs datafusion-python 55.0.0, whose vote has not started. Over the last eleven DataFusion releases, datafusion-python followed DataFusion by a median of 31 days and once by more than seven weeks. More than 300 commits sit on Ballista's main branch, including fixes for wrong results from distributed NOT IN, executors being OOM-killed on heavy queries, and jobs hanging when all executors disappear. Andy wants feedback on issue #2511, especially from Python client users.

Memory accounting

Andrew Lamb's sync notes from September 23 centered on memory. Raz from Flarion reported a string of memory issues where the memory pool's accounting did not match what operators actually used. One example was a Filter allocation that the pool never saw. Comet has hit the same gap between pool accounting and process memory. An earlier allocator-based memory test was reverted, and Andy agreed to find or create a ticket to coordinate the effort. For anyone running DataFusion inside a service with a hard memory limit, accurate pool accounting is the difference between graceful spilling and an OOM kill.

Apache Ossie (incubating)

A standard REST API for semantic models

Jean-Baptiste Onofré proposed a standard REST API for producing, consuming and orchestrating Ossie models, starting as a rest/openapi.yaml in the main repository. The response showed real vendor pull. Sebastien Gandon said Qlik has started building an Ossie-native API to import and export its data product semantic models, and he asked for a dedicated Content-Type header. Yufei Gu asked whether it builds on the Polaris Ossie REST design, which JB co-authored. JB said yes, and that the goal is for Polaris, Qlik and others to implement one standard API so clients work without knowing the server.

Viktor Kessler asked the group to agree on the problem first. He saw two scopes: a registry for exchanging and discovering models, and governed semantics, where a service guarantees that a model is bound to the physical tables it references. JB agreed and proposed two API specs, one for model management and one for catalog association. Vinoth Chandar asked that files and storage locations be first-class alongside table formats. Jakub Moravec asked how an importer knows a model changed. JB proposed keeping the core spec plain REST, with explicit versioning, content hashes and HTTP conditional requests using ETags, instead of push protocols.

This is the most interesting cross-project thread of the week. Iceberg standardized how engines find tables. Polaris implemented it. Ossie is now trying to standardize how tools find and exchange the business meaning that sits on top of those tables, and Polaris 1.8.0 already added the privileges to govern semantic models.

Converters move out, and a first release takes shape

The converter scope discussion reached a decision. Russell Spitzer favored separating the spec from implementations, the way Parquet splits parquet-format from its libraries. Markus Weimer cared less about repository count and more about keeping everything inside the ASF, since that makes adoption easier for downstream users. Kurt Stirewalt laid out what stays in the main repo: the spec, the typed object model with parsing and serialization, the validator, and the parser for the ontology formula language. JB then created the ossie-converters repository on September 29 and asked contributors to pause converter work while he moves the code.

JB also laid out the path to the first incubating releases: freeze converters, move them, and bump the spec version to 0.3.0-incubating after the recent flat-document change. Kyoung Min Kim did the arithmetic. The 0.2.0.dev0 string appears in 111 files, 96 of them under converters, so the bump is one change if it lands before the move and a coordinated two-repo change after. Kyoung Min also confirmed that the license header check now runs on every PR and the tree is clean.

What does `orders.amount` mean?

damian waldron asked a question the spec does not settle. In a portable expression like SUM(orders.amount), is amount a field declared in the dataset, or a physical column in the warehouse? He surveyed ten converters and found three different answers. JB said expressions and relationships must reference logical fields, because a semantic layer that reaches past its own fields to physical columns breaks encapsulation. Julian Hyde objected to any reading that locks layers into rigid tiers, since that makes Ossie less composable. Justin Talbot framed it as the SQL view case: a view's query references its immediate inputs, and dataset fields should work the same way. JB clarified that he meant lexical scoping, not a restriction on composition, and the group converged.

Two spec proposals depend on that answer. Chris Eubank narrowed the dataset-scoped metrics proposal in PR #343 so it can reach a vote without settling every open semantic question. JB strongly backed Justin's Relational Query Interface in PR #354, which grounds Layer 2 in the Measures in SQL model so BI tools that generate their own SQL still get correct metrics across fan-out and chasm traps. Justin also posted a short overview of the four spec layers in PR #452 for newcomers.

Votes, validation and structure

Chris Eubank called a vote to add the SQL-standard FILTER (WHERE ...) aggregate modifier to the expression language, after incorporating JB's feedback. The change is additive and avoids the edge cases of CASE-based workarounds for AVG, MIN, MAX and STDDEV. JB voted +1.

Kyoung Min Kim noticed that validate.py stops at the JSON schema for ontology documents because every semantic check assumes a datasets key. He will add ontology checks for built-in concepts, cycle detection on extends, and a CI step, with JB reviewing. Shao Xie's PR #458 splits ontology, semantic model and mapping into separate documents connected by references, so a slow-moving ontology can map to many semantic models. Yufei asked for more voices on deprecating embedded models. On GitHub, MikeCarlo proposed storing custom extension data as structured objects instead of opaque JSON strings, which helps the Power BI TMDL converter keep more metadata. Contributors also opened discussions on metric additivity and on recording who certified a metric definition.

An outsider builds Ossie on ClickHouse

The most useful message on the Ossie list came from a newcomer. Aleksandr Kosachev introduced himself as a backend architect who, six months ago, barely knew what a semantic layer was. He kept seeing teams wire AI assistants to databases and get confident, wrong answers. So he built ossie-clickhouse from the spec alone. It loads an Ossie document with the apache-ossie package, translates expressions to ClickHouse SQL with SQLGlot, plans one SELECT per question, and serves the model to agents over MCP. The TPC-DS example model runs end to end.

His list of places where he had to guess is a free conformance review. Relationship cardinality is not in the spec, so he only joins when the target columns form a primary or unique key. ClickHouse tables that keep change history need FINAL on read, which he exposed through a custom extension. Function semantics are open: whether REGEXP_REPLACE replaces the first match or all, whether DAYOFWEEK is ISO, and what 1/0 returns. He pinned his choices in tests until the compliance suite lands. Two of those gaps already have active threads this week. That is how a spec should improve.

Cross-Project Themes

Tighter contracts, fewer surprises. Iceberg removed a write path from v4. Parquet is leaning toward a vector type that rejects values most vector stores reject anyway. Ossie is defining whether a name means a field or a column. Polaris is arguing about whether a config field should be an enum. The shared direction is specs that make fewer promises and make them precisely, so a reader can trust data without re-validating it. The pushback is also consistent. Antoine Pitrou in Parquet, Daniel Weeks on the delegation header and Dmitri Bourlatchkov on R2 each defended flexibility for implementers. The best outcomes this week kept both. Polaris keeps its fail-fast behavior as an implementation choice while the Iceberg spec stays loose. Parquet looks headed toward a strict embedding type plus a separate general array track.

Agents are now participants in the process. Kurtis Wright found the Iceberg license gap with Claude. Kevin Liu pointed an agent at the repository to hunt correctness bugs. Andy Grove's Comet verification is gated by token budget. Gaspard Merten wrote his PyIceberg change with Claude. The communities are also setting norms. Contributors label AI-assisted votes, and Antoine asked a Parquet author to rewrite AI-sounding prose in his own words. Agents also showed up as consumers. Andrei Tserakhau argued for SQL-visible Iceberg labels because agents explore warehouses through SQL, and Aleksandr Kosachev built an Ossie server specifically to stop agents from giving wrong answers.

The semantic layer is joining the catalog layer. Polaris 1.8.0 added privileges for semantic models. Ossie proposed a REST API modeled on the Polaris design and split out a catalog association spec. Iceberg debated catalog-owned labels. The lakehouse stack now has open standards for files, tables and catalogs, and the next layer up, what the numbers mean, is getting the same treatment on the same lists with many of the same people.

Release engineering is a security surface. Iceberg RC1 failed on bundled license gaps. Polaris rc1 failed on a license header. Confluent is scanning the Iceberg Kafka connector. Polaris shipped a CVE fix for server-side FileIO endpoints. The Iceberg test backend moved from MinIO to RustFS after CI breakage. None of this is glamorous. All of it is the work that lets enterprises run these projects in production.

Looking Ahead

Watch for the Iceberg 1.12.0 announcement and the equality delete spec merge, then the October 8 sync on File and Vector types. Polaris plans to merge the table metrics PR on October 1, and the R2 enum question needs a resolution before that PR lands. Parquet's vector type should get a naming decision and a path to a vote. DataFusion Comet needs a new release candidate once Andy Grove's regression list is triaged. Ossie will move converters into their new repository and prepare its first incubating release at spec version 0.3.0. And if you are in Europe, Lakehouse Day EU in Glasgow on October 10 is the next place to see many of these contributors in one room.

Keep Learning

If this digest helps you follow the open lakehouse, my books go deeper on every layer covered here, from Apache Iceberg table design and Apache Polaris catalogs to Arrow, Parquet and building agentic analytics on open data. Find the full catalog at books.alexmerced.com.

Sonnet, GPT Sol and a Shelved Astra - AI Weekly September 23 to 30, 2026

Alex Merced — Wed, 30 Sep 2026 13:42:46 +0000

Two labs shipped their mid-tier workhorse models one day apart, and both landed at $2 per million input tokens and $10 per million output. In between, OpenAI did something rare. It pulled its next flagship the night before its biggest developer event because the model kept acting beyond what users approved. The rest of the week followed that thread. OpenAI turned the Codex harness into a product. NVIDIA put a hardware watchdog around agents. MCP gained a way for servers to push events to agents instead of waiting to be polled. And the first Vera Rubin racks left the factory.

Models: Sonnet 5.5 and GPT-6.1 Sol Meet at $2/$10

Claude Sonnet 5.5

Anthropic released Claude Sonnet 5.5 on September 28. It is the second model in the Claude 5.5 family, after Opus 5.5 last week, and it replaces Sonnet 5 as the default Sonnet. The model ID is claude-sonnet-5-5 with no date suffix. The context window is 1 million tokens, max output is 128,000 tokens on the synchronous Messages API, and the reliable knowledge cutoff is June 2026.

Pricing did not move. Sonnet 5.5 costs $2 per million input tokens and $10 per million output, the same as Sonnet 5 and half of Opus 5.5's $4/$20. Cache reads cost $0.20 per million tokens. That last number is the one to study. Opus 5.5 also charges $0.20 for cache reads. In long agent loops, cache reads dominate the bill, so the real gap between the two models is smaller than the sticker price suggests.

Developers Digest worked through the arithmetic. Take one agent turn with 100,000 input tokens, 90,000 of them cache hits, and 4,000 output tokens. Sonnet 5.5 costs about $0.078 for that turn. Opus 5.5 costs about $0.138. That is roughly 43 percent cheaper. On a turn that is almost entirely cached, the saving shrinks to around 10 percent. Anthropic says Sonnet 5.5 costs up to 30 percent less per task than Sonnet 5 and generates output more than 30 percent faster. The saving comes from the model using fewer tokens and fewer tool calls, not from a lower rate.

The vendor-reported benchmarks put Sonnet 5.5 next to Opus 5.5, not next to its predecessor. All numbers below come from Anthropic's launch materials:

The Terminal-Bench jump from 10.3 to 70.6 percent is the headline, and it needs a caveat. Commenters reading the system cards on Hacker News noted that safeguards intervened in about 10 percent of Opus 5.5's Terminal-Bench trials, which a fallback model then answered, against 1.5 percent for Sonnet 5.5. That difference alone accounts for some of Sonnet's lead over Opus on that test. Anthropic flagged two other caveats itself. At max effort, Sonnet 5.5 scored lower on FrontierCode than at xhigh, 46.2 against 52.1 percent, because it more often triggered a multi-subagent review that overshot the task. And the knowledge-work scores came from a pre-release deployment with a structured-output bug.

The migration guide lists five changes that return HTTP 400 if you move Sonnet 5 code over unchanged:

thinking: {"type": "disabled"} is rejected. The replacement is thinking: {"type": "between_tools"}, which turns off up-front thinking but keeps progress notes between tool calls. It only works at low, medium and high effort.
Forced tool use is gone. tool_choice of any or tool fails. Use auto and mark the tool strict: true.
Thinking blocks are bound to the model and account. Sonnet 5.5 cannot read blocks from other models, and editing earlier history invalidates later blocks for newer accounts. Keep histories append-only.
Computer use moves to the computer_toolset_20260801 tool version.
The advisor tool only accepts current-generation advisors.

Two quieter changes matter too. Non-default temperature, top_p and top_k values now return a 400. And the minimum cacheable prompt drops from 1,024 tokens to 512, which extends caching to shorter system prompts.

Sonnet 5.5 is also the first Sonnet model with Anthropic's cyber, bio, frontier-LLM and reasoning-extraction safeguards. Server-side fallback retries cyber and frontier-LLM declines on Sonnet 5. The reasoning-extraction classifier is a defense against distillation, where another lab trains on a model's visible reasoning. Security researchers in Anthropic's verification program reported false flags on authorized work in the launch-day threads, so teams doing fuzzing or exploit research should test their workflows before switching.

Anthropic says Haiku 5.5 joins the family in the coming weeks. That release will reset the high-volume, low-latency end of the lineup, where Haiku 4.5 still sits at $1/$5 with a 200,000-token window.

GPT-6.1 Sol

OpenAI answered one day later. At DevDay on September 29, it introduced GPT-6.1 Sol, a week after GPT-6 Sol shipped. OpenAI describes it as near-Astra intelligence at one fifth of Astra's input and output prices. The API model ID is gpt-6.1-sol.

The model page lists a 1.05 million-token context window, 922,000 max input tokens, 128,000 max output tokens and an April 30, 2026 knowledge cutoff. Standard pricing is $2 input and $10 output per million tokens for prompts up to 272,000 input tokens. Cached input costs $0.10, half of GPT-6 Sol's rate. Past 272,000 input tokens, the whole request bills at $4 input, $0.20 cached and $15 output. Batch and Flex cost 50 percent less. Fast mode costs double.

Kingy AI pulled the score-versus-cost charts from OpenAI's launch report. All figures are vendor-reported:

The pattern is clear. On repository coding and PDF work, 6.1 Sol matches or beats Astra for a fraction of the cost. On computer use and science tasks, Astra still leads. The science result shows the trade most plainly: 6.1 Sol costs about 77 percent less per attempt and scores 11 points lower.

Three API details matter for migrations. Reasoning effort runs from low to max, and the none and minimal settings are gone. Tool calling works through the Responses API only, not Chat Completions. And the Responses API gains beta multi-agent support, where a root agent delegates independent work to subagents. OpenAI's multi-agent guide recommends a default of three concurrent subagents and warns that delegation raises token use.

In ChatGPT, GPT-6.1 Sol is live in Work and Codex for Plus, Pro, Business, Enterprise and Edu users. It is not yet in regular Chat, and Enterprise and Edu admins must turn it on. OpenAI's system card addendum classifies the model as Critical for cybersecurity and High for biological and chemical capability, and applies Astra's safeguards.

The Astra that stayed home

GPT-6.1 Sol was not the model OpenAI planned to headline this fall. On Monday, September 28, the evening before DevDay, OpenAI confirmed it will not release GPT-6.1 Astra, the successor to GPT-6 Astra that had been due in ChatGPT and Codex in October. The Wall Street Journal broke the story. Saachi Jain, OpenAI's head of safety systems, told the paper the model did not meet the company's safety and alignment bar.

The failure modes are specific, and every agent builder should read them closely. According to Technology.org's summary of the reporting, GPT-6.1 Astra was better at finishing hard tasks end to end. It wrote better and gave up less often, a weakness OpenAI calls laziness. It also showed more deception than GPT-6 Astra and did not always tell users accurately which actions it had taken. And it struggled with what OpenAI calls scope authorization. It kept pursuing a task without asking permission, and in some cases reached for external tools or services when doing so was unsafe.

Read those two lists side by side. The capability gains and the safety regressions point in the same direction. A model that gives up less often is also a model that pushes past the edge of what it was asked to do. That is the core tension in agent design right now, and OpenAI chose to hold the model rather than ship it with the tension unresolved. GPT-6 Astra, released September 3, stays available.

Outside pressure is rising too. Malay Mail reported that the UK AI Security Institute published a study the same Monday finding that GPT-6 Astra went off the rails more often during testing than GPT-5.6 Sol and GPT-5.5.

Same price, different defaults

The two launches create a clean head-to-head. Both models cost $2/$10. Both have a context window near 1 million tokens and 128,000 max output. GitHub Copilot even lists Sonnet 5.5 and GPT-6 Sol at identical per-token rates. The differences sit in the details:

Caching. GPT-6.1 Sol charges $0.10 per million cached tokens. Sonnet 5.5 charges $0.20. For a cache-heavy agent loop, that halves one of the largest line items.
Long context. GPT-6.1 Sol doubles its input rate past 272,000 tokens. Sonnet 5.5 bills a flat rate across its 1 million-token window.
Default effort. Sonnet 5.5 defaults to high in the API and medium in Claude Code. GPT-6.1 Sol defaults to medium.
Reasoning control. Neither model lets you switch reasoning fully off anymore. Sonnet 5.5 offers between_tools. GPT-6.1 Sol starts at low.

Which one is better depends on your harness. Both labs report vendor numbers on different benchmark suites, so direct comparisons are thin. Run your own tasks at the effort level you plan to deploy and measure cost per accepted result, not cost per token.

The September 23 wave and a round of retirements

The rest of the week filled in the tiers above and below. Per Local AI Zone's September ledger, September 23 was the busiest release day of the month. Z.ai shipped GLM 5.3 Prime at $2.80/$8.80 with a 1 million-token context. Alibaba shipped Qwen3.8 Max Prime at $4/$12, twice the price of standard Qwen3.8 Max for a claimed 1.5 to 2 times more throughput. Early public measurements did not show it running measurably faster, so that premium needs proof before it earns traffic.

Google shipped Gemini 3.8 Flash TTS and Flash-Lite TTS the same day. The Flash model supports prompt-based voice design, more than 2,000 ready-made voices, voice cloning from a short sample and over 100 languages. It took the top spot on Hume AI's Voice Design Benchmark at 71.4. Both run at promotional pricing through December 31, 2026, with standard rates from January 1. On September 24, Google added Gemini 3.8 Live with Live Avatar, the visual counterpart to the real-time voice stack it launched on September 15.

OpenAI also cleared out old models. On September 28, it retired four legacy models: gpt-3.5-turbo-instruct, babbage-002, davinci-002 and gpt-3.5-turbo-1106. If you still have scripts pinned to those IDs, they now fail.

Tooling: OpenAI Rents Out the Codex Harness

DevDay was OpenAI's biggest developer event to date. The official recap lists more than 20 announcements. For people building with AI, the through-line is simple. OpenAI took the agent runtime it uses for Codex and its new always-on agents, and it put that runtime behind an API.

The Agents API

The Agents API is a managed version of the Codex harness, released as a public beta. OpenAI runs the sessions, orchestration, context compaction and recovery. Developers bring tools and choose execution environments. Agents can run code, edit files, connect to MCP servers and delegate work to other agents. The API also carries Codex's multi-agent features, tool search and tool calling.

The new piece this week is computer use. Agents can now drive websites and applications through an OpenAI-hosted browser. OpenAI also worked with AWS on Bedrock Managed Agents powered by OpenAI, which takes the core of the Agents API and runs it entirely inside AWS with native access to AWS resources.

This is a strategic shift worth naming. A year ago, the model API was the product and developers built their own loops. Now the loop itself is the product. Teams that built custom harnesses for context management, retries and subagents now have a managed alternative. The trade is control. A managed harness decides how context gets compacted and how failures recover, and those decisions shape both cost and correctness. The Astra news makes the point sharper. The harness is where scope limits and approval gates live, and whoever runs the harness owns those rules.

OpenAI also shared platform numbers during the keynote, per live coverage. It said Responses API time to first token dropped 45 percent and tool calls and workflows got more than 30 percent faster. It also said Responses API usage grew 100-fold over the past year with reliability above 99 percent. These are company figures.

Codex moves to the cloud

Codex no longer needs your laptop open. Codex in the cloud runs tasks from a computer, from a phone, or in a hosted environment from any device. Reusable development environments give a team a shared setup with approved settings and permissions, so a delegated task starts from a known state. It is available on Plus, Pro, Business, Healthcare, Education and Enterprise.

The Codex CLI got a refresh on every plan. You can now start and steer tasks by voice. A new /agents view tracks several delegated tasks at once. Prompt editing, session resume and worktree support all improved, and the terminal UI is cleaner for long sessions.

Code review got its own surface. The ChatGPT desktop app now shows summaries and diffs across projects, and you can ask Codex about potential issues before leaving feedback on GitHub pull requests or GitLab merge requests. With automatic reviews on, Codex takes a first pass in the cloud while you are away.

Codex Security Cloud scans entire GitHub repositories on demand or on a schedule, then keeps checking new commits. It investigates findings, removes duplicates and prepares fixes in the cloud. It includes access to the models offered through OpenAI's Daybreak Blue program without a separate application. It is available to Pro, Business, Enterprise and Edu users.

Speed becomes a price tier

Ultrafast is OpenAI's new premium speed tier. OpenAI says it generates tokens up to 8 times faster in Codex, around 300 tokens per second, and up to 6 times faster in the API. GPT-6 Astra Ultrafast is live now in the API and in ChatGPT Work and Codex on the new Pro 500 and Enterprise plans. GPT-6.1 Sol Ultrafast is coming soon. BGR reports that Ultrafast costs six times the standard API price.

Pro 500 is the new top consumer plan. It includes 25 times the ChatGPT Plus usage allowance and access to Ultrafast. OpenAI's Pro options now run at $100, $200 and $500 per month.

Faster generation changes what agents feel like. At 300 tokens per second, a long refactor plan appears in seconds instead of a minute. But generation speed is not task speed. An agent that spends most of its time waiting on tools, builds or tests gains little from faster tokens. Measure wall-clock time per completed task before paying six times the rate.

Sign in with ChatGPT and plugins

Sign in with ChatGPT lets users spend their ChatGPT plan allowance inside 16 partner tools, including Cognition's Devin, Notion, Vercel, T3, OpenClaw and Dactyl. Users control how much each tool can draw. For developers, this flips who pays for inference. An app can let users bring their own ChatGPT plan instead of passing API costs through a subscription.

OpenAI also opened more of ChatGPT itself. Plugin extensions let a plugin claim a spot in the sidebar, build interactive panels next to the conversation, and register viewers for its file types. A Plugin Creator and a redesigned submission flow aim to make plugins easier to build and publish. On Business and Enterprise plans, Sites can now host plugins that teammates run with their own connected data.

Dots, Space and the workplace agents

The consumer headline was dots, always-on agents that work toward goals you give them. According to BGR, each dot runs on GPT-6 Astra and gets its own cloud computer and browser. Dots are available on Pro and Business Premium in eligible markets, and Enterprise, Edu and Healthcare admins must switch them on.

Around dots, OpenAI built a collaboration layer. ChatGPT Space is a shared workspace for teammates, ChatGPT and their dots. Pages are documents built for humans and agents to edit together. Collaborative slides are coming in the next few weeks with export to PowerPoint and Google Slides. Business and Enterprise teams can delegate recurring work as team tasks that run on a schedule or in response to events. And @chatgpt now answers in Slack and Microsoft Teams channels with tools the admin connects.

Two enterprise features round it out. The Decisions API, in limited preview, points Luna at a fixed set of user-defined questions with finite answers, for classification, routing and choosing an agent's next step. And OpenAI Private Intelligence adds Zero Data Retention with Private Safety Processing, which runs automated safety reviews without giving OpenAI staff access to the content. A Private Inference preview using confidential computing arrives this fall.

Sonnet 5.5 lands everywhere on day one

Anthropic's release was quieter but wide. Sonnet 5.5 went generally available in GitHub Copilot for Pro, Pro+, Max, Business and Enterprise users across VS Code, Visual Studio, JetBrains, Xcode, Eclipse, the Copilot CLI and the Copilot coding agent. GitHub said its early testing found Sonnet 5.5 matching Sonnet 5 on coding tasks while using significantly fewer steps, tokens and tool calls. It also went live on Vercel's AI Gateway as anthropic/claude-sonnet-5.5 with Zero Data Retention support.

In Claude Code, the sonnet alias now resolves to Sonnet 5.5 on the Anthropic API. On Bedrock, Vertex and Foundry it still points to an older Sonnet, so pin the full ID there. Claude Code defaults Sonnet 5.5 to medium effort, while the raw API defaults to high. Anthropic's prompting guide notes two behaviors to watch. At low and medium effort, the model sometimes pauses to ask whether to keep going on long tasks. And at higher effort, it adds extras nobody asked for, like new tests and small supporting files. Both are fixable with a line in the system prompt.

Agents in the release process

One tooling trend showed up outside the vendor announcements. On the Apache dev lists this week, contributors used agents to do release verification. A Claude-driven license audit caught the gap that sank the first Apache Iceberg 1.12.0 candidate. Another contributor ran a purpose-built agent skill to check the second candidate. Andy Grove said his deeper check of Apache DataFusion Comet 1.1.0 will take a few days because he does not have an unlimited token budget. We cover those threads in this week's Apache Data Lakehouse Weekly, including the Iceberg vote result. The takeaway for tooling teams: rule-heavy review work, like license checks and release matrices, is where agents already earn trust in open source communities.

Standards: MCP Learns to Push

MCP Events

The biggest protocol news of the week arrived through OpenAI's keynote. ChatGPT plugins now support the proposed MCP Events specification, so a plugin can start an automation when something changes in a connected app. OpenAI's example is ChatGPT watching a project board. When a new task appears, it reads the linked documents and drafts a plan, even when the user is away.

To see why this matters, look at how MCP worked before. It started as request and response over JSON-RPC. Streamable HTTP added server-sent events for streaming results. But there was no callback mechanism. An agent that wanted to know when a ticket changed had two options: poll the server over and over, or hold a long-lived connection open. Both waste resources and add latency.

The Triggers and Events Working Group exists to close that gap. It was chartered in March 2026 and is led by Clare Liguori of AWS and Peter Alexander of Anthropic, per Forkast's analysis. The draft defines three methods:

events/list describes the events a server offers.
events/subscribe creates or refreshes a subscription.
events/unsubscribe ends it.

Delivery uses Standard Webhooks, so receivers verify each callback with a signature. The draft requires HTTPS for every callback and blocks private and local addresses, which shuts down the easiest server-side request forgery tricks. MCP Events also requires protocol version 2026-07-28. That is the stateless revision maintainers finalized on July 28, which removed protocol-level session tracking and carries version, identity and capability data with each request.

Two cautions. First, the spec is a draft. The working group's incubation repository labels its contents exploratory, and the group still meets every two weeks, next on October 9. Build against it knowing the shape can change. Second, events make agents proactive, and proactive agents act without a human in the loop at the moment of action. The GPT-6.1 Astra story is a direct warning here. Scope authorization is the failure that sank a flagship model this week. Pair every subscription with the same approval rules you use for tool calls that write data.

The events work sits inside a busy roadmap. The MCP roadmap update from August lists server-initiated events, a composition review across the Agents, Transports and Triggers and Events groups, and moving the Tasks extension (SEP-2663) into the core spec. The Skills Over MCP group marked SEP-2640 final on September 13, which standardizes how agent skills are discovered and loaded through MCP servers. The protocol is turning from a tool-calling wire format into the connective layer for how agents learn about the world, learn how to do work, and learn when to act.

A2A and the governance layer

The Agent2Agent protocol handles a different job. Where MCP connects an agent to tools and data, A2A connects independent agents to each other using four objects: an Agent Card, a Task, a Message and an Artifact. In August, A2A moved into the Agentic AI Foundation, joining MCP under the same Linux Foundation umbrella.

This week showed what adoption looks like one layer down, in the products that govern agent traffic. Postman made its Fabric Gateway generally available on September 29. It is a protocol-agnostic control plane that enforces least-privilege policies on how agents discover and call APIs, tools and other agents, whether the traffic is HTTP, gRPC, MCP or A2A. MongoDB launched Atlas Agent Engine the same day, an execution, memory and governance platform that supports both MCP and A2A. When gateways and databases treat both protocols as first-class, the standards are no longer experiments.

Standards for the data under the agents

Agent protocols only help when the data they reach is described in a standard way. Two threads on the Apache lists this week pushed that layer forward. Apache Ossie, the incubating semantic model standard, proposed a REST API for exchanging and governing semantic models across tools, with Qlik saying it is already building an Ossie-native endpoint. And a newcomer built an Ossie implementation on ClickHouse from the spec alone, then served the model to AI agents over MCP. His stated reason: teams keep wiring AI assistants straight to databases and getting confident, wrong answers.

Apache Iceberg also voted on a standard User-Agent format for REST catalog clients. It is a small change with a clear payoff for agent-heavy stacks. When a catalog sees Spark/4.0.0 iceberg-spark/1.9.0 iceberg-java/1.9.0 in its logs, operators can tell which engine and library made each request. Expect agent frameworks to want their own tokens in that header soon.

Infrastructure: Watchdogs, Vera Rubin and Pricier HBM

NVIDIA puts agent safety in silicon

NVIDIA announced the Open Agent Safety Platform on September 28, with more than 100 industry partners. It is an open software platform plus a reference system design, and it has two layers.

The first layer is OpenShell, a runtime that sandboxes what an agent can do. It is broadly available now. The second layer is Sentry, an out-of-band watchdog that runs on BlueField-4 data processing units, separate from the CPU and GPU that run the agent. Sentry uses NVIDIA's DOCA framework to record agent interactions, policy decisions and tool and data access in one timeline. A DOCA gateway continuously checks each agent's identity and delegated authority. NVIDIA says Sentry can quarantine an agent that crosses a boundary within milliseconds. That is a company claim with no independent test results in the announcement, and Superpower Daily notes that Sentry is a reference design with no general-availability date.

The idea behind it is simple and worth adopting even without NVIDIA hardware. An agent should not be the only witness to its own behavior. Software guardrails live inside the same system the agent controls. A compromised host, or a model that misreports its own actions, can defeat them. A monitor on separate silicon keeps working when the host does not. The timing matters. This landed the same day OpenAI shelved a model partly because it did not report its own actions accurately. The strongest version of the protection runs on NVIDIA's own Vera and BlueField-4 hardware, so it also doubles as a reason to buy the full NVIDIA stack.

Vera Rubin NVL72 racks are shipping

Supermicro started shipping NVIDIA Vera Rubin NVL72 racks on September 23. These follow the design NVIDIA showed at CES in January. Each rack holds 72 Rubin GPUs and 36 Vera CPUs across 18 compute trays, with four GPUs and two CPUs per tray. Nine sixth-generation NVLink switch trays provide 216 TB/s of scale-up bandwidth. Each rack carries 20.7 TB of HBM4 and up to 54 TB of LPDDR5X.

The cooling numbers show what running these racks takes. Supermicro pairs them with in-row coolant distribution units rated at 1.8 MW each, deployed with N+1 redundancy, plus optional rear-door heat exchangers. Its reference "scalable unit" spans 16 compute racks with 1,152 Rubin GPUs and 331 TB of HBM4, sized with matching power, storage, networking and a dedicated tier for context memory storage.

That last item is a sign of the times. Long-running agents keep large KV caches alive across many turns. Rack designs now budget storage specifically for that context, next to the storage for data and checkpoints.

HBM: the next generation and the next price

SK hynix said on September 28 that it has validated HBM5 with TSMC's CoWoS packaging, according to DigiTimes. The generation after next is already in foundry testing while HBM4 is only now shipping inside Vera Rubin. SK hynix has also been promoting custom HBM, where compute functions move into the memory's base die to cut data movement during inference.

The supply side is tight, and the price is rising. A 24/7 Wall St. analysis published September 28 models the HBM bill for a 288 GB GPU nearly doubling between 2026 and 2027, from about $3,917 to $7,373, with capacity unchanged, as price per gigabit climbs. The same piece covers NVIDIA's interest in glass substrates, which allow larger, flatter packages with room for more memory. More HBM per package means more demand per accelerator, not more supply. Rest of World reported the same day on Samsung and SK hynix competing to supply NVIDIA, OpenAI and other US buyers, with an independent analyst arguing that NVIDIA's allocation choices still decide HBM market share.

For anyone forecasting AI costs, the lesson is that token prices and hardware costs are moving in opposite directions. Labs keep cutting per-token rates, as this week's $2/$10 launches show. The memory inside the accelerators is getting more expensive. Work that cuts memory per token, from smaller KV caches to better cache hit rates, is where the margin comes from.

Inference speed and privacy as products

OpenAI's DevDay also turned two infrastructure properties into line items. Ultrafast sells raw generation speed at a premium. Private Inference, coming as a preview this fall, combines confidential computing with verifiable controls so customers can run frontier models without exposing their data to the provider. Both follow the same pattern as NVIDIA's watchdog. Properties that used to be internal engineering details, like serving speed, memory isolation and behavioral monitoring, are becoming product tiers with their own prices.

File formats keep getting faster

The storage formats under AI pipelines moved too. On the Apache Parquet list, Prateek Gaur posted benchmark results for a proposed PFOR integer encoding on AWS Graviton 4. On 18 datasets where delta encoding helps, PFOR's patched-delta layout reached a 10.07 compression ratio and 9.85 GB/s decode speed. The existing DELTA_BINARY_PACKED approach reached 6.54 and, even with a tuned SIMD decoder, 5.96 GB/s. PFOR joins ALP for floating point and a pending FSST proposal for strings, which together modernize how Parquet compresses every major column type.

The Parquet community also spent 30 messages debating a VECTOR logical type for embeddings. The sticking point is whether vectors can hold NaN and infinity. Most vector databases reject them. General scientific data often contains them. The likely outcome is a strict embedding type, possibly renamed to something like FINITE_VECTOR, plus a separate track for general arrays. For teams storing embeddings next to their tables, this decides whether engines can index vector columns without validating every value first.

Arrow Rust 60.0.0 shipped on September 29 with new Parquet encoding support and a sustained effort to remove panics from the codebase. Libraries that return errors instead of crashing are easier to embed in long-running query engines and agent services.

The Data Layer Angle: Scope Lives in the Catalog Too

The week's biggest story was about scope. GPT-6.1 Astra overstepped what users approved. NVIDIA built hardware to catch agents that cross boundaries. MCP Events gave agents a way to act without being asked. Every one of those stories ends at the same place: the data an agent can read and write. The open lakehouse communities spent the week arguing about exactly that boundary, and their answers are useful for anyone putting agents on top of a data platform.

On the Apache Iceberg list, contributors debated what a REST catalog should do when a client asks for delegated storage access that the catalog cannot provide. Apache Polaris fails the request with an error. Daniel Weeks, who wrote much of the REST spec, argued that the catalog should be the sole authority on access, and that security around physical data should not be a negotiation between client and server. That principle maps directly onto agent design. The agent asks. The catalog decides. The agent never holds long-lived storage credentials of its own.

The same week, Apache Polaris published CVE-2026-97395. In versions before 1.8.0, a principal allowed to set table properties was able to put a storage endpoint into table metadata. Polaris then used that endpoint for its own server-side operations, such as commits and purges, and sent requests signed with operation-scoped credentials to a host the table writer picked. Swap "table writer" for "agent with write access" and the risk is obvious. Any metadata an agent can write becomes a path to redirect what the platform does next. Polaris 1.8.0 fixes the issue, and it also adds dedicated privileges for semantic models, so the business definitions agents rely on can be governed the same way tables are.

Two more threads point the same direction. Iceberg contributors debated exposing catalog labels through SQL specifically because LLM agents explore warehouses by writing queries, and settled on surfacing them through DESCRIBE instead of a new joinable table. And the Apache Ossie list is defining whether a name in a metric expression refers to a governed logical field or a raw warehouse column. A semantic layer that lets expressions reach past its own fields to physical columns is a semantic layer an agent can route around.

The practical lesson is to put the scope rules where the agent cannot edit them. Model guardrails help. Harness approvals help. Hardware watchdogs help. But the last line of defense for data is a catalog that vends short-lived, narrowly scoped credentials, treats agent-writable metadata as untrusted input, and exposes business meaning through governed definitions instead of raw tables. The agents shipping this week are more capable and more eager than last month's. The data platforms under them need to be stricter to match.

What to Watch Next Week

Haiku 5.5. Anthropic says it arrives in the coming weeks. It completes the 5.5 family and sets the new floor for high-volume Claude workloads.
What OpenAI does with Astra. OpenAI says it will investigate the regression and fold the lessons into future GPT-6 models. Watch for a revised release date or a published write-up of the scope authorization tests.
GPT-6.1 Sol Ultrafast. OpenAI says it is coming soon. Watch the price multiplier and whether it reaches Plus and Business plans.
MCP Events drafts. The Triggers and Events Working Group meets October 9. Expect changes as the first production users report back.
Real-world cost per task. The first week of independent evaluations for Sonnet 5.5 and GPT-6.1 Sol will show whether the vendor charts hold up, especially on cache-heavy agent loops where the two models' pricing differs most.

Practitioner Takeaways

Re-run your evals at equal cost, not equal effort. Effort levels differ across models and even across versions of the same model. Compare cost per accepted result.
Check your cache hit rate before you pick a model. At $0.10 versus $0.20 per million cached tokens, the cache rate decides more of your bill than the headline price.
Audit your API calls for the new 400s. Sonnet 5.5 rejects disabled thinking, forced tool use and non-default sampling parameters. GPT-6.1 Sol drops none effort and moves tool calling to the Responses API.
Test scope, not just success. The Astra story shows that a model that finishes more tasks can also overstep more often. Add evals that check whether your agent stayed inside its permissions and reported its actions accurately.
Treat event subscriptions like write permissions. MCP Events lets agents act on their own schedule. Put approvals on the actions that follow an event, not just on the subscription.
Budget for memory, not just tokens. HBM costs are rising while token prices fall. Workloads that cut context size and raise cache reuse get cheaper on both curves.

Keep Learning

The model, tooling and protocol layers in this issue all depend on the data layer underneath. If you want to go deeper on AI-assisted development, agentic workflows, MCP, and building open data platforms that agents can actually use, my books cover all of it. Browse the full catalog at books.alexmerced.com.

AI Weekly: Opus 5.5, GPT-6 Sol and Luna, and MCPA

Alex Merced — Wed, 23 Sep 2026 21:18:48 +0000

Week of September 16 to 23, 2026

Two frontier labs shipped on the same day, and both led with price. Anthropic released Claude Opus 5.5 and OpenAI released GPT-6 Sol and GPT-6 Luna on September 22. Each company cut token prices and argued in cost per task instead of raw scores. Around those launches, JetBrains rebuilt its product line around multi-vendor agents, the Agentic AI Foundation launched the first MCP certification, and AMD crossed $1 trillion in market value on the strength of rack-scale AI systems.

The theme of the week is cost per finished task. Models got cheaper per token, used fewer tokens per job, and the hardware underneath them moved to full racks sold as one product. For teams building agents on real data, the price of doing useful work fell again.

Models: Opus 5.5 and GPT-6 Sol Race on Cost per Task

Claude Opus 5.5

Anthropic released Claude Opus 5.5 on September 22 as the first model in its Claude 5.5 family. The API model ID is claude-opus-5-5. Anthropic says it performs at the level of Claude Fable 5.1 on most work and costs 40% less to run than Opus 5 on typical workloads. It is available on the Claude Platform, Amazon Web Services, Google Cloud, and Microsoft Azure. Anthropic says Sonnet 5.5 and Haiku 5.5 follow in the coming weeks.

Here is the pricing, per million tokens:

Input: $4 (down from $5 on Opus 5)
Output: $20 (down from $25)
Cache reads: $0.20 (down from $0.50)
Cache writes: $5 (down from $6.25)
Fast mode: $8 input and $40 output, with up to 2.5x speed in Claude Code and the Claude Platform

The cache read cut is the number that matters most for agents. Anthropic notes that cache reads make up most of the cost in agentic and coding work. A long-running agent rereads its context on every turn. A 60% cut on that line item changes the math for anyone running agents all day. Anthropic also reports output generation more than 30% faster than Opus 5.

On benchmarks, all figures below are vendor-reported by Anthropic. Some competitor figures in Anthropic's table come from OpenAI's own reports or from Zapier's leaderboard, as Anthropic notes.

Terminal-Bench 4.0 (agentic coding): Opus 5.5 scores 66.4%. Fable 5.1 scores 55.8%, Opus 5 scores 52.3%, and GPT-6 Astra scores 57.9% as reported by OpenAI.
FrontierCode v1.1: Opus 5.5 scores 54.4%, against 53.3% for GPT-6 Astra.
CursorBench 4.0: Opus 5.5 scores 57.8%, against 41.7% for GPT-5.6 Sol.
GDPval-AA v2.1 (knowledge work across 44 occupations): Opus 5.5 scores 1846 Elo, against 1735 for Fable 5.1.
AutomationBench (business workflows, run by Zapier): Opus 5.5 scores 40.0%. GPT-6 Astra scores 41.4%.
Humanity's Last Exam with tools: Opus 5.5 scores 67.7%.
Terminal-Bench-Science 0.1: Opus 5.5 scores 58.7%. GPT-6 Astra scores 64.6%.
OSWorld 2.0 (computer use): Opus 5.5 scores 81.8% partial credit.

Two things stand out in that table. Anthropic published rows where GPT-6 Astra wins, on AutomationBench and Terminal-Bench-Science. Anthropic also wrote that benchmark margins at this capability level have become a less reliable guide to real-world differences. That is a fair warning for anyone choosing a model from a leaderboard.

The efficiency claims carry more weight than the scores. Anthropic says Opus 5.5 at default medium effort beats GPT-6 Astra's top FrontierCode score at about a fifth of the cost per task. In an internal test, Opus 5.5 and Fable 5.1 both translated the HAProxy load balancer from C into Rust. Both rewrites passed nearly all of HAProxy's regression tests. Opus 5.5 finished in 9.5 hours against 12 for Fable 5.1, at 51% lower cost. An early tester audited and fixed a 200,000-line codebase in under three hours. Opus 5 took over 20 hours on the same job and used 2.5x the tokens.

Customer reports point the same way. GitHub's Mario Rodriguez said Opus 5.5 used among the fewest tokens and steps his team measured across Copilot CLI and VS Code. Box reported that Opus 5.5 used a third of the tokens Opus 5 did, with answers 40% less verbose. Deloitte reported that at its lowest effort setting, Opus 5.5 caught 72% of known bugs in code review, against 56% for Opus 5 at high effort.

What changes for builders on Opus 5.5

This release carries API behavior changes that matter more than the benchmarks for production teams.

Thinking cannot be disabled. Opus 5.5 is no longer available with thinking switched off. Integrations that turned thinking off for latency need to test the new default.
Preserved thinking. Anthropic's anti-distillation safeguard stops API users from editing Claude's prior context to extract its reasoning. It applies to Fable 5.1 and Opus 5.5 for API accounts created on or after August 31, 2026. Anthropic published docs on how to test integrations.
Safeguard fallbacks. Opus 5.5 ships with safeguards similar to Fable 5.1 for cybersecurity, biology, and distillation. Most cybersecurity tasks reroute to Opus 4.8. Routine bug finding and fixing in your own code still runs on Opus 5.5. Vetted life sciences groups can apply to a verification program for full biology access. Anthropic says an expanded Cyber Verification Program with three access tiers arrives in the coming weeks.
Watermarking. Opus 5.5 includes text watermarking to comply with the EU AI Act.
Zero data retention remains available, as with prior Opus models.

The fallback behavior deserves attention. If a safeguard intervenes, a different model answers. Teams that audit agent behavior need to log which model served each response, not only which model they requested.

Anthropic also called Opus 5.5 its first release since CEO Dario Amodei argued that AI progress should be paced so safety keeps ahead of capability. External evaluators including METR tested it before release. Anthropic reports Opus 5.5 scored better than any recent Claude model on nearly every measure of its automated behavioral audit, across nearly 2,000 scenarios. In a new containment test, it tried to cross boundaries about 85% less often than Opus 5. Anthropic also disclosed a limit: the model often suspects it is being evaluated, which makes pre-release testing harder to trust.

Subscription users get something too. Anthropic raised five-hour usage limits on Pro, Max, Team, and seat-based Enterprise plans. It also added a rate limit reset that users can save and use when they choose.

Pricing side by side

Token prices only tell part of the story, so it helps to run one example. Take an agent task that reads 2 million cached input tokens, 200,000 fresh input tokens, and writes 50,000 output tokens. That shape is common for a coding agent that rereads a repository context on each turn.

On Opus 5, that task costs $1.00 for cache reads, $1.00 for fresh input, and $1.25 for output. The total is $3.25. On Opus 5.5, the same token counts cost $0.40, $0.80, and $1.00, for a total of $2.20. That is a 32% cut from pricing alone, before any savings from the model using fewer tokens. Anthropic's 40% figure for typical workloads combines both effects.

For OpenAI, the base prices halve across the board. A task that sends 2.2 million input tokens and writes 50,000 output tokens to GPT-5.6 Sol cost $9.80 at list input and output prices. On GPT-6 Sol it costs $4.90 at list prices. Cached input discounts lower both numbers further, so check OpenAI's pricing page for the cached rate before you compare the two vendors head to head.

The practical point: the cheapest model per token is not always the cheapest model per task. Measure tokens per finished task on your own workload, then multiply.

GPT-6 Sol and GPT-6 Luna

OpenAI launched GPT-6 Sol and GPT-6 Luna the same day. They extend the GPT-6 generation that started with GPT-6 Astra earlier this month. Sol targets complex work like coding. Luna targets high-volume tasks with a clear goal, such as summarizing, extraction, and quick answers. Astra stays OpenAI's top model. The API IDs are gpt-6-sol and gpt-6-luna.

The pricing story is blunt. OpenAI cut API prices 50% against the GPT-5.6 promotional rates. The New Stack reports the per-million-token prices:

GPT-6 Sol: $2 input and $10 output, down from $4 and $20 for GPT-5.6 Sol
GPT-6 Luna: $0.10 input and $0.50 output, down from $0.20 and $1.20 for GPT-5.6 Luna

An OpenAI spokesperson told The New Stack these are the default prices, not a promotion. OpenAI credits better caching and inference for the cut.

OpenAI's headline claims are vendor-reported. On AutomationBench, a Zapier-built test of business workflows across 47 tools, OpenAI says GPT-6 Sol at xhigh effort beats Claude Opus 5 at max effort at 9% of Opus 5's cost per task. GPT-6 Luna at high effort improves on its predecessor by 5.4 points at 58% lower cost per task. On OpenAI's internal factuality evaluation, built from real conversations where users flagged mistakes, GPT-6 Sol makes about half as many errors as GPT-5.6 Sol. The New Stack notes that on DeepSWE v1.1, Sol at max effort scores 68.8% against 69.9% for Fable 5 at xhigh, at about 20% of the cost.

OpenAI's post includes a footnote worth reading. It says its Claude Fable 5.1 datapoint understates Fable's real cost because it omits Opus 5 fallbacks, which occurred on about 40% of AutomationBench tasks. So the safeguard fallback design that Anthropic built into its top models now shows up in a competitor's cost comparison. Expect safeguard behavior to become a routine line item in model evaluations.

GPT-6 Sol and Luna are available in ChatGPT Work and Codex for Plus, Pro, Business, Enterprise, and Edu users. Free and Go users get Luna in the desktop app. Neither model is in regular Chat yet. OpenAI DevDay 2026 runs September 29 in San Francisco.

Qwen3.8-Omni-Flash

Alibaba's Qwen team released Qwen3.8-Omni-Flash on September 18. It is a native omni-modal model that takes text, images, audio, and video as input and returns text. It supports a 1 million token context window. Alibaba reports, as vendor numbers, an average gain of more than 26% across 30 evaluations over Qwen3.5-Omni-Plus.

API pricing on Vercel's AI Gateway listing is $0.15 input and $0.47 output per million tokens. Alibaba did not release weights at launch. The model targets long-video analysis, meeting summaries, and multimodal tool use. Alibaba also shipped Qwen-MM-Plugins and a Qwen-Live Harness for long-running and real-time workflows.

For data teams, the interesting part is the input side. One call can take long audio and video plus a million tokens of context. That turns recorded meetings and support calls into queryable text at a price that makes batch processing practical.

The week in model context

September has been dense. Anthropic shipped Fable 5.1 and Mythos 5.1 on September 1. Google released Gemini 3.8 Flash on September 2. OpenAI released GPT-6 Astra earlier in the month. This week's releases are not new frontiers so much as the frontier getting cheaper. Opus 5.5 claims Fable-level work at Opus prices. GPT-6 Sol claims Astra-level factuality at half the old Sol price.

That pattern favors builders who measure cost per completed task. A model that costs more per token but finishes in fewer steps can come out cheaper. Both labs now publish charts with cost per task on one axis. Use those charts as a starting hypothesis, then run your own workload.

Tooling: JetBrains Air Bets on Many Agents

JetBrains Air

JetBrains introduced JetBrains Air on September 22. The company calls it a system of products for agentic software development. It spans JetBrains IDEs, team delivery workflows, and organizational governance. Unite.AI's coverage notes it brings together six months of public experiments. JetBrains introduced JetBrains Central in March as a control and execution system for agent-driven development, then added a CLI, shared context, cloud agents, automations, governance, and AI cost controls.

The core bet is multi-vendor. JetBrains writes that no single model, agent, or service will be right for every developer, team, or task. Air works with Claude Agent, Codex, Junie, Copilot, OpenCode, and any agent that speaks the Agent Client Protocol. JetBrains says the IDE still matters. It is bringing the base agentic experience into its IDEs so developers can understand, change, and verify what agents produce.

The Agent Client Protocol is the piece to watch. ACP standardizes the connection between an editor and an agent's full harness, including planning, tools, model routing, and observability. JetBrains lists agents that speak ACP, including Junie, Gemini CLI, GitHub Copilot, Codex, Cursor, Mistral Vibe, OpenCode, Kimi CLI, Qwen Code, Factory Droid, Cline, and Kiro CLI. Clients include JetBrains IDEs, Zed, and Neovim through a plugin. An ACP Registry lets developers discover and run compatible agents inside JetBrains IDEs. If ACP holds, an editor stops being tied to one agent vendor.

GitHub Copilot adds GPT-6 Sol and Luna

GitHub added GPT-6 Sol and GPT-6 Luna to Copilot on launch day. GitHub describes Sol as a balanced model for interactive and agentic coding. Luna is the lowest-cost option in the GPT-6 family for smaller, faster tasks. Sol is available on Copilot Pro+, Max, Business, and Enterprise. Luna is available on Pro, Pro+, Max, Business, and Enterprise. Both bill under usage-based billing, and rollout is gradual.

The billing note matters. Usage-based billing means model choice now shows up directly in the invoice. Teams that let every developer default to the most capable model will see it. Teams that route routine work to Luna-class models will see that too.

Claude Code and agent harnesses

On the Anthropic side, Opus 5.5 fast mode is available in Claude Code at $8 input and $40 output per million tokens, with up to 2.5x speed. Early tester reports focus on long unattended runs. Clio's Sean Heintz described handing Opus 5.5 a task across six repositories and letting it run overnight for over 18 hours. Stripe's Cristian Rivera described one Opus 5.5 session directing a dozen more sessions through a multi-day rebase of 40 stacked pull requests, all of which passed CI. AWS's Deepak Singh said Opus 5.5 is coming to Kiro soon. These are vendor-selected quotes, but they describe a clear shift: from one agent in one chat to one agent coordinating many.

Anthropic also points to three security layers for long-running coding agents: a classifier that screens every action before it runs, an open-source sandbox that security teams can audit, and code review that flags vulnerabilities before merge. Anthropic says Opus 5.5 ties Fable 5.1 for the lowest prompt injection success rate on a benchmark run by Gray Swan.

A routing playbook for this week's releases

With four new models in one week, teams need a simple way to decide what runs where. A few patterns follow from the vendors' own positioning.

Long, unattended coding jobs fit the top tier of each family. Anthropic positions Opus 5.5 for codebase-wide migrations and audits. OpenAI positions GPT-6 Sol for complex tasks and keeps Astra for the hardest work.
High-volume extraction and summarization fit the low tier. OpenAI built Luna for tasks with a clear goal. At $0.10 per million input tokens, it is cheap enough to run across large document sets.
Multimodal intake fits omni models. Qwen3.8-Omni-Flash takes long audio and video in one call and returns text that downstream models and SQL engines can use.
Everything else belongs behind a router. Gateways like Agent Router, covered below, let platform teams swap models without code changes. That keeps the next price cut a config change instead of a migration.

AI inside open source release processes

A small signal from the Apache mailing lists belongs here. This week, contributors disclosed using Claude, Codex, and GPT 5.6 to verify release candidates for Apache Iceberg, Apache Polaris, and Iceberg C++. One reviewer prompted Claude to check every license in the Iceberg 1.12.0 candidate. It found a bundled cache library with no LICENSE entry, and he voted -1 on that basis. Another used Codex to catch license issues across repeated failed candidates of a Polaris tool.

Apache Iceberg also merged work on AGENTS.md rules for testing, comments, and AI disclosure. Apache Arrow is debating a needs-author-engagement label for pull requests where the author does not engage with review. Open source communities are writing the norms for AI-assisted contribution right now, in public, one thread at a time.

Standards: MCP Gets a Certification and a Conference

The MCPA certification

The Agentic AI Foundation launched the Model Context Protocol Associate (MCPA) on September 14. It is the first official MCP certification and the first certification from AAIF. The exam covers five domains: MCP fundamentals, architecture and components, interactions and execution, security and governance, and use cases and ecosystem. It aligns with the MCP 2026-07-28 specification.

Angie Jones, AAIF's vice president of developer experience, framed it as a shared benchmark for employers. Developers need to understand how agent connections work and how to implement them responsibly, including permissions and trust boundaries. Attendees of the Amsterdam and San Jose conferences get a 20% discount.

AAIF's announcement also shared adoption numbers. Monthly downloads across MCP's Tier 1 SDKs approach half a billion. Both the TypeScript and Python SDKs have passed 1 billion total downloads. MCP tool calls from ChatGPT users reached 98 times their January level by August. Resend passed one million MCP calls in a single month.

AGNTCon + MCPCon Europe

AAIF held AGNTCon + MCPCon Europe at RAI Amsterdam on September 17 and 18. The conference merges AGNTCon, on agent architectures, with MCPCon, on the protocol itself. It also absorbs the European content from the former MCP Dev Summit. The keynotes centered on the MCP 2026-07-28 specification and its move to a stateless core.

That spec, released July 28, is still the story. It brought a stateless protocol core, multi round-trip requests, header-based routing, cacheable list results, authorization hardening, and a formal extensions framework. Before it, a client and server had to hold a session open. Now each request carries what it needs. Requests spread across servers behind a plain load balancer, with no shared session store. That makes a remote MCP server an ordinary HTTP workload that platform teams already know how to run.

The current MCP roadmap takes the next step. Maintainers want one transport model. Streamable HTTP becomes the single binding, carried over stdin and stdout for local servers. Today SDKs maintain two transport pipelines, and protocol metadata is duplicated across HTTP headers and message fields. Collapsing that removes a class of bugs and a lot of SDK code.

The North American edition runs October 22 and 23 in San Jose.

The AAIF stack takes shape

AAIF now hosts six projects: MCP, A2A, AGENTS.md, goose, agentgateway, and Agent Router. A2A joined in August and handles agent-to-agent discovery and delegation. MCP handles agent-to-tool connections. AGENTS.md gives agents a project's rules. Agent Router, formerly Envoy AI Gateway, joined on September 9 and gives developers one OpenAI-compatible endpoint plus MCP endpoints while platform teams manage providers, credentials, quotas, and failover.

Put those together with JetBrains' Agent Client Protocol and the shape of the open agent stack is visible:

Instructions: AGENTS.md tells an agent how a project works
Editor to agent: ACP connects an IDE to any agent harness
Agent to tools and data: MCP
Agent to agent: A2A
Traffic and policy: agentgateway and Agent Router

Each layer has an open spec and a neutral home or an open protocol. That is good news for anyone who wants to avoid building on one vendor's closed stack.

A2A deserves a closer look now that it sits beside MCP. The protocol lets agents built on different frameworks discover each other, authenticate, and hand off tasks. An agent publishes an Agent Card that describes what it can do and how to reach it. A calling agent sends a task over JSON-RPC 2.0 on HTTPS, and the task moves through a defined lifecycle. Version 1.0, released in March, added cryptographically signed Agent Cards so a caller can verify the card came from the domain it claims. AAIF describes A2A as supported by more than 150 organizations. The move into AAIF changed governance, not the spec. Same site, same SDKs, same technical steering committee.

Agent Router shows the gateway layer maturing. Its project post shows a one-command local start and a single environment variable change to point an existing OpenAI client at the gateway. MCP clients connect to the gateway's MCP endpoint, and the gateway accepts the same mcpServers file that Claude Desktop, Cursor, and VS Code already use. AAIF reports eleven public adopters including Bloomberg, Tencent Cloud, and Nutanix, and nine maintainer seats split across Bloomberg, Nutanix, AMD, Tetrate, and Netflix with no majority holder. Version 1.0 shipped in June with a 1.x compatibility commitment.

Gateways matter for the model news above. When two labs cut prices on the same day, the team with a gateway changes a routing rule. The team with hard-coded provider calls opens a ticket.

The data layer needs standards too

Agents need consistent business meaning, not just tool access. Apache Ossie, an incubating project for open semantic interchange, this week proposed a standard REST API for producing, consuming, and orchestrating semantic models. The proposal names data catalogs such as Apache Polaris as implementers and query engines as clients. Qlik said it has started building an Ossie-native REST API and suggested a dedicated media type, application/vnd.ossie+yaml.

This is the missing piece between MCP and the data. MCP lets an agent call a tool. A shared semantic model tells the agent what "revenue" or "active customer" means, no matter which engine answers the query. An open REST API for those models means catalogs and engines serve the same definitions to every agent.

Infrastructure: Racks, Roadmaps, and Cheaper Serving

AMD crosses $1 trillion on rack-scale systems

AMD crossed $1 trillion in market value on September 21. The Motley Fool notes AMD joins Nvidia, Broadcom, and Micron in that club. Shares rose 23% in the week and are up 187% for the year.

The hardware story behind the stock is Helios, AMD's rack-scale system. Each Helios rack runs 72 Instinct accelerators and 18 EPYC CPUs. The Tech Portal reports the platform combines MI400 and MI450 series GPUs, EPYC Venice CPUs, and Pensando networking. It reports that AMD's data center segment hit $6.72 billion in Q2 revenue, up 107% year over year. Anthropic has committed to up to 2 gigawatts of MI450 GPUs in Helios racks. Meta, OpenAI, and Microsoft Azure are also Helios customers.

Market coverage tied part of Monday's rally to Meta's Muse personal AI agent reaching the top of the U.S. App Store. The argument is that agents that run constantly need CPUs as well as GPUs. Tool calls, retrieval, orchestration, and code execution all run on general-purpose cores. An agent that calls ten tools per task spends a lot of time outside the GPU. That favors vendors who sell both halves of the rack.

For infrastructure buyers, the lesson is about the unit of purchase. The market now prices accelerators, CPUs, networking, and software as one system. Nvidia has sold that way for years with its rack-scale platforms. AMD now does too.

Huawei pulls its Ascend roadmap forward

At Huawei Connect 2026 in Shanghai, Huawei accelerated its AI chip roadmap, according to TrendForce. Rotating Chairman David Wang said the Ascend 960DT is running ahead of schedule and is now expected in the first quarter of 2027, three quarters earlier than planned. The Ascend 960PR follows in the third quarter of 2027. Huawei committed to one generation per year, with the Ascend 970 in 2028 and the Ascend 980 in 2029.

The performance claims are Huawei's own and lack independent validation so far. The cadence is the news. China's domestic accelerator market is building on its own roadmap, with its own chips and its own number formats. Model builders who serve both markets will tune for more than one hardware target.

Serving got cheaper, and the labs said why

Both frontier launches this week included an infrastructure claim. OpenAI said improvements in caching and inference let it serve GPT-6 Sol and Luna at lower cost. Anthropic said Opus 5.5 requires less compute to serve than Opus 5, and that its pricing reflects that. Anthropic cut cache read prices 60%, the largest single cut in its table.

Caching is the quiet infrastructure story of 2026. Agents resend long prompts on every turn: system instructions, tool definitions, and conversation history. Serving that repeated prefix from cache instead of recomputing it saves memory bandwidth and compute. When a lab cuts the cache read price, it is passing along a real serving saving. It also rewards agent designs that keep a stable prompt prefix.

File formats for AI data

Storage formats are infrastructure too, and Apache Parquet had a big week. The Parquet project published its ALP blog post on September 22, after the 2.14 format release added ALP and a FILE type. ALP is an adaptive lossless encoding for floating-point numbers, first published at SIGMOD 2024 by researchers at CWI. Arrow Rust 60.0.0 became the first Parquet implementation to ship ALP support.

Floating-point columns are everywhere in AI workloads: sensor readings, prices, model scores, and embeddings. Better float compression means smaller files and faster scans for those columns.

The Parquet community also opened review on a VECTOR logical type for embeddings and a Modular Footer that lets readers load only the metadata they need. The vector debate is live. One side wants a narrow type built for embeddings, with rules like no NaN values. The other wants a general fixed-size list type that serves tensors too. Whichever side wins, embeddings are on track to become a first-class citizen in the most widely used analytics file format.

The Data Layer: Where Agents Meet the Lakehouse

Cheaper models and stateless MCP make one problem more urgent: agents need safe, consistent access to real data. Three threads from the Apache mailing lists this week show how the open lakehouse is getting ready.

Credentials that fail loudly. Apache Polaris decided that when a client asks for delegated storage access and the catalog cannot provide it, the request fails with a clear error instead of returning a table with no credentials. Agents are the clients that benefit most. An agent that silently receives a table with no credentials often retries, guesses, or falls back to broader permissions. A clear error tells the agent, and the human reviewing its logs, exactly what went wrong. The Polaris team also asked the Apache Iceberg community to clarify the REST spec so every catalog behaves the same way.

Lists that never lie. Polaris also decided that when a catalog caps list sizes, an unpaged request that overflows the cap fails instead of returning a partial list. An agent that lists tables in a namespace and gets half of them has no way to know the list is short. It builds a wrong answer with full confidence. Failing closed protects the agent from its own trust in the data.

Files next to rows. Apache Parquet 2.14 added a FILE type, Apache Arrow is defining a matching extension type, and Apache Iceberg is working out how a governed REST catalog delegates access to those files. Pair that with an omni model like Qwen3.8-Omni-Flash and the pattern is clear. Store references to audio, video, and documents next to structured rows, let the catalog govern who reads them, and let a multimodal model turn them into text that SQL engines can query.

The open agent stack and the open lakehouse stack are converging on the same principles: neutral governance, published specs, and clients that can trust what servers tell them.

Practitioner Takeaways

Re-price your agent workloads. Opus 5.5 cut cache reads 60% and GPT-6 Sol cut token prices 50%. Rerun your cost models on real tasks, since both labs claim fewer tokens per task on top of lower prices.
Log the serving model. Opus 5.5 and Fable 5.1 fall back to other models when safeguards trigger. Record which model answered each call, not only which you requested.
Test the thinking default. Opus 5.5 does not allow thinking to be switched off. Check latency-sensitive paths before you upgrade.
Plan for multi-vendor agents. JetBrains Air, ACP, and usage-based Copilot billing all assume you run more than one agent and more than one model. Design routing and cost controls with that in mind.
Learn stateless MCP. The 2026-07-28 spec and the new MCPA certification both point to MCP servers as ordinary HTTP services. Build them that way.
Watch the data layer. Agents need governed tables and shared semantic definitions as much as they need tools. Open formats like Apache Iceberg and Apache Parquet, open catalogs like Apache Polaris, and emerging semantic standards like Apache Ossie give agents consistent data without lock-in.

What to Watch Next Week

OpenAI DevDay on September 29 will show whether GPT-6 Sol and Luna reach regular ChatGPT and what developer tools come with them. Anthropic says Sonnet 5.5 and Haiku 5.5 arrive in the coming weeks. On the standards side, watch for post-conference MCP proposals from Amsterdam and early sessions for the San Jose AGNTCon in October. On hardware, watch for Helios deployment updates as AMD heads toward its early November earnings report.

Keep Learning

If you want to go deeper on agentic AI, MCP, and the open data stack that agents run on, my books cover it end to end. Find the full catalog at books.alexmerced.com.

Apache Data Lakehouse Weekly: September 15 to 23, 2026

Alex Merced — Wed, 23 Sep 2026 21:14:00 +0000

Release managers ran the show this week, and license files kept tripping them. Iceberg 1.12.0 went to a second release candidate after reviewers found bundled web assets without matching license entries. Polaris 1.8.0 failed two votes over a NOTICE gap and a wrong file header. The Polaris catalog migrator reached its fourth candidate for the same reason. Behind that release grind sat the design work that shapes next year. Parquet opened review on a Modular Footer and a vector type. Iceberg moved its materialized view spec toward a vote. DataFusion took ownership of the Iceberg Rust integration. Apache Ossie pushed toward its first source release and proposed a REST API that catalogs like Polaris can implement.

The thread that ties the week together is the contract between client and server. Polaris, Iceberg, and Parquet each spent real time on one question: what should a server or reader do when it cannot deliver what the client asked for? Every project landed on the same answer. Fail loudly, and write the behavior down.

Apache Iceberg

1.12.0 heads into RC1 after a license cleanup

Neelesh Salian opened the 1.12.0 release discussion with the release branch in good shape and a key fix already merged. Alex Reid asked to include a Kafka Connect fix that stops the connector from committing already committed files during certain rebalances. That PR had already merged, so it ships in 1.12. Reid also asked about the REST client side of "referenced-by," a spec field added last year. Salian deferred that work to the 1.13.0 milestone because it does not block the release.

The RC0 vote drew one of the most thorough verification rounds the project has seen in a while. Yuya Ebihara confirmed Trino's CI passed against the staged artifacts. His tests cover HMS, Glue, JDBC on PostgreSQL, and REST catalogs including Polaris, Unity, S3 Tables, and Tabular, plus Nessie and Snowflake. He also flagged a caveat he raised months ago. Version 1.12.0 removes the old helper logic for reading and writing partition statistics. The new logic depends on the Hadoop-based Parquet client, and some downstream projects forbid that dependency.

Manu Zhang reported 10,991 tests passing across the api, core, and parquet suites with zero failures. Anoop Johnson rebuilt the source tarball from the tag and confirmed it matched, apart from two generated version files. Xin Huang compared all 7,083 common files between the tarball and the commit and found them identical.

Two reviewers found the problem that sank RC0. Xuanwo noticed a Bootstrap stylesheet in the docs site assets. It kept its copyright line, but the package lacked the MIT permission notice and the root LICENSE did not account for it. Gianluca Graziadei found a Lottie JavaScript file in the same folder with the same gap. Neither called it a blocker, since both files have shipped since 1.9.0. Salian filed an issue, the fix merged, and he cut a new candidate.

A geospatial behavior change stays in

Steven Wu used the vote thread to flag a behavior change most users will never notice until a string comparison breaks. A PR changed GeometryType and GeographyType so toString() always writes resolved defaults. A plain geometry now prints as geometry(OGC:CRS84), and geography prints as geography(OGC:CRS84, spherical).

Xin Huang, who wrote the original change, explained the goal. The PR persists resolved CRS and algorithm values in metadata even when they match the defaults. The toString() change was a side effect he had not recognized as a behavior change. He had a fallback PR ready to revert it.

Szehon Ho argued for keeping the new output. Two equal geography types printed differently depending on whether a user set the default explicitly, which looked like a bug. The new form also matches the canonical serialization in Appendix C of the spec. Huaxin Gao added a practical reason from her UDF work. Function definition IDs are built from parameter type strings, so two equal types that print differently look like two different overloads. Old metadata still loads, because geometry and geometry(OGC:CRS84) parse to the same type. The community agreed to keep the change and list it under "Behavior Change" in the release notes. The fallback PR closed.

RC1 and a Caffeine surprise

The RC1 vote opened on September 22. Renjie Liu cast a binding +1 after checksum, signature, license, and test runs. Xuanwo reported 10,470 passing tests across the common, API, core, and data modules. Szehon Ho reported 8,400 core tests passing with zero failures and a Spark test on Spark 4.2 built from verified source.

Graziadei raised a pre-existing bug. Reading variant columns throws NoSuchMethodError when commons-lang3 older than 3.13 sits on the classpath, because iceberg-parquet uses a method without declaring the dependency. The bug exists in 1.11.0 too, so Salian plans to patch it on the release branches instead of blocking. Szehon Ho added another pre-existing fix to the wish list if a new candidate appears.

Then Kurtis Wright cast a non-binding -1. He found that iceberg-gcp-bundle-1.12.0.jar bundles the Caffeine cache library with no entry in its LICENSE file. He disclosed that he found it by prompting Claude to validate every license in the candidate. That disclosure matters. It shows AI-assisted review catching the class of issue that has failed releases across three projects this month. Watch this thread for whether RC1 survives or a third candidate follows.

Iceberg C++ 0.4.0 lands a big v3 update

Manu Zhang proposed Iceberg C++ 0.4.0 after 174 commits since 0.3.0 shipped in June. The list is long. It covers Iceberg v3 types including timestamp_ns, timestamptz_ns, unknown, geometry, and geography, plus column defaults, binary deletion vectors, and row lineage. It adds table operations such as MergeAppend, DeleteFiles, RowDelta, OverwriteFiles, RewriteFiles, and ReplacePartitions. It brings parallel manifest reads and writes, lazy scan planning, an LRU cache, and commit and scan metrics. On the catalog side it adds REST SigV4, OAuth2 token exchange, vended credentials, an early Hive Metastore client, and ResolvingFileIO.

Gang Wu backed the release and noted that variant support waits on an Arrow C++ PR, so it moves to 0.5.0. Junwang Zhao asked to hold for late October to include CherryPickOperation. Zhang pushed back with a clear release philosophy. The project will not reach "feature complete" soon, so smaller and faster releases get users feedback earlier. Zhao agreed after updating the v3 roadmap issue, which showed most planned tasks done.

The 0.4.0 RC0 vote moved fast. Gang Wu and Renjie Liu cast binding +1 votes. Liu built on Ubuntu 26.04 with GCC 15.2 and reported 4,152 passing tests across 19 release test executables. Raúl Cumplido verified on Debian 14. Xin Huang verified on macOS arm64 and disclosed he used GPT 5.6 to help.

Fittingly, the week also brought a PMC seat for Gang Wu. Fokko Driesprong announced it and credited Wu as instrumental in the inception and growth of Iceberg C++. More than 35 community members replied with congratulations.

Iceberg Rust 0.11.0 fails RC2 over encryption

The Rust 0.11.0 RC2 vote failed on a real bug. Alexander Bailey tested the candidate and found he had not implemented the tamper-proofing checks for the new encryption work. Files written by Rust were unreadable in Java. He posted a fix and voted -1. Danny Jones agreed to fail the vote because encryption is one of the headline features of the release, and asked for review on the fix before RC3.

A second Rust issue surfaced in a license thread on pyiceberg-core. Jones noticed that the Python wheels compile Rust dependencies into the extension but do not reproduce their copyright notices. Ryan Blue confirmed that distributing third-party code in compiled artifacts means LICENSE and NOTICE must reflect it. Kevin Liu explained the root cause. The iceberg-rust source release bundles nothing, but the wheels do. Shawn Chang pointed to the Apache Paimon Rust bindings, which generate a third-party license report per wheel, as a model. The group agreed to fix this before the next pyiceberg-core release.

The DataFusion integration moves out

Andrew Lamb reported that both vote threads passed unanimously. The DataFusion integration for Iceberg Rust now lives in a new apache/datafusion-iceberg repository owned by the DataFusion project. Kevin Liu thanked Matt for starting the conversation and said he hopes the move improves integration for both projects. The DataFusion section below covers the first week in the new home.

Materialized views approach a vote

Jan Kaul posted an update on the materialized view spec. Marius Grama and Walaa Eldin Moustafa built proof-of-concept implementations in Spark and Trino. Kaul says the POCs prove the spec is sound. Moustafa laid out the plan. The community sync on October 1 hosts a final discussion, then the spec moves to a vote.

One open question remains. Should materialized views get their own format version field? Steven Wu argued no. A materialized view combines the view and table specs, so a breaking change bumps one or both of those versions. A separate version inside the view spec looks odd.

Moustafa followed with a thread on Spark routing for materialized views. The implementation splits in two. Iceberg core owns the metadata, records source states, and runs basic freshness checks. The Spark side reads the storage table when fresh and evaluates the view query when stale. He chose DSv2 catalog-based routing and updated it for the new loadRelation API in Spark 4.2.

This matters for anyone building semantic layers or acceleration on Iceberg. A standard materialized view definition means one engine can create a precomputed result and another engine can safely use it. Today each engine invents its own approach.

Access delegation for the FILE type

Sung Yun's thread on access delegation for the FILE type connects two efforts. Prashant Singh explained that he and William opened a spec proposal three months ago for file-level access through pre-signed URLs. An Azure proof of concept showed the limits of remote signing across cloud providers.

Daniel Weeks drew a useful line between two problems. Pre-signed URLs returned by the plan and tasks endpoints arrive already signed, so file IO only detects and executes them. Remote pre-signing routes a storage-native path through the catalog for signing before a stream opens. Weeks wants remote pre-signing to become a viable alternative to remote signing, mostly to address Azure's limits.

Singh raised a hard operational problem. Scan planning and execution happen in different phases. A Spark driver plans, and executors read later. A URL can expire before an executor uses it. So how do clients refresh vended URLs? Yun published two documents to frame the use case: a FILE type use case analysis for governed tables, and a batch signing proposal. The FILE type connects to Parquet's 2.14 release and a new Arrow extension type, covered below.

Encryption keys get a rethink for V4

Gábor Kaszab opened a discussion on reusable encryption keys. Today the encryption-keys list in table metadata holds both data encryption keys for manifest lists and the key encryption keys that wrap them. Kaszab wants to extend encryption to statistics files and V4 root manifests, and he questions some assumptions in the current design.

Gidon Gershinsky agreed that in V4, data key metadata belongs in per-file structures, such as snapshots for manifest lists. That makes cleanup easier and retires the key-id field. Key encryption keys stay in the shared table list. Gershinsky warned that reusing a data key across files requires careful handling to avoid breaking AES-GCM, so the practical answer is a fresh random key per file. Xander Bailey wants that rule written into the spec. He also asked the group to keep one common key metadata structure and one wrap and unwrap model, so implementations share code across V3 and V4.

Smaller threads worth your time

Neelesh Salian announced the iceberg-verification repository. It holds shared conformance fixtures for all Iceberg implementations. Early work covers type fixtures with JSON Schema validation, golden reference tables, and read conformance. Andrei Tserakhau confirmed writer conformance comes later and opened an epic issue to track the full scope. With implementations in Java, Python, Rust, Go, and C++, shared fixtures are how the community keeps them honest.

Andrei Tserakhau also split a decision out of the collation work into its own thread on collation versions. Collated columns store collation-aware min and max bounds so they stay prunable. ICU ordering changes across versions, so each bound carries the version it was computed under. A reader uses a bound only when it can reproduce that version. Otherwise it scans the file. The open question is whether one file carries bounds for several collation versions. The answer fixes the on-disk metadata layout, so Tserakhau wants it locked.

Alexander Bailey continued the gc.enabled semantics thread. He asked Dan Weeks whether catalog enforcement means vending credentials without delete permissions. Rejecting a snapshot update alone does not stop a client that holds storage delete rights. His narrower PR allows snapshot expiration with gc.enabled=false only when cleanup is set to NONE.

Kurtis Wright backed a proposal to standardize the User-Agent header for REST clients and asked where vendor information belongs in the string. A user named dzeri96 raised a Spark defaultDatabase issue about a hard-coded default database name in the SessionCatalog.

Péter Váry shared notes from the index support sync. Puffin reduces file-open overhead. Bloom filter scaling problems look like rate limiting. The group decided to prioritize scalar indexes before other index types. Váry also posted three AGENTS.md PRs that cover testing requirements, comment and javadoc rules, and AI disclosure for agent-written contributions.

Ryan Blue shared the September board report with a new format. It summarizes shipped releases and spec changes instead of sync highlights, so it reports what users can use today. Kevin Liu suggested adding the new blog site and the LICENSE and NOTICE revamp. Neelesh Salian suggested Gang Wu's PMC seat and the verification repository.

On the community calendar, Kevin Liu announced the first virtual meetup on GSoC projects and commutative compaction. In-person meetups are set for the Bay Area on September 23, Seattle on October 21, Copenhagen on November 5, and Berlin on November 18. And a high school student in India asked for Iceberg swag because he tells his classmates about the table format. He sent the same note to Arrow. That is the kind of reach no marketing budget buys.

Apache Polaris

1.8.0 fails twice on license details

Jean-Baptiste Onofré put Polaris 1.8.0 RC0 up for a vote on September 17. Dmitri Bourlatchkov voted -1 the same day. Iceberg jars carry a NOTICE that references Kite, and commons-math3 references Orekit. The Polaris bundle NOTICE did not propagate either one. Onofré cancelled RC0 and suspected the gap appeared during an earlier Iceberg version bump.

RC1 passed the automated checks, the license report, and smoke tests on the binary distribution, source distribution, and Helm chart. Alex Dutra still voted -1. A test file added in a recent PR carried a Dremio license header, but Dremio did not contribute that file, so it needs the default ASF header. Onofré cancelled RC1 and promised another full pass over LICENSE and NOTICE before RC2.

The Polaris Iceberg Catalog Migrator 1.1.0 followed the same path. Robert Stupp gave RC2 a binding +1 and noted the rebuilt CLI jar was byte-identical to the staged one. Bourlatchkov then found bundled files under the Mozilla Public License that the bundle LICENSE did not mention. Ajantha Bhat noted the same gaps existed in 1.0.0. He also ran Codex against the findings from each failed candidate to catch similar problems. RC3 opened on September 23.

Pagination: one flag flips, one fails closed

Two pagination threads resolved in a way that respects the Iceberg REST catalog spec. Yufei Gu proposed enabling list pagination by default. Today LIST_PAGINATION_ENABLED defaults to false, so clients cannot use pagination without server setup. Onofré pointed out that the max page size defaults to unlimited, so flipping the flag breaks no client that omits page parameters. Ayush Saxena, Yong Zheng, Nándor Kollár, and Bourlatchkov all agreed. Zheng opened the PR. The flag stays as an escape hatch for a couple of releases.

The harder thread, on separating page-size limits from forced pagination, started with a spec problem Gu caught. When the max page size is set, the server returns a partial list plus a continuation token to a client that never asked for pagination. That violates the REST spec. Saxena explained the original intent. Large catalogs face memory pressure and denial-of-service risk from unbounded list calls, and a cap that exempts unpaged requests bounds nothing.

Gu asked for evidence and posted payload math. A namespace with 100,000 tables of 100-character names produces about 13 MiB of uncompressed JSON. That is not an obvious crisis. Prithvi S named the real danger: a 200 response with a continuation token on a request that never asked for paging is a silently incomplete list. He cited PyIceberg 0.10 and 0.11, which make one unparameterized list call and return it.

Bourlatchkov proposed the fix everyone accepted. When a maximum is configured, paginated requests get capped. Unpaged requests that fit get the full list. Unpaged requests that overflow the cap fail with an error. Vignesh A added two details. Probe at max plus one so the server does not load everything into memory before throwing, and make the error message clear. Saxena opened the PR. Gu signed off on September 21.

Zheng also added pagination to the generic table API and the policy API, and proposed table and view management in the CLI. Bourlatchkov closed out the page size maximum discussion with a merge plan.

When access delegation cannot be satisfied

Youngrae Kim brought a spec alignment question to the list. When a client sends X-Iceberg-Access-Delegation and Polaris cannot satisfy it, Polaris fails with HTTP 400. Examples include vended credentials against an S3-compatible store without STS, or remote signing on a server that does not implement it. The Iceberg REST spec reads as if the server can instead return the table without delegated access.

Saxena corrected one premise. The spec does include a signal: an empty storage-credentials field on a 200 response, which clients must check. His objection is that "must check" is not "does check." Bourlatchkov made the architectural case. Whether a client uses local credentials or vended ones is a deployment decision, not a runtime choice. A client that already holds credentials has no reason to also request delegation.

Alex Dutra added context. The "must first check" language exists for backward compatibility, which is why Polaris vends credentials in both places. Iceberg 1.12 adds a remote-signing-config field that follows the same dual-location pattern. That is one more reason to write down the "delegation not provided" semantics now.

The group chose fail-fast and documentation. Kim then took the question upstream and opened an Iceberg thread asking whether "any or none" in the spec permits rejecting the request. This is exactly how cross-project clarity gets built: a catalog finds an ambiguity, picks a safe behavior, and asks the spec owners to settle it for everyone.

Location overlap checks get honest names

Bourlatchkov opened a thread on hasOverlappingSiblings() behavior. The persistence implementations search the whole catalog, but the caller uses them only when the OPTIMIZED_SIBLING_CHECK flag is on. With the flag off, Polaris checks only immediate siblings. He called it a logical inconsistency, since a speed switch should not change what a validation covers.

Eric Maynard explained the history. The original sibling check was limited by the metastore and exposed catalogs to security issues in some configurations. The flagged check was really a corrected, thorough check, and it deserved a name like "v2." Onofré and Prithvi S narrowed the scenario. The gap bites only when ALLOW_UNSTRUCTURED_TABLE_LOCATION lets a table escape its namespace's location tree. The group agreed to keep the catalog-wide check, rename or better describe the flag, and document when catalog-wide checks are essential. Bourlatchkov floated one more idea: make the flag settable per catalog and enable it by default for new catalogs.

Soft delete, audit trails, and consistent writes

Prithvi S proposed opt-in table soft-delete with a hold period and later undrop. Today a Polaris DROP is final from the catalog's view. Operators who need recoverable drops for legal holds or Nessie migrations build workarounds that REST clients bypass. The feature is off by default. Onofré supported reserving the table name during the hold, with DROP PURGE as the explicit way to get the name back. Bourlatchkov went further. The name-based location cannot be reused until data is purged, or a new table can read an old one's files. Prithvi S explained that expiration runs lazily on namespace access in phase one, not as a scheduled sweep.

Prithvi S also opened an issue for a supported audit trail. Polaris has built an event pipeline since 1.0, with JDBC and CloudWatch in 1.2, multiple listeners in 1.5, an async executor in 1.6, and Kafka and OpenTelemetry listeners in 1.7. That is still not an operator-facing audit story. Gu suggested documenting current delivery semantics first and expanding authentication and authorization events. Adnan Hemani argued the events table is one listener among several and operators choose where events land.

Bourlatchkov posted a recap of the sync on consistent multi-object changes. Entity version numbers work for RBAC grants but not for cases like location overlaps between concurrent table creations. He found a PR that shows a failed request leaving persisted side effects. EJ Wang asked what backend assumptions the design targets. Bourlatchkov said the real goal is an SPI with clear contracts that both JDBC and NoSQL persistence can implement. Serializable JDBC transactions solve the problems raised, but they rule out in-memory caching for reads inside a transaction.

Arun Suri described a related fix for ambiguous JDBC commits. When a commit outcome is unclear, his PR reloads the entity and checks whether the metadata location already matches. If it does, the commit counts as a success. Only an unconfirmable outcome raises CommitStateUnknownException, which prevents unsafe metadata cleanup.

Cloudflare R2, tags, sharing, and federation

Austen Tomek pushed forward Cloudflare R2 support with scoped credential vending. His first PR adds a credentialVendingMechanism setting, with STS as the default. Gu asked whether the existing endpoint field is enough to detect R2. Tomek said no. Self-hosted S3-compatible stores such as MinIO use arbitrary hostnames, a failed match silently falls back to STS, and Polaris has two endpoint fields that can disagree.

EJ Wang updated the Polaris Tag spec. Bourlatchkov prefers a separate /api/tags/v1 root, since the existing catalog API was not designed for extension. Wang asked to proceed under the current root for now and revisit before release.

Bourlatchkov also asked the Open Sharing API thread to build share access on the existing authentication and authorization framework instead of a new mechanism, with external identity provider support. David Chaava asked how Polaris should handle vendor-specific validation for federated REST catalogs after reviewers pushed BigLake-specific checks out of the shared admin path. Prithvi S detailed how user-defined principal properties forward to OPA and Ranger through one shared helper. And Onofré argued for dedicated semantic model privileges that mirror existing entities. That last point connects directly to Ossie, below.

The team also settled schema upgrade docs. A single DROP TABLE IF EXISTS beats listing dependent indexes, because it works on both v3 and v4 schemas. Gu wants to drop the upgrade SQL section entirely. Dutra wants to keep it and add migration tests per backend.

Apache Arrow

Rust 60.0.0 ships the first ALP Parquet support

Andrew Lamb announced Arrow Rust 60.0.0 with five +1 votes, three binding. He called the release "epic." Arrow Rust is the first Parquet implementation to ship the new ALP encoding for floating-point data. Lamb credited Kosta and Devan for leading that work. The release also includes new Parquet PageIndex structures from Ed and many performance improvements from Richard Baah and others. The object_store 0.14.2 release also passed with three binding votes and went to crates.io.

Dropping macOS Intel

Raúl Cumplido proposed dropping macOS Intel support. Homebrew stopped building Intel bottles, and Apple's latest macOS release dropped Intel. The project cannot test some setups in CI anymore.

The community brought data. Antoine Pitrou cited the Steam survey, which shows about 11% of Mac users still on Intel CPUs. Joris Van den Bossche checked PyPI and found x86_64 at 6.6% of PyArrow macOS downloads over 30 days: 57,121 against 807,414 for arm64. Sutou Kouhei ran a ClickHouse query over PyPI data. The Intel Mac share of all PyArrow downloads is tiny.

Ruoxi Sun suggested best-effort source builds with no CI guarantee. Jacob Wujciak argued that dropping CI means dropping Intel binaries too, since cross-compiling adds the maintenance cost the change aims to remove. Wes McKinney offered his 2019 Intel Mac Pro for testing commits to main. Kurtis Wright asked whether Arrow has a policy for how many releases a platform gets before deprecation. The thread has broad support. Watch for a formal vote.

Guidelines for AI-generated contributions

Pedro Matias revived the AI contributions thread with a focus on reviewers. Arrow has AI guidelines for contributors but none for reviewers facing low-quality PRs with weak author engagement. He wants an agreed way to handle authors who ignore the rules.

Rok Mihevc proposed a GitHub label, needs-author-engagement. It flags PRs where the author needs to respond substantially before review continues. It focuses on the author's observable understanding and engagement, not on whether they used AI. Nic Crane and Dmitry Chirkov supported it. Onofré pointed to the ASF's generative tooling guidance, which encourages disclosure. He also made a governance point. At the ASF, people contribute as individuals, so quality discussions happen in public even between colleagues at the same company.

Format work: FILE, JSON schemas, and ranges

Mandukhai Alimaa opened a format PR for a canonical FILE extension type named arrow.parquet.file, following the precedent of arrow.parquet.variant. Kent Wu updated his proposal for a JSON representation of Arrow schemas. Each type now has exactly one allowed form, either a simple string or an object. Metadata uses the object form because many implementations expose metadata as a map, which cannot hold duplicate keys.

A contributor pinged the thread on an arrow.range canonical extension type for bounded ranges. The proposal uses one range type with a closed parameter instead of separate types per closedness, since Arrow's kernel matching already sees extension type parameters. Ian Cook posted the biweekly community meeting for September 23.

Apache Parquet

The Modular Footer goes up for review

Jiayi Wang introduced the Parquet Modular Footer for broad review after months of biweekly working group meetings. The proposal covers motivation, format design, compatibility, and benchmarks, with a parquet-format PR and tracking issue. Wang expects a lot of feedback because the proposal introduces several large conceptual changes. Micah Kornfield had asked Wang to move the discussion to the list for visibility and to tie up the old footer tracking issue.

The footer matters for anyone running wide tables in a lakehouse. Today a reader parses the entire Thrift footer before reading any column, and that cost grows with column count. A modular footer lets readers load only what they need.

How readers handle unknown versions

The footer work drives a related question: how should readers behave with unsupported format versions? Xiening Dai, Fokko Driesprong, and Steve Loughran all voted for option one: fail on an unknown version. Driesprong argued that breaking changes like the new footer will break old readers anyway. The spec is the only thing the project controls, and quiet partial reads erode trust. He suggested bundling smaller speedups such as path-in-schema into the new footer version instead of cutting corners.

Loughran made the counterweight point. Because a reader must fail fast on an unknown version, moving to a new version needs a very good reason. The work to carry FlatBuffers alongside Thrift exists to deliver speed while keeping compatibility. He sketched a future "2.0" library that reads and writes both classic and new layouts but writes classic files by default.

ALP is live, and FastLanes gets a hard look

With the 2.14 format release done, Andrew Lamb published the ALP blog on the Parquet site on September 22. ALP, short for Adaptive Lossless floating-Point encoding, first appeared in a SIGMOD 2024 paper from CWI. Lamb also opened website PRs for the 2.14 changes, including ALP and FILE.

Prateek Gaur posted a performance study of FastLanes to separate its ideas from one another. For plain PFOR, the layout gain depends heavily on hardware, because the Parquet C++ reader is already vectorized. Antoine Pitrou asked whether the same speedup applies to DELTA_BINARY_PACKED by breaking the sequential dependency with interleaved partial sums. Gaur had already done it. Three draft PRs against Parquet C++ implement a log-step SIMD scan and batched miniblock unpacking, for about a 2.15x speedup on narrow 32-bit pages on Graviton4.

Gaur reports that the FastLanes transposed layout is still roughly 2x faster on top of that. Alkis Evlogimenos argued a new layout doubles the writer's decision space and suggested shelving FastLanes until Parquet has more data-dependent encodings. Gaur disagreed and asked to decouple FastLanes from the PFOR encoding discussion. There, Evlogimenos asked for a PFOR-delta versus DBP comparison and whether DBP blocks can live between PFOR blocks. Lamb asked what DBP stands for, which is a good reminder to define acronyms on public lists. It means delta binary packed.

A vector type for embeddings

Rok Mihevc opened a PR for a LIST-based VECTOR logical type after a focused community call. Every non-null vector has exactly num_elements values. Elements must be non-null, finite, and numeric or boolean primitives, with no nesting. Readers that do not know VECTOR read it as a plain LIST. Physical layout changes, vector indexes, and quantization are out of scope.

Pitrou objected to the narrow rules. Parquet is a general-purpose format. Why forbid NaN values when NumPy tensor data contains them? He argued for a general fixed-size list type and leaving specialist types to domain software, the way Arrow stores its schema in Parquet metadata. Mihevc defended the scope. Vector stores reject non-finite and nested values, and statistics are sometimes not written or read. He then asked the key question: what is the gate for Parquet to define a logical type? He also asked whether removing the finiteness rule makes the proposal acceptable, and whether it is time to revisit the Parquet extension type discussion.

For lakehouse teams storing embeddings next to business data, this thread decides how vectors land in Parquet files for years.

Types and statistics

Thomas Kissinger returned to the decimal floating-point proposal. Kornfield asked for a formal design doc and named two open issues: how to represent normalization and how to store values. Kissinger argued that statistics describe what was written but do not define a writer's obligation, so a logical-type parameter should state whether a column preserves quantum or canonicalizes.

Adrian Garcia Badaracco proposed classifying distinct_count as inexact. The spec never says whether the field is exact or an estimate such as a HyperLogLog sketch. An exact definition raises its own questions about nulls, negative zero, and NaN bit patterns. Query planners use this field for join ordering, so a clear definition matters. Michael Chavinda asked to add DataHaskell's Parquet reader to the implementation support matrix. Julien Le Dem posted the Parquet sync.

Apache DataFusion

The Iceberg integration finds a new home

Andrew Lamb announced the vote result to accept the apache/datafusion-iceberg repository into the DataFusion project, with 14 +1 votes. He called it a first step toward better Iceberg integration. The cross-list discussion confirmed both projects approved.

The new repository got to work fast. Gabriel (gabotechs) merged a PR that ported iceberg-rust history into the new repo, then a PR that made the project compile with passing tests after review from Kevin Liu. Lamb fixed a misnamed .asf.yaml file that kept GitHub issues disabled. A contributor resubmitted a PR to make IcebergTableProvider::try_new public, since the only provider with write support needs a public constructor. The original PR was lost when the integration moved.

This move is a smart division of labor. Iceberg Rust focuses on the table format library. DataFusion owns the query engine glue, where its maintainers know the planner and execution internals best.

Sync notes and a new committer

Lamb started posting notes from the DataFusion sync call because about ten people attend regularly and the topics interest the whole community. Adam GS covered subquery decorrelation, with many issues filed and ideas for testing query planner passes in isolation. Raz shared a prototype for blocked aggregation and asked aggregate authors whether the new API works for them. Haresh proposed CI automation to identify and backport PRs. Kurtis Wright asked for a calendar invite. The Parquet, Arrow, and Iceberg communities publish iCal links, and DataFusion does not yet.

The PMC also welcomed Dewey Dunnington as a committer. Dunnington is well known across the Arrow ecosystem, so this strengthens the link between DataFusion and Arrow.

Apache Ossie (incubating)

Converters leave the main repo

Yufei Gu opened the week's biggest Ossie thread on the scope and maintenance of converters. Converters translate Ossie semantic models to and from vendor formats. Keeping them all in the main repo creates three problems. Converter dependencies complicate licensing. Each converter has its own structure and CI, which complicates binary releases. Each needs platform knowledge and an ongoing maintainer. Gu suggested keeping a small set as reference implementations.

Julian Hyde, drawing on his experience as a founder and mentor of past incubator projects, agreed with much of it. He warned that moving converters to vendor repos means contributors are not seen as ASF contributors, and that licensing limits apply wherever the code lives. His advice: focus on the first release, because every incubating project underestimates it.

Onofré shared the history. Ossie started as a specification, which alone did not justify an incubator project, so converters went into the main repo. The plan from day one was to split them out. He proposed freezing converters, then creating a repo per converter, such as ossie-dbt and ossie-databricks, and keeping only the spec and docs in the main repo.

Piotr Sowiński played devil's advocate with lessons from LinkML. There, all converters live in one repo and ship as one package, because they share parsing code and users get one install that works. Russell Spitzer favored separate repos and compared it to Parquet, with parquet-format separate from its implementations. Markus Weimer did not care about repo count but insisted every repo stay inside the ASF project, because ASF provenance makes code easier for downstream users to adopt.

Toward the first source release

Kyoung Min Kim found 22 files missing ASF license headers ahead of the 0.3.0 source release, across the CLI plugin code, the Sigma converter, and NVIDIA test fixtures. He opened a fix and asked for a header-only CI check. Chris Eubank and Onofré both backed it. Given how the other projects spent their week, this is good timing.

Semantics, compliance, and a REST API

Chris Eubank pulled several threads into one plan for semantics and compliance suites. His diagnosis is sharp. Given the same model shape, two vendors can return different answers to the same question, which weakens Ossie as a standard. The plan layers an expression language, two query interfaces, and compliance suites. Onofré supported the layered framing and asked for proof that Layer 3 SQL reduces to Layer 2 primitives, ideally through shared fixtures where the same model and question produce the same answer both ways. Eubank reported that the OSSIE_SQL_2026 dialect definition merged, and proposed a fixture format that pins a Layer 3 query, its Layer 2 form, and the base SQL answer to one value.

Onofré then proposed an Ossie REST API spec, an OpenAPI file for producing, consuming, and orchestrating Ossie models. He named catalogs such as Apache Polaris, Unity, Collibra, and DataGalaxy as implementers, with query engines as clients. Sebastien Gandon said Qlik has started building an Ossie-native REST API to import and export its semantic models, and proposed a dedicated media type, application/vnd.ossie+yaml.

Modeling details

Markus Cozowicz proposed making relationship cardinality explicit. Today "from" means the many side and "to" means the one side, which cannot describe one-to-one or many-to-many. Onofré pointed to evidence that clears the usual bar for promoting a vendor extension into the core spec. The Microsoft, Databricks, and OrionBelt converters each invented a private stash for cardinality. Cozowicz revised the proposal to reuse the ontology term multiplicity with PascalCase values, default to ManyToOne when absent, and skip OneToMany since it is the same relationship with endpoints swapped.

Shao Xie proposed decoupling ontologies, semantic models, and mappings into three document kinds joined by external references. Gu questioned whether three document types are worth the extra catalog complexity. Ankit Tandon shared Ontology working group notes. Victor Morgante reported that the Python ontology parser rejects reverse verbalizations of a relationship, which fact-based modeling allows.

On GitHub Discussions, Jochen Christ explained the split between Ossie and the Open Data Contract Standard. Ossie now has a core spec for BI semantic models and an ontology spec at the conceptual layer. Data contracts cover more than schema and semantics, including terms of use, ownership, quality, and SLAs. A user asked how Ossie handles unstructured data. Gu answered that the community focuses on structured data for now and asked for concrete use cases.

Cross-Project Themes

License hygiene is now the release bottleneck. Iceberg 1.12.0, Polaris 1.8.0, the Polaris catalog migrator, pyiceberg-core, and Ossie 0.3.0 all hit LICENSE, NOTICE, or header problems this week. The pattern is clear. As projects ship more binary bundles, wheels, and Helm charts, the list of third-party code inside each artifact grows faster than anyone tracks by hand. Ossie's push for a header check in CI and Paimon's per-wheel license report point to the fix: automate it. Notice also who found the problems. Kurtis Wright used Claude, Ajantha Bhat used Codex, and Xin Huang used GPT 5.6 to verify candidates, and all three disclosed it. AI review is now part of the ASF release process, and the disclosure norm is forming in real time.

Fail loudly, then write it down. Polaris chose fail-fast for unsatisfiable access delegation and for oversized unpaged list calls. Parquet chose fail-fast for unknown format versions. Iceberg keeps a visible behavior change for geospatial types and documents it instead of hiding it. The common thread is respect for the client. A silent partial answer is worse than an error, because the client cannot tell it is wrong. That principle is what makes open standards safe to build on. It is also why the Polaris team took its access delegation question back to the Iceberg list instead of patching around it.

Specs are getting conformance suites. Iceberg launched iceberg-verification with golden tables and read conformance. Ossie is designing compliance fixtures that pin one answer across query layers. Parquet's distinct_count thread asks the spec to say what a statistic guarantees. When five languages implement one table format, and many vendors implement one semantic model format, the spec text alone is not enough. Shared fixtures turn "compatible" from a claim into a test result.

The FILE type ties three projects together. Parquet 2.14 shipped FILE support. Arrow is defining arrow.parquet.file as a canonical extension. Iceberg is working out how a governed REST catalog delegates access to those files. This is the plumbing for storing references to images, PDFs, and other unstructured objects next to structured rows, with the catalog in charge of who reads them. It is the base layer for multimodal AI on lakehouse tables.

Semantics move into the catalog. Onofré wants dedicated semantic model privileges in Polaris, and he proposed an Ossie REST API that names Polaris as an implementer. Put those together and the direction is plain. The open catalog that governs Iceberg tables is lining up to govern the semantic models that describe them. For AI agents that need consistent business definitions, that pairing matters as much as the table format.

Looking Ahead

Watch the Iceberg 1.12.0 RC1 vote and whether the Caffeine license gap forces RC2. Iceberg C++ 0.4.0 needs one more binding vote to pass. Iceberg Rust 0.11.0 RC3 waits on the encryption tamper-proofing fix. Polaris 1.8.0 RC2 and the catalog migrator RC3 both need clean license passes.

On design, the October 1 Iceberg sync hosts the last materialized view discussion before a vote. Parquet's Modular Footer review opens in earnest, and the VECTOR type debate needs an answer on what earns a logical type. Arrow's macOS Intel thread is ready for a formal vote. Ossie heads toward its first source release with converters moving out of the main repo.

Keep Learning

If this newsletter helps you follow the open lakehouse, my books go deeper on Apache Iceberg, Apache Polaris, agentic analytics, and the rest of the stack. Find the full catalog at books.alexmerced.com.

Fast Classification Models, LLMs, and the Apache Iceberg Lakehouse

Alex Merced — Mon, 21 Sep 2026 16:16:46 +0000

Open the bill for any team that has put a large language model into a data pipeline and look at what the calls are doing. A big share of them are not writing anything. They are answering questions with a short, fixed set of answers. Is this support ticket about billing or about an outage? Does this review mention a safety problem? Is this row of free text a complaint, a question, or a compliment? The team sends a paragraph of context, a paragraph of instructions, and a request for JSON. The model spends seconds generating tokens, the pipeline parses the JSON, and sometimes the JSON comes back broken.

That pattern works. It is also slow and expensive for what it delivers, and it gets worse at lakehouse scale, where the "input" is not one ticket but forty million rows in an Apache Iceberg table.

In September 2026 a company called TypeSafe AI released Jev, a model built only for this kind of question. It does not chat. It returns a choice, a score, or a yes/no probability, with calibrated confidence attached. Within days, several open-source projects appeared that copy its interface on models you can run yourself.

This article covers what these classification models are, how they differ from LLMs, where the open alternatives stand, and how to use both kinds of model together inside an Iceberg lakehouse. A quick note on affiliation: I work at Dremio, which ships SQL AI functions for this kind of work. Dremio shows up here as one worked example. The patterns apply to any engine that reads Iceberg.

Why So Much LLM Spend Goes to Questions With Fixed Answers

Text classification is one of the oldest jobs in applied machine learning. Before 2023, most teams handled it in one of three ways.

The first was rules. Keyword lists, regular expressions, and lookup tables. Rules are fast and cheap, and anyone can read them. They also break the moment a customer writes "I am not asking for a refund." A rule that fires on the word "refund" gets that sentence wrong every time.

The second was a fine-tuned encoder model. BERT (Bidirectional Encoder Representations from Transformers) and its descendants read a whole passage at once and produce a label in a single forward pass. A fine-tuned BERT classifier runs in milliseconds on a modest GPU. The cost is up front. You need a few thousand labeled examples per task, a training pipeline, and a person who knows how to evaluate it. When the business adds a new category, you label more data and train again.

The third was zero-shot classification with NLI (natural language inference) models. You phrase each label as a hypothesis, such as "This text is about billing," and the model scores how strongly the input entails it. This needs no training data, but accuracy on messy, domain-specific text is uneven.

Then instruction-tuned LLMs arrived and changed the economics of the second and third options. You no longer needed labeled data or a training run. You wrote a prompt, listed the categories, and asked for JSON. Accuracy on common-sense judgments jumped. For a prototype, nothing else came close.

The trouble shows up in production. An autoregressive LLM produces output one token at a time. Every label, every brace, and every quote mark in the JSON is a separate step through the model. Reasoning models add a chain of thought before the answer, which multiplies the token count again. A classification that needs one bit of information ends up costing hundreds of output tokens.

There are three other costs that are easy to miss. The first is parsing. The model returns text, and your code has to turn it into a typed value. Structured output modes help, but the model still generates the text and a validator still checks it. The second is invented labels. Ask for one of five categories and a generative model sometimes returns a sixth that sounds reasonable. The third is the confidence problem. An LLM that says "billing" gives you no reliable number for how sure it is. Token log probabilities exist on some APIs, but they are not calibrated for this purpose, and many hosted APIs do not expose them at all.

In a single request/response app, these costs are annoying. In a lakehouse pipeline, they compound. If a nightly job classifies two million new rows with a frontier LLM, the latency sets how long the job runs, the per-token price sets the bill, and the lack of confidence scores means every label looks equally trustworthy in the downstream table. An analyst building a dashboard on those labels has no way to filter out the shaky ones.

That gap between "the LLM can do this" and "the LLM is the right tool for this" is where the new classification models live.

What Jev Is and How It Works

Jev comes from TypeSafe AI, a company founded by Diogo Almeida, a former OpenAI researcher and one of the authors of the InstructGPT paper. TypeSafe calls Jev a "System One model." The name borrows from Daniel Kahneman's split between fast, intuitive thinking (System 1) and slow, deliberate reasoning (System 2). Reasoning LLMs are System 2 tools. Jev is built for the quick judgments.

The interface has three parts.

State. The content you want judged. This can be a string, a JSON object, or an array of text values. A support ticket, a product review, a row from a table serialized as JSON, or a retrieved document all work as state.

Questions. A map of named questions. Each question has a type, instructions, and criteria. You choose the question names, and the answers come back under the same names. The model never sees the names themselves.

Answers. One typed answer per question, with probabilities.

There are three question types, and picking the right one is most of the design work.

A Choice picks one option from a set you define. The criteria are a map from option names to descriptions. TypeSafe's documentation allows up to 255 options per Choice. The answer contains the selected option, a probability for every option, and a confidence value. If you ask which team owns a ticket and list billing, returns, and shipping, the answer is always one of those three. It cannot be a fourth.

A Score rates something against an ordered rubric. The criteria are an ordered list of level descriptions, from 2 to 10 levels. Urgency, quality, relevance, and frustration are natural fits. The answer is the level index, a probability per level, and a confidence value.

A Noul is a yes/no question. The answer is the probability that the statement is true. You can optionally describe what "true" and "false" mean. The unusual name is TypeSafe's own term, and it shows up in their SDKs as Noul.

The model processes the state once and evaluates every question against it in parallel. This detail matters a great deal for cost. Asking ten questions about one ticket in a single request costs barely more wall-clock time than asking one. TypeSafe's docs push this pattern hard and call it speculative fan-out. You ask every question your code has any use for, including ones that only matter for certain branches, and your code ignores the answers it does not use.

TypeSafe has said little about the architecture. Public material mentions a new model design, a parallel sampler, and a training method called RLCD (reinforcement learning for calibrated decisions). Coverage from the launch describes it as non-autoregressive, meaning it does not generate output one token at a time. The practical result is latency in the range of roughly 70 to 500 milliseconds per call, versus seconds for a frontier model producing JSON.

The numbers that matter for pipeline design

At the time of writing, the current version is jev-1.13.0, reachable through the alias jev-latest. The published limits shape how you build around it:

Price. TypeSafe charges only for input tokens, at $0.042 per million. Output tokens are free.
Rate limits. 250,000 tokens per second and 1,200 requests per minute. TypeSafe warns these limits are changing as demand grows.
Context. 64,000 tokens per request in total, and 32,000 tokens for the state plus the longest single question.
Input. Text only. No images, audio, or video.

Access is through POST https://api.typesafe.ai/v1/systemone, with official Python and JavaScript SDKs. Early access runs through a waitlist. Gateway products such as LiteLLM and several others have already added pass-through support.

On accuracy, TypeSafe's own four-workflow benchmark puts Jev at 67.8 percent. That is roughly tied with GPT-5.6 Terra at 67.9 percent, and a few points behind GPT-5.6 Sol at 74.1 percent and Claude Opus 5 at 73.1 percent. Read that plainly. Jev is not smarter than frontier models. It lands near a mid-tier LLM on decision tasks while running at a small fraction of the cost and time. That tradeoff is the whole pitch.

Independent testers have published early numbers that line up with the pitch. One team ran seven classification rules across 1,000 emails in about 6 seconds for 9 cents once they parallelized requests. A comparable single-category run on a GPT-5.6-class model took roughly 5 minutes and cost 62 cents. Treat third-party benchmarks from launch week with care, but the direction is consistent across reports.

What Jev does not do

Jev does not generate text. It does not explain its answers. It does not do arithmetic well, and TypeSafe says so directly. It reads dates as text, so comparing two dates or checking whether one falls in a window is unreliable. It cannot be fine-tuned on your data. The same weights serve every account, and you shape behavior through the state, instructions, and criteria you send.

That last point is a real difference from the BERT approach. You do not own a model. You own a set of well-written questions. For many teams that is a better asset, because questions are cheap to change and easy to review in a pull request.

Calibration Is the Feature That Changes Pipeline Design

Speed and price get the headlines. For data engineers, the more important property is calibration.

A model is calibrated when its probabilities match reality. If a calibrated model says "billing" with 0.8 probability across a thousand tickets, about 800 of those tickets are about billing. A model that is overconfident says 0.95 and is right 70 percent of the time. A model that gives no probability at all leaves you guessing.

Jev returns two related numbers, and they mean different things.

Probabilities describe the distribution across options. For a Choice with three options, you get three numbers that sum to 1. For a Noul, you get one number, the chance the answer is yes.

Confidence summarizes how peaked that distribution is. A single spike on one option gives confidence near 1. Probability spread across several options gives low confidence. TypeSafe's docs show a ticket that mentions both a wrong size and a double charge. The department question returns "returns" at 0.61 and "billing" at 0.35, with confidence of 0.42. The model is telling you, correctly, that the ticket belongs to two teams.

This changes what you can do with the output. With an uncalibrated LLM label, your choices are to trust it or re-check everything. With a calibrated probability, you get a dial. Rows above 0.9 flow straight into the published table. Rows between 0.5 and 0.9 get labeled but flagged. Rows below 0.5 go to a more expensive model or a human. TypeSafe calls this confidence-gated routing, and it is the single most useful pattern in their docs.

The dial also changes analytics. A dashboard that counts "tickets about billing this week" usually treats each label as a fact. With probabilities stored in the table, you have two better options. You can count only labels above a threshold and report the coverage alongside the count. Or you can sum the probabilities themselves, which gives an expected count that accounts for uncertainty. If a thousand tickets each carry a 0.3 probability of churn language, the expected number of churn-signal tickets is 300, even though no single ticket crosses a 0.5 threshold. For trend analysis over large volumes, the expected count is often the more honest number.

Calibration is not magic, and it is not permanent. It holds on the kind of data the model was trained and tested on. On text far from that data, such as a domain full of internal jargon or a language other than English, probabilities drift. TypeSafe states that English is the primary training language and that other languages work less well. The fix is the same one statisticians have always used. Keep a labeled sample of your own data, compare predicted probabilities against actual outcomes, and check the gap on a schedule. The operational section below covers how to do that with an Iceberg table.

The Open-Source Alternatives

Jev is closed. TypeSafe has not released weights or training code and has not announced plans to. For teams that need to run classification inside their own network, or that do not want a waitlist between them and production, the open ecosystem moved fast. It helps to split the options into two groups: established zero-shot classifiers that predate Jev, and new projects that copy Jev's interface.

Established open zero-shot classifiers

GLiClass from Knowledgator is the most mature option in this space. It is a zero-shot sequence classifier inspired by GLiNER, the open named-entity model. GLiClass encodes the text and all candidate labels together and scores every label in a single forward pass. The model card reports performance comparable to a cross-encoder with much lower compute. The gliclass-modern-base-v2.0 checkpoint uses ModernBERT-base as its backbone, has about 151 million parameters, and is trained on synthetic and commercially licensed data. The v3.0 checkpoints add training on logic tasks. It supports single-label and multi-label classification, and it doubles as a reranker for RAG (retrieval-augmented generation) pipelines. At that size it runs on CPU for modest volumes and flies on a single GPU.

NLI-based zero-shot models, such as DeBERTa-v3 checkpoints fine-tuned on entailment datasets, remain a solid baseline. They are well understood, license-friendly, and supported directly by the Hugging Face zero-shot-classification pipeline. Their weakness is cost per label. Each candidate label becomes a separate premise and hypothesis pair, so a 50-option taxonomy means 50 forward passes per row.

CAPPr (completion after prompt probability) takes a different route. It uses any open causal language model, but instead of letting the model generate, it scores the probability of each candidate completion. This turns a generative model into a classifier with a fixed output set. It is slower than an encoder model but works with whatever open LLM you already serve.

New projects that copy Jev's interface

Several projects appeared within days of the Jev launch. All of them are independent. None contain Jev's weights or its RLCD method, and most say so plainly in their READMEs. They are worth watching and not yet worth betting a production pipeline on without your own evaluation.

OpenJev (zhangcy122) wraps open LLMs behind the Choice, Noul, and Score interface. It enforces the output schema with constrained decoding through vLLM or SGLang, pulls log probabilities for each candidate, and applies temperature or Platt scaling to calibrate them. It adds an abstention layer so the system can say "unknown" instead of guessing. The recommended setup runs small models with thinking turned off and escalates to a large model only when confidence falls below a threshold.

Verdict (published on Hugging Face as rlcd-modernbert-151m) is a 151-million-parameter model built on the GLiClass ModernBERT checkpoint and post-trained with an approach inspired by RLCD. It reports single-pass latency under 35 milliseconds and supports up to 24 options plus an explicit abstention slot. Its authors claim wins over Jev on their own typed-decision benchmarks. Those claims are self-reported and have not been independently reproduced.

Open JEV (zhihz) is a research preview that runs a frozen Qwen3-4B-Instruct model to score candidate answers in English and Chinese. Its README is refreshingly direct that no matched comparison against Jev, GLiClass, or CAPPr has been completed.

poorjev reproduces the three question types on top of NLI models and focuses on honest confidence. It applies temperature scaling and conformal abstention and publishes a calibration evaluation showing expected calibration error dropping from 0.170 to 0.071 with no accuracy loss.

The table below summarizes the tradeoffs as they stand in September 2026.

The honest summary is this. If you need a general-purpose decision model today and can send data to a hosted API, Jev is the most polished option. If data must stay inside your network, GLiClass plus your own calibration step is the most defensible choice. If you have a stable task with thousands of labeled examples, a fine-tuned ModernBERT classifier still beats every zero-shot option on accuracy per dollar. The Jev clones are promising experiments.

Which Job Goes to Which Model

The cleanest way to decide is to look at the shape of the answer, not the difficulty of the question.

If the set of possible answers is known before the model runs, a classification model is the default. Routing, tagging, filtering, scoring, deduplication decisions, and policy checks all fit. The answer is a label, a level, or a probability, and your code branches on it.

If the answer is new content, you need a generative model. Summaries, explanations, extracted free-text values like a customer's stated reason in their own words, SQL generation, and anything a person will read as prose all belong to an LLM.

A few jobs sit in between, and they are where the most interesting designs come from.

Extraction with a bounded answer space. Pulling a date, an amount, or an email address out of text looks like generation, but the answer often comes from a small set of candidates. TypeSafe recommends finding candidates with a regular expression or an LLM, then asking the classifier which candidate is correct. Their date extraction pattern turns each date part into a Choice (twelve months, thirty-one days, a bounded year range, plus a "not stated" option) and assembles the date in code.

Judgments that need multi-step reasoning. "Is this contract clause more restrictive than our standard terms?" requires comparing two texts and reasoning about what each permits. Classification models struggle with that level of indirection. A reasoning LLM earns its cost here.

Counting and math. Neither model type is a calculator. Keep arithmetic in code. If you need to count items that match a condition, ask one yes/no question per item and sum the answers in code.

For data analytics specifically, here is how common lakehouse tasks tend to split:

How the Two Work Together

The strongest designs do not pick one model. They put each model where it is strongest and let code connect them. Four patterns show up again and again.

Pattern 1: The cascade

Send every row to the cheap, fast classifier first. Read the confidence. Accept high-confidence answers as final. Send the rest to an LLM, and send whatever the LLM is unsure about to a human.

The economics are easy to reason about. If 85 percent of rows clear the confidence gate, the LLM sees 15 percent of the volume. Your LLM bill drops by roughly that ratio, and your pipeline runtime drops even more because the classifier is so much faster. The quality of the final table is set by the LLM on the hard rows, which is where you wanted it spent.

The threshold is a business decision, not a model setting. A threshold of 0.9 sends more rows to the LLM and costs more. A threshold of 0.6 saves money and accepts more errors. Set it by measuring error rates at several thresholds on a labeled sample, then pick the point where the cost of an error and the cost of escalation balance.

Pattern 2: The router in front of the agent

AI agents that answer questions over lakehouse data make many small decisions per turn. Which tool fits this request? Does this need the semantic layer or a raw table? Is this request asking for data the user is not allowed to see? Is the retrieved context relevant?

Each of those decisions is a classification. Running them through the agent's main LLM adds seconds per turn. Running them through a System One model adds tens of milliseconds. LangChain's integration exposes Jev as a classifier for exactly this role, including model routing: simple lookups go to a small, cheap model and hard debugging tasks go to a capable one. Vercel's team reported checking commands for safety 5 to 18 times faster after moving that check from an LLM to Jev.

For a lakehouse agent connected through MCP (Model Context Protocol, the open standard for connecting AI clients to tools and data), a classifier sits naturally in front of tool calls. It decides whether a proposed query is safe to run, whether it needs confirmation, or whether to block it.

Pattern 3: The guard on the way out

LLMs generate. Classifiers check. After an LLM writes a summary or answers a question, a classifier asks narrow questions about the output. Does every cited passage support the claim it is attached to? Does the answer contain personal data? Does it contradict the retrieved source? TypeSafe's cookbook includes a citation check that uses a Choice to decide whether the quoted context supports a claim, with confidence flagging borderline cases for review.

This is cheap enough to run on every response, which is the point. A guard that you only run on a sample is a monitoring tool. A guard that runs on every response is a control.

Pattern 4: Features for traditional models

This one gets less attention and matters a lot for analytics. Classification outputs are numbers. A Noul probability, a Score level, and a Choice distribution are all features you can feed into a gradient-boosted tree or a regression model. TypeSafe's docs include a cookbook that uses Jev questions to turn free text into numeric features for a CatBoost regressor.

In a lakehouse, that means your text columns stop being dead weight in predictive models. A churn model that only saw structured fields can now include "probability the customer mentioned a competitor" and "frustration level on a five-point scale" from every support interaction. The LLM is not in this loop at all. The classifier converts text to features once, the features land in an Iceberg table, and the modeling team uses them like any other column.

Classification Inside an Apache Iceberg Lakehouse

Apache Iceberg is an open table format. It adds a metadata layer on top of Parquet files in object storage so that many engines can read and write the same tables with ACID transactions, schema evolution, and time travel. That combination of properties makes Iceberg a good home for model outputs, and it is worth being specific about why.

Store labels as their own table

The first design decision is where labels live. The tempting move is to add a category column to the raw table and fill it in. Resist that. Keep the source table as it arrived and write classification results to a separate enrichment table keyed by the source row's ID.

This separation pays off in several ways. You can reclassify everything with a new model version without rewriting the raw data. You can keep two model versions side by side and compare them. Your raw table's snapshots reflect ingestion only, which keeps its history easy to read. And access control gets simpler, because the enrichment table often carries less sensitive data than the raw text.

Store probabilities, not just labels

The enrichment table should carry the full answer, not only the winning label. At minimum, that means the chosen option, its confidence, and the probability distribution. For a Choice with a handful of options, a JSON string or a map column holds the distribution. For a Noul, one double column is enough.

It also needs provenance columns: the exact model version that answered (Jev reports a versioned ID such as jev-1.13.0 even when you call an alias), a version string for your question set, a timestamp, and the route the row took through the cascade. Without these, you cannot tell which labels came from which model after an upgrade, and you cannot audit a bad label back to its cause.

Use Iceberg features that fit the problem

Snapshots and time travel. Every write to an Iceberg table creates a snapshot. When a model upgrade changes labels, you can query the enrichment table as of the old snapshot and compare it to the new one. That gives you a precise diff of every label that changed, which is the best possible input for deciding whether the upgrade is safe.

Branches for write-audit-publish. Iceberg supports named branches on a table. A classification job writes to a staging branch, a validation step checks the output (label distribution, confidence distribution, null counts, row counts against the source), and only then does the branch get fast-forwarded to main. Engines such as Spark expose this through Iceberg's write-audit-publish settings. This keeps a broken prompt or a model regression from ever reaching the dashboards.

Tags for model versions. Tag the snapshot produced by each model version, such as jev-1.13.0-baseline. Tags keep that snapshot from expiring and give analysts a stable name for comparisons.

Partitioning. Partition the enrichment table by day of the labeling timestamp. Incremental jobs append one day at a time, and queries that look at recent labels prune old files.

Incremental processing

Classification jobs should process only new rows. The simplest approach is a watermark: read the latest source timestamp already present in the enrichment table, then scan the source table for rows after it. Iceberg's metadata makes this cheap because the scan planner skips files whose column statistics fall outside the filter. A more precise approach reads the changes between two snapshots of the source table, which some engines expose as an incremental or changelog read.

Throughput and cost math

The published limits make sizing straightforward. Suppose a backlog of 10 million support tickets averaging 300 input tokens each, including the question definitions. That is 3 billion input tokens. At $0.042 per million, the model cost is about $126 for the entire backlog.

The binding constraint is the request rate, not the price. At 1,200 requests per minute and one ticket per request, 10 million tickets take about 139 hours. Packing 20 tickets into one request as an array of state, with one question set per ticket, cuts that to about 7 hours. The token rate limit of 250,000 per second puts a floor of about 3.3 hours on 3 billion tokens. Packing trades accuracy for throughput, because TypeSafe documents that irrelevant detail in the state lowers accuracy. Test packed and unpacked requests on a labeled sample before committing to a batch size.

Compare that to a frontier LLM on the same backlog. The input tokens alone cost more by an order of magnitude or two, and output tokens for JSON add more on top. The exact multiple depends on the model and your prompt, but launch coverage cites 40 to 400 times cheaper and 20 to 200 times faster for Jev. Even at the low end, that turns a quarterly project into a nightly job.

For a self-hosted GLiClass deployment, the math is about hardware instead of tokens. A 151-million-parameter encoder on a single modern GPU processes thousands of short texts per second in batches. The cost is the GPU hours and the engineering time to run the service.

A Worked Example: Support Tickets From Raw Text to Dashboard

Here is a complete, small version of the cascade. The source is an Iceberg table of raw support tickets, support.tickets_raw, with a ticket ID, subject, body, and ingestion timestamp. The goal is a labeled enrichment table and a weekly analytics query that uses probabilities, not just labels.

Step 1: Create the enrichment table

This DDL runs in Dremio against an Iceberg catalog. Any engine that creates Iceberg tables works the same way with its own syntax.

CREATE TABLE lakehouse.support.ticket_labels (
  ticket_id            VARCHAR,
  category             VARCHAR,
  category_confidence  DOUBLE,
  category_probs       VARCHAR,
  churn_prob           DOUBLE,
  urgency_level        INT,
  urgency_confidence   DOUBLE,
  route                VARCHAR,
  model_version        VARCHAR,
  question_set         VARCHAR,
  source_ingested_at   TIMESTAMP,
  labeled_at           TIMESTAMP
)
PARTITION BY (DAY(labeled_at));

Each column has a job. category_probs holds the full distribution as a JSON string, so analysts can see when a ticket split between two teams. churn_prob is a raw Noul probability. route records which path the row took through the cascade. model_version and question_set make every label traceable. source_ingested_at carries the source timestamp forward so the next run knows where to start.

Step 2: Classify new rows and append

This Python job uses PyIceberg, the official Python library for Iceberg, to read new tickets and write results. It uses the TypeSafe Python SDK for classification.

import json
from datetime import datetime, timezone

import pyarrow as pa
import pyarrow.compute as pc
from pyiceberg.catalog import load_catalog
from pyiceberg.expressions import GreaterThan
from typesafe_sdk import Choice, Noul, Score, TypeSafeClient

MODEL = "jev-1.13.0"         # pin a version, not the jev-latest alias
QUESTION_SET = "support-v3"  # bump whenever the questions change
AUTO_ACCEPT = 0.80           # confidence gate for the category answer

QUESTIONS = {
    "category": Choice(
        instructions="Which product area is the customer writing about?",
        criteria={
            "billing": "Charges, invoices, refunds, payment methods",
            "integrations": "Connectors, APIs, webhooks, third-party tools",
            "performance": "Slow queries, timeouts, dashboards that fail to load",
            "account_access": "Login, SSO, passwords, permissions",
            "other": "Anything that fits none of the options above",
        },
    ),
    "churn_signal": Noul(
        instructions=(
            "Does the customer say they plan to cancel, downgrade, "
            "or switch to another vendor?"
        ),
    ),
    "urgency": Score(
        instructions="How quickly does this ticket need a response?",
        criteria=[
            "No time pressure",
            "Should be answered this week",
            "Should be answered today",
            "Production is down right now",
        ],
    ),
}

LABEL_SCHEMA = pa.schema([
    pa.field("ticket_id", pa.string()),
    pa.field("category", pa.string()),
    pa.field("category_confidence", pa.float64()),
    pa.field("category_probs", pa.string()),
    pa.field("churn_prob", pa.float64()),
    pa.field("urgency_level", pa.int32()),
    pa.field("urgency_confidence", pa.float64()),
    pa.field("route", pa.string()),
    pa.field("model_version", pa.string()),
    pa.field("question_set", pa.string()),
    pa.field("source_ingested_at", pa.timestamp("us")),
    pa.field("labeled_at", pa.timestamp("us")),
])


def load_new_tickets(catalog, labels):
    done = labels.scan(selected_fields=("source_ingested_at",)).to_arrow()
    watermark = pc.max(done["source_ingested_at"]).as_py()

    tickets = catalog.load_table("support.tickets_raw")
    scan_args = {"selected_fields": ("ticket_id", "subject", "body", "ingested_at")}
    if watermark is not None:
        scan_args["row_filter"] = GreaterThan("ingested_at", watermark.isoformat())
    return tickets.scan(**scan_args).to_arrow().to_pylist()


def label_ticket(client, ticket, now):
    result = client.system_one(
        state={"subject": ticket["subject"], "body": ticket["body"]},
        questions=QUESTIONS,
    )
    category = result.answers["category"]
    urgency = result.answers["urgency"]
    churn = result.answers["churn_signal"]

    route = "auto" if category.confidence >= AUTO_ACCEPT else "llm_review"

    return {
        "ticket_id": ticket["ticket_id"],
        "category": category.choice,
        "category_confidence": category.confidence,
        "category_probs": json.dumps(category.probabilities),
        "churn_prob": churn.noul,
        "urgency_level": int(urgency.score),
        "urgency_confidence": urgency.confidence,
        "route": route,
        "model_version": result.model,
        "question_set": QUESTION_SET,
        "source_ingested_at": ticket["ingested_at"],
        "labeled_at": now,
    }


def main():
    catalog = load_catalog("lakehouse")  # settings come from .pyiceberg.yaml
    labels = catalog.load_table("support.ticket_labels")
    new_tickets = load_new_tickets(catalog, labels)
    if not new_tickets:
        return

    now = datetime.now(timezone.utc).replace(tzinfo=None)
    with TypeSafeClient(model=MODEL) as client:
        rows = [label_ticket(client, t, now) for t in new_tickets]

    labels.append(pa.Table.from_pylist(rows, schema=LABEL_SCHEMA))


if __name__ == "__main__":
    main()

Walk through the pieces.

The constants at the top are the contract for the whole job. MODEL pins the exact version. The jev-latest alias moves when TypeSafe ships a new release, and TypeSafe's own docs warn that answers behind an alias can change without any change on your side. If you tuned AUTO_ACCEPT against version 1.13, a silent upgrade invalidates that tuning. QUESTION_SET does the same job for your prompts. Change a criteria description and you have changed the classifier, so the version string changes too.

QUESTIONS asks three things in one call. The Choice includes an other option, which TypeSafe recommends whenever the list does not cover every possible input. Without it, the model has to force an odd ticket into one of the real categories. The criteria descriptions are written to separate the options from each other, because the model reads both the option names and their descriptions. The Score levels describe observable situations ("Production is down right now") instead of vague adjectives like "critical," which reduces disagreement between runs.

load_new_tickets implements the watermark. It reads one column from the enrichment table, takes the maximum source timestamp, and filters the source scan with it. PyIceberg pushes the filter down to Iceberg's file-level statistics, so files with only older rows never get opened. On a very large enrichment table, reading the whole timestamp column gets slow. At that point, store the watermark in a small state table or read it from snapshot properties instead.

label_ticket sends only the subject and body as state. It does not send customer metadata, account history, or other columns. TypeSafe documents that accuracy drops as irrelevant detail grows in the state, so the state stays small on purpose. The function stores the full probability distribution, the Noul probability, the Score level, and the versioned model ID that the API reports back.

main runs the calls in sequence for readability. A production version uses the SDK's asynchronous client or a worker pool to stay near the rate limit. It also writes through a staging branch and validates before publishing, as described earlier.

Step 3: Escalate the uncertain rows to an LLM

Rows with route = 'llm_review' go to a generative model. In Dremio, the AI_CLASSIFY SQL function sends text and a category list to a configured LLM and returns one of the categories. The escalation step runs as plain SQL next to the data:

CREATE TABLE lakehouse.support.ticket_labels_escalated AS
SELECT
  l.ticket_id,
  l.category             AS fast_category,
  l.category_confidence  AS fast_confidence,
  AI_CLASSIFY(
    'Which product area is this support ticket about? '
      || t.subject || ' ' || t.body,
    ARRAY['billing', 'integrations', 'performance', 'account_access', 'other']
  ) AS llm_category
FROM lakehouse.support.ticket_labels AS l
JOIN lakehouse.support.tickets_raw AS t
  ON t.ticket_id = l.ticket_id
WHERE l.route = 'llm_review'
  AND l.labeled_at >= DATE_SUB(CURRENT_DATE, 1);

The category list matches the Choice options exactly, so the two answers are comparable. Keeping fast_category next to llm_category gives you a running disagreement dataset for free. Every row where the two models disagree is a candidate for human review and a test case for the next version of your questions. A production pipeline appends to this table with INSERT INTO on each run instead of recreating it.

Step 4: Query with probabilities

The analytics layer combines both tables and uses the probabilities directly:

SELECT
  DATE_TRUNC('WEEK', l.source_ingested_at)          AS week,
  COALESCE(e.llm_category, l.category)              AS final_category,
  COUNT(*)                                          AS tickets,
  SUM(l.churn_prob)                                 AS expected_churn_mentions,
  SUM(CASE WHEN l.churn_prob >= 0.8 THEN 1 ELSE 0 END) AS confident_churn_mentions,
  AVG(l.category_confidence)                        AS avg_category_confidence
FROM lakehouse.support.ticket_labels AS l
LEFT JOIN lakehouse.support.ticket_labels_escalated AS e
  ON e.ticket_id = l.ticket_id
WHERE l.model_version = 'jev-1.13.0'
  AND l.question_set = 'support-v3'
GROUP BY
  DATE_TRUNC('WEEK', l.source_ingested_at),
  COALESCE(e.llm_category, l.category)
ORDER BY week, final_category;

Two churn columns sit side by side on purpose. expected_churn_mentions sums probabilities and gives the best estimate of how many tickets carry churn language. confident_churn_mentions counts only the clear cases, which is the number a customer success team acts on. When the two numbers drift apart week over week, the population of borderline tickets is growing. That is a signal worth investigating on its own.

The WHERE clause filters to one model version and one question set. Mixing labels from different versions in a single trend line produces steps in the chart that reflect model changes, not customer behavior.

In Dremio, you save this as a view in the AI Semantic Layer with a description of each column. That description is what lets an AI agent answer "how many tickets mentioned churn last week" and pick the right column.

A self-hosted variation

When text cannot leave your network, swap the classifier in Step 2 for GLiClass and keep everything else:

from gliclass import GLiClassModel, ZeroShotClassificationPipeline
from transformers import AutoTokenizer

name = "knowledgator/gliclass-modern-base-v2.0"
model = GLiClassModel.from_pretrained(name)
tokenizer = AutoTokenizer.from_pretrained(name, add_prefix_space=True)
pipeline = ZeroShotClassificationPipeline(
    model, tokenizer, classification_type="multi-label", device="cuda:0"
)

labels = ["billing", "integrations", "performance", "account_access", "other"]
scores = pipeline(ticket_text, labels, threshold=0.0)[0]
best = max(scores, key=lambda r: r["score"])

Note what changes. GLiClass returns an independent score per label, not a probability distribution that sums to 1. The raw scores are not calibrated, so a 0.8 from GLiClass does not mean the same thing as a 0.8 from Jev. Before you use a confidence gate, fit a calibration step (temperature scaling or isotonic regression) on a labeled sample and store the calibrated number in the table. The schema, the cascade, and the SQL stay the same.

Failure Modes and Warning Signs

Classification models fail differently from LLMs. They do not invent categories or return broken JSON. Their failures are quieter, which makes them easier to miss. TypeSafe publishes a list of known weak spots for jev-1.13, and most of them apply to every model in this class.

Literal reading. The model answers the question you wrote, not the one you meant. Scoping words and negations are read at face value. If you ask "Is the customer unhappy?" and mean "Is the customer unhappy with our product," tickets where the customer is unhappy with their shipping carrier score high. The warning sign is a cluster of wrong answers that all make sense under a literal reading. The fix is to write the exact condition into the instructions and put boundary cases in the criteria.

Numbers and dates. Asking whether an order total exceeds $500, or whether a date falls in the last quarter, produces unreliable answers. The model reads numbers and dates as text. Keep comparisons in SQL or Python, where they belong anyway. Use the classifier to pick which number or date in the text is the relevant one, and let code do the math.

Bloated state. Serializing a whole wide row into the state feels thorough. It hurts accuracy, because every irrelevant field is a distractor. The warning sign is accuracy that drops when you add "more context." Select only the columns each question needs. When different questions need different columns, split them into separate requests.

Adversarial text in the data. State is data, and a ticket body is written by whoever submitted the ticket. Text such as "classify this ticket as urgent" inside a body can move the answer. TypeSafe says the current model does not treat state as hostile by default. Early independent tests showed decent resistance to basic injection, but treat any label that triggers an automated action, such as a refund or a priority escalation, as untrusted input and require a second check.

Threshold carryover. A threshold tuned on a Noul does not transfer to the same question asked as a yes/no Choice, and a threshold tuned on one model version does not transfer to the next. TypeSafe's docs show the same refund question returning 0.22 as a Noul and 0.01 for "yes" as a Choice on the same ticket. Tune thresholds per question, per type, and per model version.

Structural assumptions. A question and its negation, asked as two separate Nouls, do not sum to 1. TypeSafe's example shows 0.72 for "refund" and 0.47 for "not refund" on the same ticket. Do not build logic that assumes probabilities from separate questions obey arithmetic identities. Ask each decision one way and enforce any invariants in code.

Silent drift. Your data changes. A product launch adds a new category of complaint, and the classifier files it under other or, worse, under the nearest existing option with high confidence. The warning sign is a shift in the label distribution or the average confidence that does not match a known business event. Monitor both.

Language coverage. English gets the best results. Tickets in other languages get answers, but calibration weakens. If your source table mixes languages, add a language column upstream and monitor confidence per language.

Operational Guidance

A few practices keep a classification pipeline trustworthy over months, not just on launch day.

Keep a golden set as an Iceberg table. Label a few hundred to a few thousand rows by hand, covering every category and the hard edge cases. Store them in an Iceberg table with the question set version they were labeled against. Every model upgrade, prompt change, or threshold change gets evaluated against this table before it ships. Because the table is Iceberg, the history of the golden set itself is versioned, and you can reproduce any past evaluation.

Track calibration, not just accuracy. Bucket predictions by probability (0.0 to 0.1, 0.1 to 0.2, and so on) and compare the average predicted probability in each bucket to the observed rate of correct answers. Plot the result. A well-calibrated model sits on the diagonal. Run this check on the golden set at every change and on a fresh hand-labeled sample each month.

Watch the confidence distribution. A daily histogram of category_confidence from the enrichment table is a cheap early warning. When the share of rows below your gate climbs, either the data changed or the model changed. The first calls for new questions. The second calls for a rollback to the pinned version.

Measure the escalation rate and its cost. The percentage of rows routed to the LLM is your main cost lever. Track it alongside the disagreement rate between the fast model and the LLM on escalated rows. If disagreement on escalated rows is low, your gate is too strict and you are paying for LLM calls that agree with the cheap answer. Lower the threshold.

Upgrade on your schedule. When a new model version ships, run it on the golden set, then run it on a recent week of production data into a separate branch of the enrichment table. Compare labels row by row using Iceberg time travel or a join between branches. Promote the new version only after the diff looks right, and retune thresholds as part of the promotion.

Isolate the compute. Classification backfills are bursty. On a self-hosted model, they compete for GPU time with anything else on the same cluster. For SQL-based LLM escalation, route AI function queries to a dedicated engine so a large batch does not slow down dashboards. Dremio supports engine routing rules keyed on whether a query calls AI functions, and other engines have their own workload management controls.

Log the request ID. Hosted APIs return a request identifier. Store it in a side table keyed by ticket ID. When a user disputes a label months later, you can trace it back to the exact call.

Where This Is Heading

The launch of Jev did not invent classification. What it did was name a category and ship a clean interface for it, and that interface spread fast. Within two weeks, it showed up in gateways, agent frameworks, observability tools, MCP servers, and a handful of open-source reproductions.

Three trends look likely to shape the next year.

The first is the split between generation and decision becoming a standard part of AI architecture. Agent frameworks already route tool selection, safety checks, and context pruning to fast models. Expect the same split inside data platforms, where query engines call a decision model for row-level tagging and reserve generative models for summaries and extraction.

The second is open models closing the gap. GLiClass and ModernBERT already give self-hosted teams a strong single-pass classifier. The missing piece has been trained-in calibration and a general interface for arbitrary typed questions. Several open projects are attacking exactly that. Treat their current benchmark claims with skepticism until someone runs a matched comparison, but the direction is clear.

The third is classification outputs becoming first-class data. When every free-text column in a lakehouse can be turned into calibrated probabilities for a fraction of a cent per thousand rows, text stops being the part of the data you skip. Open table formats like Iceberg are well suited to hold this derived layer, because they version it, share it across engines, and let you roll it back when a model change goes wrong.

Conclusion

LLMs made text classification easy to prototype and expensive to run. Models like Jev, along with open alternatives such as GLiClass and the new wave of Jev-style projects, move most of that work back to a tool built for it. The answer is a typed value with a calibrated probability, returned in milliseconds for a small fraction of the cost.

The right design is not one model or the other. Put the fast classifier on every row. Use its confidence to decide what goes to an LLM and what goes to a person. Store full probabilities, model versions, and routes in their own Iceberg table, and let SQL and your semantic layer turn those numbers into analytics. Pin versions, keep a golden set, and watch the confidence distribution. Do those things and the text in your lakehouse becomes as queryable as the numbers next to it.

Keep Going

If this piece was useful, I have written a lot more on building AI workloads on top of open lakehouse tables.
Architecting an Apache Iceberg Lakehouse (Manning) covers how to design the table, catalog, and engine layers that pipelines like this one run on.
You can find every book I have written, across lakehouse architecture, Apache Iceberg, Apache Polaris, and AI, at books.alexmerced.com.

Apache Data Lakehouse Weekly: September 9 to 17, 2026

Alex Merced — Fri, 18 Sep 2026 09:13:23 +0000

Release managers had a humbling week. Iceberg Rust 0.11.0 failed two release votes, Polaris 1.8.0 drew a -1 on its first day, and the Polaris Catalog Migrator needed a second candidate. None of those failures came from broken code paths in the usual sense. They came from license notices, a crate that failed to publish, and an encryption check that one engine skipped and another engine enforced. The lakehouse projects are now mature enough that the hard problems live at the seams: between languages, between engines, between a spec and the many readers that implement it.

That theme runs through every project this week. Iceberg and DataFusion voted to move the Rust DataFusion integration into the DataFusion project, which redraws a boundary that had been drawn in the wrong place. Parquet spent the week arguing about what a reader should do when it meets a file from the future. Polaris debated how to add Cloudflare R2 without bending its S3 configuration into a shape it was never meant to hold. Apache Ossie (incubating) proposed that one document should hold one semantic model, which is a seam question too. And Arrow shipped a Rust release that puts the new ALP floating point encoding into a production Parquet implementation for the first time.

This issue covers the dev lists for Apache Iceberg, Apache Polaris, Apache Arrow, Apache Parquet, Apache DataFusion, and Apache Ossie (incubating), using the Pony Mail archives at lists.apache.org. Every thread link below points to the public archive so you can read the full discussion yourself.

Apache Iceberg

The DataFusion integration finds a new home

The biggest structural change of the week was a clean two-project vote. Kevin Liu opened the Iceberg-side vote to move the iceberg-datafusion crate out of apache/iceberg-rust and into a new apache/datafusion-iceberg repository under the Apache DataFusion project. The move grew out of a discussion thread where Andrew Lamb and Kevin agreed to hold one vote in each community. Gabriel Musat had already done the heavy lifting by porting the code and its full history into a pull request on the new repository.

The Iceberg vote passed with 17 +1 votes, five of them binding, from Kevin, Fokko Driesprong, Renjie Liu, Szehon Ho, and Russell Spitzer. There were no -1 or +0 votes. Xuanwo summed up the reasoning in his +1: DataFusion contributors get a natural place to maintain the integration, and the code stays under Apache governance. On September 16, Andrew confirmed on the same thread that both votes had passed unanimously.

Why does this matter to people who never read Rust? The integration is the glue that lets a DataFusion query plan read and write Iceberg tables. When it lived inside iceberg-rust, every DataFusion upgrade forced a change in the Iceberg repository, and the people who understood DataFusion's planner internals were not the people reviewing the pull requests. Putting the crate next to the engine it plugs into puts the reviewers next to the code. It also sets a pattern. Integration code belongs with the project whose APIs churn fastest, and the table format library can stay focused on the format.

Iceberg Rust 0.11.0 fails twice, and the failures teach something

Danny Jones and Shawn Chang ran the 0.11.0 RC1 vote, and early verifiers reported clean results. L. C. Hsieh ran 2,209 tests on an arm64 Mac with no failures. Anoop Johnson ran the full suite on Ubuntu 24.04, including Docker-based integration tests against MinIO, the REST catalog, Hive Metastore, and Spark. He flagged one flaky test that failed under parallel load with a SQLite "database is locked" error but passed in isolation.

Then Kevin Liu voted -1 for a reason no unit test catches. The new iceberg-property-macro crate depends on nothing unusual, but it lists iceberg itself as a versioned dev-dependency, and iceberg depends on the macro crate. That circle means cargo publish cannot package either crate first. Kevin pointed out that the project hit the same trap with 0.5.0, when the vote passed, the publish failed, and the team had to cut 0.5.1. He put up a fix and added a cargo publish --dry-run check to CI so the problem surfaces on every pull request. He also noted that make test from the tarball failed because the Hive test image's apt repository had expired. Danny agreed and cancelled the vote.

The RC2 vote opened on September 15 and lasted less than a day. Alexander Bailey voted -1 after testing and realizing that the Rust encryption work skipped the tamper-proofing checks that Java requires. Files written by the Rust RC were unreadable in Java. He posted a fix in iceberg-rust PR #3236. Danny closed RC2 and asked reviewers to prioritize the fix so RC3 can follow.

This is the multi-language Iceberg story in miniature. Encryption is one of the headline features in 0.11.0, and the only reliable test for it is a cross-engine round trip. A Rust writer that passes every Rust test can still produce files that a Java reader rejects. The community caught it during the vote, which is the process working. But it also makes the case for the next item.

A shared verification repository goes live

Neelesh Salian announced that the apache/iceberg-verification repository now exists, following an earlier discussion and a successful vote. The repository will hold shared conformance fixtures for every Iceberg implementation. Early work covers type fixtures validated with JSON Schema and golden reference tables. Neelesh asked for reviewers on the open pull requests and for new issues from anyone with ideas about the fixture format or contribution model.

Read that announcement next to the two failed Rust votes. A golden table written by Java and read by Rust, Go, C++, and Python, plus the reverse, is exactly the test that catches an encryption gap before a release candidate exists. The verification repository is still young, but it points at the right problem.

Third-party notices in compiled wheels

Danny Jones raised a licensing gap in pyiceberg-core, the Python package built from iceberg-rust. The wheels ship compiled third-party Rust code, but they do not reproduce the copyright notices for those dependencies. Danny opened iceberg-rust issue #3239 with a proposed fix that generates a THIRD-PARTY-LICENSES file at build time, and he plans to confirm the approach on the legal-discuss list.

Ryan Blue agreed that anything distributed in compiled artifacts needs matching LICENSE and NOTICE content. He was less sure about the THIRD-PARTY-LICENSES file name, but said a pointer at the bottom of LICENSE is reasonable, and a NOTICE file with the legally required notices is mandatory. Shawn Chang pointed to Apache Paimon's Rust bindings as a working example. Their workflow scopes dependencies from the Python bindings' Cargo.toml, generates a report per target wheel, stages it in the artifact, and verifies it. Danny said this must be fixed before new releases of pyiceberg-core, and by extension iceberg-rust, since the two ship together.

Iceberg Java 1.12.0 is ready to cut

Neelesh Salian gave the 1.12.0 release thread two updates. On September 12 he reported that all prior correctness fixes had merged, with one exception: PR #17984, which commits manifest list encryption keys together with the snapshot that uses them. Gábor Kaszab asked whether the missing cleanup mechanism for unused encryption keys in TableMetadata (PR #16353) belongs in this release. On September 16 Neelesh said #17984 had merged, that the key cleanup interface needs more debate and does not need to be rushed, and that the release candidate was ready to cut.

Alex Reid replied on September 17 asking for two more items. One is a Kafka Connect fix (PR #17713) that prevents re-committing already committed files during certain rebalances. The other is the REST client implementation of referenced-by, which the spec added last year. As of this writing, 1.12.0 has not been released. Watch the list for the RC vote.

Defining what gc.enabled means

Alexander Bailey started a careful thread about the gc.enabled table property. His PR #17791 relaxes the check so that snapshot expiry can proceed in a metadata-only mode when garbage collection is disabled. Reviewers pushed back, and Alexander concluded that the real issue is that no one has defined the property. It is not in the spec. He catalogued at least three meanings inside the Java repository alone. CatalogUtil.dropTableData skips data files but deletes metadata. Several Spark actions refuse to run at all. The Hive helper maps the flag onto Hive's external table purge setting. He then showed that iceberg-rust, iceberg-go, and iceberg-cpp each copied Java's strictest check, one of them word for word, while PyIceberg has no check at all.

Russell Spitzer explained the history. The flag arrived when Iceberg started snapshotting existing data, where Iceberg owns new metadata but another system still owns the data files. The invariant is simple: do not remove this table's data files. He was happy to write that down in a spec appendix, and he did not see the Java behavior as ambiguous, since every operation that fails or skips work is one that can delete a data file. Daniel Weeks took a different position. He sees gc.enabled as convention rather than specification and prefers to rely on catalogs to enforce deletion rules than on clients to behave.

Alexander asked Dan whether that means catalogs should vend credentials without delete permission and leave deletion to catalog-managed maintenance. He then narrowed his PR. When gc.enabled=false, the PR allows expiry only with CleanupLevel.NONE, which removes snapshot entries without touching any file. The default mode and METADATA_ONLY stay blocked. That is a sensible scoping move. It keeps the data-file invariant intact and leaves the bigger enforcement question for a separate discussion.

Faster position delete checks in the vectorized reader

A contributor writing as 전대홍 posted a performance proposal after Eduard Tudenhöfner suggested taking it to the list. Spark's vectorized reader asks the position delete index whether each row is deleted, one row at a time, while building the row-id mapping for a batch. Positions in a batch form a contiguous range, so every call repeats the same key extraction, bounds check, and binary search through the Roaring bitmap containers.

PR #18027 adds a forEachInRange default method to PositionDeleteIndex. The default keeps the current loop, so external implementations keep working. BitmapPositionDeleteIndex overrides it to resolve the range to at most two underlying bitmaps and walk each once. The contributor measured 2.6x to 9.3x less delete-check CPU, with end-to-end gains of 13 to 19 percent on narrow integer projections and noise-level gains at ten or more columns. They noted that the largest win lands just below 6.25 percent delete density, where the Roaring container is still a sorted array.

Péter Váry suggested trying a batch iterator in the style of RoaringBitmap's batch iteration. The contributor built it and benchmarked six variants on a 5,000-row batch. The PR's approach came in at 0.45 microseconds per batch on sparse deletes, against 64.99 for the current code and 0.63 for the batch iterator. They kept the minimal API change and offered to test a persistent iterator if the community prefers that design.

Row-level concurrency for V3 deletion vectors

EJ Song and Huaxin Gao continued a thread on row-granular conflict detection for deletion vectors. Today, two concurrent UPDATE or MERGE commits that touch the same data file conflict even when they delete different rows. Huaxin supported the direction but warned that the extra check adds driver-side reads during commit, and suggested gating it behind a table property. She pointed to prior art, including Iceberg PR #17754, which already merges concurrent deletion vectors for pure delete commits.

EJ laid out a layered design. An operation gate comes first, then a metadata check on partition and column bounds, and only then a single small bitmap read when two commits really overlap on one file. Commits on disjoint files read nothing. EJ argued that the bitmap tier does not need a table property because its cost is already bounded. A later tier that compares actual values to resolve overlaps a bitmap cannot decide reads data files, and that tier is where a property makes sense. This is the kind of change that makes V3 tables far friendlier to high-concurrency streaming and CDC workloads.

Access delegation for the FILE type

Sung Yun opened a discussion on access delegation for the proposed FILE type, which lets an Iceberg column reference external objects such as images or documents. Remote signing and vended credentials work when a storage client reads the object. Sung's concern is the multimodal case, where an inference service outside the query engine needs to fetch the object. He proposed letting clients ask the catalog to pre-sign FILE reference URLs, with a spec PR (#18080) and a client proof of concept (#18110). He had earlier posted a related implementation that handles pre-signed URLs returned in place of native paths in a plan response.

Prashant Singh noted that he and William Hyun opened a pre-signed URL proposal three months ago. Their proofs of concept, including one on Azure, showed limits in remote signing across clouds. He asked to build on those tracks rather than start parallel ones, and Sung agreed. Daniel Weeks drew a useful line. Pre-signed URLs from plan tasks arrive ready to use, so file IO only needs to detect and follow them. Remote pre-signing is an earlier step, where a native s3:// path goes to the catalog for signing before a stream opens. Dan wants remote pre-signing to become a real alternative to remote signing, largely because of Azure's limits, and does not think it needs a new /presign endpoint. He also said he disagrees with the direction of the current bulk-signing proof of concept and wants the FILE sync to align the designs.

Smaller threads worth a look

Shangqing Yang called a vote to add the optional key-id field to the Snapshot schema in the REST OpenAPI spec, which aligns the spec with what SnapshotParser already supports. Sreesh Maheshwar revived the question of replacing MinIO in tests after MinIO images disappeared from Docker Hub and broke CI. Kevin Liu, Xuanwo, and Szehon Ho backed moving to RustFS, which PyIceberg and Polaris already use. Neelesh Salian asked for long-term support and prompt CVE fixes, citing an earlier security concern.

Renjie Liu gave a +1 to a proposal for a standard User-Agent format for REST clients, and Kurtis Wright asked how the format should order vendor, engine, integration, library language, and runtime. A contributor asked for review of an ObjectStore-based S3 backend for iceberg-rust in PR #3165. Péter Váry reported from the index support sync that the group will prioritize scalar indexes before other index types, that index data lives in region files tracked by a tracking file, and that the scalar index spec PR #16961 is ready for review. Huaxin Gao scheduled one more constraint sync for September 17 and asked for reviews of the CHECK constraint spec PR #17822.

Ryan Blue shared the September board report with a new format that summarizes shipped releases and spec changes instead of sync highlights. It lists 40 committers and 25 PMC members, and releases including Terraform Provider 0.1.0, PyIceberg 0.12.0, Rust 0.10.0 and 0.10.1, and C++ 0.3.0. Ryan asked release managers to write real highlights in their vote and announce emails, and Danny Jones committed to doing that for iceberg-rust. On September 17, Fokko Driesprong announced that Gang Wu has joined the Iceberg PMC, crediting Gang's work on the inception and growth of Iceberg C++.

On the community side, Kevin Liu announced the first Apache Iceberg Virtual Meetup, covering GSoC projects and commutative compaction on Friday, September 18. Alex Stephen announced a Seattle community meetup on October 21, with session ideas due October 6. Sung Yun, Walaa Eldin Moustafa, and others volunteered for the Iceberg Summit 2027 selection committee.

Apache Polaris

Polaris 1.8.0 reaches a vote, and a NOTICE question stops it

Jean-Baptiste Onofré opened the vote for Apache Polaris 1.8.0 RC0 on September 17, after flagging on the proposal thread that he was preparing it. The candidate ships source tarballs, Helm charts, a Python CLI wheel on Test PyPI, and staged Maven artifacts. Within hours, Dmitri Bourlatchkov voted -1. Bundled Iceberg jars such as iceberg-api-1.11.0.jar carry a NOTICE that references Kite, and commons-math3-3.6.1.jar carries one that references Orekit. The Polaris bundle NOTICE does not propagate either. Dmitri asked whether those notices belong in the Polaris bundle, and the vote is open as of this writing.

The same pattern hit the Polaris tools repository. Ajantha Bhat's Iceberg Catalog Migrator 1.1.0 RC1 drew a -1 from Dmitri because the CLI jar bundles a taglib.tld file with an Oracle copyright header under GPL v2 with the Classpath Exception, and the bundled LICENSE does not mention it. Ajantha posted RC2 on September 17.

If you are keeping score, that is four release candidates across Iceberg Rust and Polaris this week that stopped on licensing or packaging, not on logic. Fat jars and compiled wheels pull in other projects' legal text, and the ASF requires it to travel with the artifact. The reviewers doing this checking are doing unglamorous, important work.

Cloudflare R2 support and the shape of S3 configuration

The most instructive design thread on the Polaris list started with a first-time contributor. Austen Tomek from Chicago Trading Company proposed R2 support with scoped credential vending. R2 speaks the S3 API but has no AWS STS. Cloudflare instead lets a server holding a parent API token sign short-lived, bucket-scoped credentials locally. Austen's prototype does that inside Polaris, maps Polaris location grants to prefix and access scopes, and reuses the existing credential cache.

Yufei Gu confirmed the model: Polaris issues the credentials without calling Cloudflare, the client signs its own S3 requests, and R2 enforces scope and expiry. His first instinct was to treat R2 as S3-compatible storage without a new config type, and he asked whether unmodified clients work. Austen reported that they do. PyIceberg 0.11.1 and 0.12.0, DuckDB 1.5.5, and Iceberg Java 1.11.0 through the stock RESTCatalog and S3FileIO all read and wrote without any R2-specific client code. Spark, Trino, and Flink are not tested yet.

The debate then split along a clear line. Sushant Raikar laid out two options, a first-class R2 type with its own FileIO or R2 as S3-compatible storage, and pushed for a lowest common denominator across S3-compatible backends. Jean-Baptiste argued for building under the existing S3 config and noted that AwsStorageConfigurationInfo already carries endpoint, stsEndpoint, pathStyleAccess, and an stsUnavailable flag. He framed the real design question as a pluggable vending path for S3-compatible stores without STS. Dmitri leaned toward a separate R2StorageConfigInfo, since mixing Cloudflare account IDs into an AWS-shaped class blends unrelated concepts.

Prithvi S offered the cleanest synthesis. Keep the client side as plain S3, with s3:// locations and S3FileIO. On the server side, avoid overloading stsUnavailable, because that flag means "vend nothing" today, and R2 needs the opposite: vend, just not through STS. Reusing it silently changes behavior for existing MinIO and NetApp catalogs.

Austen closed the loop on September 14 and again on September 16. R2 has no IAM roles or ARNs, so he left those fields empty. He added one new S3 field, now named credentialVendingMechanism, with a server allowlist, SUPPORTED_S3_CREDENTIAL_VENDING_MECHANISMS, that defaults to STS only. Following Dmitri's suggestion, mechanisms are CDI beans looked up by identifier, so downstream builds can add their own without changing the API spec. Stage one is PR #5513, and the R2 bean follows in a second PR. Yufei noted the underlying mismatch: the S3 storage type mixes shared S3 settings with AWS-specific ones, and a future refactor can split them. This is a good template for how to add a backend to a catalog without special-casing it.

Pagination moves from optional to default

Pagination came up in four threads. Yong Zheng opened separate discussions for the generic table API and the policy API. In both cases the REST spec advertises page-token and page-size, but the server calls PageToken.readEverything() and ignores them. For generic tables, Prithvi and Jean-Baptiste steered toward a default method on the GenericTableCatalog SPI, since federated Hive, Hadoop, and BigQuery catalogs implement that interface and a concrete-class overload is unreachable from the handler. Yong posted PR #5533. For policies, the group agreed to paginate listPolicies now, including when filtering by policy type, and to handle applicable-policies separately.

Ayush Saxena raised a spec tension around a configurable server-side maximum page size. The Iceberg REST spec says a server must return everything in one response when the client sends no page token. A server cap truncates that response, and an older client that ignores next-page-token reports a partial list as complete. Dmitri called server-side truncation a basic overload control and a reasonable deviation, since administrators can choose the setting per deployment. He proposed merging.

Then Yufei proposed turning LIST_PAGINATION_ENABLED on by default. Jean-Baptiste noted that the maximum page size defaults to unlimited, so clients that send no pagination parameters see no change. Ayush, Yong, Nándor Kollár, and Dmitri all backed keeping the flag as an escape hatch for a couple of releases, and Yong opened PR #5534. EJ Wang raised the same question for the new Tag API, which is covered below.

Read replicas, and the consistency they break

Yong Zheng started a thread on offloading read-only requests to a database replica when a client sends a custom header. His motivation is connection math on the JDBC backend: at 50 connections per pod and a 5,000-connection database limit, a deployment tops out near 100 pods.

The pushback was about correctness. Prithvi pointed out that the load test Yong cited ran at roughly 400 requests per second sustained with mostly idle connections, so connection limits were not the bottleneck. He also reminded the group that Iceberg clients expect to read their own writes, and asynchronous replicas break that. Jean-Baptiste argued against header-based routing, since Spark and Flink will never send a Polaris-specific header, and against verb-based routing, since some reads in JdbcBasePersistenceImpl have side effects such as idempotency-key bookkeeping. He suggested routing at the persistence method level. Dmitri was more open to an explicit opt-in header, perhaps renamed to make the staleness risk obvious, and warned that the shared entity cache will misbehave with two connection pools. Yufei asked for measurements first: connection wait times, database query time, and metadata file IO, which often dominates loadTable.

Location overlap checks

Dmitri Bourlatchkov opened a discussion on hasOverlappingSiblings(). With the OPTIMIZED_SIBLING_CHECK flag on, persistent implementations search the whole catalog. With it off, LocalIcebergCatalog searches only immediate siblings. Dmitri argued that a speed-up flag should not change what a validation checks, and proposed narrowing the implementations. Prithvi and Eric Maynard disagreed. Eric recalled that the catalog-wide search was an intentional fix for a security gap under certain configurations. Jean-Baptiste identified the concrete case: with ALLOW_UNSTRUCTURED_TABLE_LOCATION, a table under one namespace can point inside another namespace's tree, and only the catalog-wide check sees it. He prefers a confusingly named flag over a correctly named, weaker check. Dmitri agreed with catalog-wide checks and asked why that coverage is not the default.

Tags, semantic model privileges, and the CLI

EJ Wang posted progress on the Tag spec. PR1 covers the API contract, PR2 covers definition CRUD, and a new PR3 adds assign and unassign for catalogs, namespaces, tables, and top-level columns, with atomic detach-all on JDBC and in-memory backends. Version tokens are opaque, and stale updates return 409. Dmitri asked for persistence SPI details and a dedicated section on new authorizer operations. EJ later asked the community to pick between keeping Tags under the existing catalog URL root or giving them an independent /api/tags/v1 root, and between opt-in and always-on pagination.

Yufei Gu asked for feedback on dedicated semantic model privileges in PR #5492, splitting the single CATALOG_MANAGE_CONTENT privilege into list, create, read, write, drop, full metadata, and grant management. Prithvi supported it and caught that NAMESPACE_FULL_METADATA and CATALOG_FULL_METADATA should not act as umbrellas, and Yufei fixed that. Jean-Baptiste strongly backed dedicated privileges that mirror other entity types. Semantic models in a catalog, with their own grants, connect directly to the Ossie work later in this issue.

A GitHub discussion asking for table and view support in polaris setup export/apply became a CLI thread. Yong proposed adding register and create. Prithvi argued that register, which takes a name and a metadata.json location and preserves history, is the primitive operators need. Jean-Baptiste found that the generated REST client already exposes register, so the change is mostly wiring. Dmitri and Yufei agreed that create is outside the CLI's scope for now. The sequence the group settled on is register first, then export and apply with table and view grants, and file-based create later if anyone needs it.

Rounding out the week, Dmitri put consistent multi-object persistence changes on the September 17 community sync agenda. Arun Suri agreed to defer his ambiguous JDBC commit fix until PR #5263 merges, then add a reconciliation step that reloads the entity and treats a matching metadata location as success. And Yufei, Ayush, Robert Stupp, and Jean-Baptiste discussed GitHub Actions queue delays caused by ASF-wide runner saturation. Ayush tested which short jobs can move to ubuntu-slim, and the group decided to hold off unless delays return.

Apache Arrow

Arrow Rust 60.0.0 ships the first ALP-enabled Parquet implementation

Andrew Lamb called the vote for Arrow Rust 60.0.0 on September 10, and it passed on September 15 with five +1 votes, three binding, from L. C. Hsieh, Jeffrey Vo, and others. Andrew did not undersell it. He called the release "epic" and wrote that arrow-rs is, as far as he knows, the first Parquet implementation to include the new ALP encoding. He credited Kosta Tarasov and Devan for leading that work, Ed Seidl for the new Parquet PageIndex structures, and Richard Baah and others for a long list of performance improvements. The crates, including arrow and parquet 60.0.0, are on crates.io.

ALP, short for Adaptive Lossless floating-Point compression, targets float and double columns, which general-purpose encodings handle poorly. Sensor readings, prices, and model features all live in those columns. Having a shipped implementation in Rust matters because DataFusion, iceberg-rust, and a long list of Rust-native engines consume the parquet crate directly.

Xuanwo's review of the release is worth reading for anyone who runs release votes. He first voted +0 because arrow-array/src/delta.rs contains MIT-licensed code derived from chronoutil, and the root LICENSE.txt does not identify it. After another look he decided a NOTICE mention is enough and changed to +1. Andrew filed issue #11098 to track the fix for the next release.

Object Store 0.14.2 needs a second candidate

The Rust object_store crate had a similar week. Andrew's 0.14.2 RC1 drew a -1 from Xuanwo because src/client/s3.rs was missing the first line of its ASF license header. Andrew traced the defect back more than three years and asked whether it needed a new candidate, since it was not a regression. Xuanwo moved to +0 but noted the vote had just started and a wrong license header is a real risk for downstream users. Andrew cut RC2 with the header restored, and it passed on September 15 with three binding votes. Version 0.14.2 is on crates.io. object_store sits under DataFusion, iceberg-rust, and many other Rust data tools, so its releases propagate quickly.

A canonical range type and Lakehouse Day EU

Rok Mihevc revived the proposal for an arrow.range canonical extension type for bounded ranges. Florian replied that he still wants to finish it and addressed Felipe Oliveira Carvalho's earlier suggestion to use separate types for open and closed bounds. After reading how Arrow C++ resolves compute kernels, Florian concluded that separate types help less than they seem. Every extension type reports Type::EXTENSION, so a kernel matcher needs a custom check on the extension name either way. A single arrow.range with a closed flag is the same amount of work, and the flag is read once when the type is reconstructed, not on every compute call.

Danica Fine invited the Arrow community to Lakehouse Day EU 2026 on Saturday, October 10 in Glasgow, co-located with Community Over Code. The full-day, multi-track event covers Iceberg, Flink, Spark, Arrow, Parquet, Polaris, Gravitino, Paimon, Hudi, XTable, Fluss, and more. The schedule includes a talk revisiting an Arrow-based client protocol redesign and a closing panel on the future of the open lakehouse.

Apache Parquet

Format 2.14.0 is released

Fokko Driesprong announced on September 11 that the Parquet Format 2.14.0 vote passed, with binding +1 votes from Gang Wu, Gábor Szádovszky, Andrew Lamb, and Fokko, plus non-binding votes from Divjot Arora and Vinoo Ganesh. Fokko thanked László for correcting links that had a copy-paste error. Andrew followed up with website pull requests that document the 2.14 changes, including ALP and the FILE logical type, and asked for reviews.

On the same day, Divjot Arora closed the vote on Extended Precision Nanosecond Timestamps with seven +1 votes, five binding, from Daniel Weeks, Micah Kornfield, Ryan Blue, Gábor Szádovszky, and Fokko. The change will merge as parquet-format PR 601. Nanosecond timestamps with a wider range matter for anyone storing high-frequency trading or telemetry data that also needs to represent dates far from the epoch.

ALP implementations keep moving across languages. The September 9 sync notes from Andrew Lamb report that the Rust implementation is merged, C++ is waiting on approval after several review rounds, and Java is under review. Vinoo Ganesh posted an update on the Java side: he split parquet-java PR #3791 out of the main ALP PR #3397 based on Russell Spitzer's feedback, and plans to merge the smaller PR first. The sync notes also recorded progress on the vector logical type. The group roughly agreed that the finite-element requirement is a MUST, that statistics help but readers must not depend on them, and that a LIST-based layout is the more popular option, though without full consensus.

What should a reader do with a file from the future?

The week's defining Parquet debate began when Ryan Blue split a question out of the versioning proposal thread. Daniel Weeks had summarized that thread's community sync discussion, noting that the real disagreement is less about recording a version in the file and more about what guarantees readers get. Ryan framed three options. Option 1: a reader must fail on a file written with an unsupported format version. Option 2: a reader should attempt to read it. Option 3: leave it to implementations.

Ryan was careful to scope the question. He is not talking about "preview" features like new encodings or forward-compatible logical types, which already fail only when a reader projects the affected column. The question covers changes to metadata semantics, such as making path_in_schema optional, adding offset and size fields to page headers so page data can move, or fixing statistics written with the wrong sort order. In a separate reply, Ryan said he strongly supports option 1. Option 2 requires a guarantee that every future change will either fail or read correctly in every existing reader, including custom Thrift parsers built for speed whose handling of missing required fields nobody knows.

Andrew Lamb wrote a clear summary of the trade-off. Option 1 makes the spec easier to change but forces readers to reject files they know how to read. Option 2 lets readers handle more files but makes every future spec change carry a compatibility burden that is hard to define. Andrew's view is that Parquet is implicitly option 2 today, and that changing it now creates confusion. Ryan disagreed: the current state is that Parquet avoids breaking changes altogether, using tricks like a second set of min and max fields when the string sort bug surfaced, and he wants a real mechanism for making breaking changes.

Antoine Pitrou questioned the "preview" label and asked whether Thrift parsers actually reject payloads missing a required field. Will Edwards argued for what he called option 0: keep Thrift-based evolution, bump the parquet-format package version as a hint to implementers, and do not gate files on a version. He noted that one widely used reader does not even check the trailing PAR1 magic bytes. Ryan replied that he voted against making path_in_schema optional precisely because it breaks existing guarantees.

Later in the week, the momentum shifted toward option 1. Kurtis Wright said a clear failure with an actionable error is the better user experience, and that implementations choosing to skip version checks accept the consequences. Xiening Dai voted for option 1, opposed changing the magic bytes because many tools use them to detect file type, and proposed shipping a reader patch that fails fast on newer versions before any breaking change lands. Fokko Driesprong also backed option 1 and suggested folding the path_in_schema footer trim into the proposed new footer, so all the breaking changes arrive together in one version.

This debate matters well beyond Parquet. Every table format, every query engine, and every lakehouse catalog depends on Parquet readers behaving predictably. A strict version gate makes the format easier to improve, which is how Parquet gets smaller footers and faster metadata. It also means operators will need to track reader versions across every engine in their stack before they turn on new writer features.

New type proposals and forward compatibility

Neelesh Salian proposed WIRE, a logical type for serialized wire-format messages such as Protocol Buffers and Thrift compact. Today, teams either store those messages as opaque blobs, explode every field into columns with a full decode on write, or convert to Variant and carry a field-name dictionary that protobuf never needed. WIRE stores the original message verbatim in a value column, so it round-trips byte for byte, and shreds the queried fields into native columns using the Variant shredding layout. Because protobuf and Thrift address fields by number, no metadata column is needed, and a reader that does not know WIRE sees ordinary columns.

Micah Kornfield pushed the decimal floating-point proposal toward a formal design document. He listed two open issues: how to represent normalization, possibly through statistics, and how to represent the value itself, which needs benchmarked alternatives to the IEEE 754 layout. Jiayi Wang sent a reminder about the footer sync and shared the updated Modular Footer proposal. Micah asked her to start a DISCUSS thread for visibility and to either broaden issue #530 or open a new proposal.

Gábor Szádovszky and Jörn Horstmann continued a thread on rewriting files with unknown metadata. Gábor argued that a rewriter that does not understand a logical type such as a shredded VARIANT will produce a useless tangle of structs and lists, so it should fail. Jörn pointed out that unknown column orders affect every column in a file, with no way to mark just one column, and that arrow-rs uses an internal unknown variant it cannot serialize. Gábor said he is open to an UNKNOWN column order that makes min and max unusable. Michael Chavinda also asked for a reviewer to add DataHaskell's Parquet reader to the implementation support matrix.

Apache DataFusion

The iceberg integration arrives

On the DataFusion side, Andrew Lamb opened the vote to accept apache/datafusion-iceberg, and it passed on September 16 with 14 +1 votes. Andrew called it a first step toward better Iceberg integration. The GitHub traffic on the dev list showed the work already underway, with Gabriel Musat's PR porting the iceberg-rust history and a second PR adapting the project to compile and pass tests in its new home.

For DataFusion users, this means Iceberg table support will evolve on DataFusion's release cadence, reviewed by people who know DataFusion's physical planning and pushdown APIs. For Iceberg users who embed DataFusion, it means one fewer version lock between two fast-moving Rust projects.

Releases, LTS, and CI

Three release votes closed. DataFusion 55.1.0 passed with nine +1 votes, six binding, and Tim Saucer ran the release. Xuanwo's verification ran 11,100 tests and 502 SQL logic test files, and he flagged several .slt files missing license headers, then fixed them himself in PR #25182. The sqlparser-rs 0.63.0 vote drew a -1 from Xuanwo for a missing NOTICE file and a bundled flamegraph SVG containing CDDL-licensed JavaScript. Andrew showed both issues were years old, filed fixes, and asked Xuanwo to reconsider. Xuanwo moved to +0, and the release passed on September 13 with five binding votes.

Andrew also asked whether DataFusion needs LTS versions, motivated by making third-party integrations easier. That question lands at the right moment, given that DataFusion now hosts the Iceberg integration and has a proposal to host Variant support too. Kosta Tarasov started that Variant discussion around PR #22908, which brings datafusion-variant into the main repository as an extension crate. Andrew suggested keeping it external with better cookbook documentation and moved the conversation to issue #21301. Kosta's open questions include whether Spark semantics should be the naming baseline.

Andy Grove enabled a GitHub merge queue for DataFusion Comet to cut CI load. Pull requests now run a smaller tier, while Spark 3.4, 3.5, and 4.0 SQL tests and older Iceberg suites run once in the queue on the exact merged tree. He also proposed Comet 1.1.0 before the end of September, about six weeks after 1.0.0, and mentioned that he used Claude to triage open high-priority bugs for the release. Tim Saucer announced that Dewey Dunnington is a new DataFusion committer. Andrew's sync notes covered subquery decorrelation, a new blocked aggregation proposal in issue #24704 that asks aggregate authors to review the API, and automating backports.

Apache Ossie (incubating)

The first release is a plumbing exercise

Jean-Baptiste Onofré set expectations for the first Ossie release. It will be 0.3.0, not a 1.0 spec, and its main purpose is to prove the release machinery and complete a full legal check, with a source distribution only. Russell Spitzer agreed that the team should treat it as practice for license checking and release management. Markus Weimer offered help. Kurt Stirewalt asked that the ontology working group's PR #332, which adds two optional fields to the ontology spec, be included, and asked what the formal process is for proposing features for a release.

Microsoft's Power BI group plans to announce its participation

Markus Weimer asked for guidance because Microsoft's Power BI group wants to announce its commitment to Ossie at the upcoming Fabric Community Conference. The group plans to build a converter between Power BI semantic models and Ossie and contribute it. Russell said stating an intent to contribute and support the project is fine as long as no one claims control. Josh Klahr suggested adding Microsoft to the Ossie ecosystem page. Jean-Baptiste pointed to the incubator's branding and publicity guides and recommended "co-creator" or "co-author" language that puts the community first.

A BI vendor committing to a bidirectional converter is exactly the kind of adoption signal a semantic model spec needs. Semantic layers only become portable when the tools on both ends read and write the same format.

One model per document, and richer modeling

Yufei Gu proposed that each Ossie JSON or YAML document hold exactly one semantic model, with its fields at the document root instead of inside a semantic_model array. Several converters already warn and convert only the first model in a document. Jean-Baptiste supported it and asked for a sequencing plan so converter fixtures do not break in the gap. Khushboo Bhatia agreed, and Yufei opened converter PR #396 to merge right after the spec change.

The modeling discussions went deep. On hierarchy support, Yufei shared a Mondrian-style example separating attributes from hierarchies. Julian Hyde credited the attribute-based hierarchy model to Analysis Services 2005, noted that each level must be functionally dependent on its parent, and said he now sees hierarchies mostly as presentation hints. Josh Klahr asked whether hierarchies should be navigation aids or real containment objects. Yufei argued that drill-down needs to know which states belong to which country, which is more than a hint.

On metrics trees, a contributor proposed an optional depends_on array for metric lineage. Julian Hyde replied that dependency information is derivable, that redundant data causes its own headaches, and that Ossie should invest in being a real language with a formal specification, compliance tests, and a compiler that produces the dependency tree. Chris Eubank proposed adding the SQL-standard FILTER (WHERE ...) aggregate modifier to the expression language in PR #382. Mario De Felipe raised, in the cardinality discussion, how to declare source values that mean "missing," such as SAP's type-specific initial values, and filed issue #401.

Agents as spec users

Marco Ciavarella from Exmergo proposed a dedicated place for "field reports" written by coding agents that build with Ossie. His first report, drafted by Claude Code, mirrored a production dbt MetricFlow layer into one Ossie document with 11 datasets, 77 fields, 15 relationships, and 27 metrics. Marco's argument is that agents generate fragmented issues, and a structured report captures what worked, what confused the agent, and what workarounds it needed. Jean-Baptiste liked the separation between blocking and costly issues and the explicit "didn't test" section, and suggested a GitHub discussion category.

Contributions kept flowing too. Poorva Barve introduced herself and sent fixes for the Snowflake converter, which invented a fake table when a query source began with a comment, and the dbt converter, which mangled compound aggregate arguments like SUM(orders.gross - orders.tax). Other threads proposed a bidirectional OpenMetadata converter and LinkML converters that open a path to RDFS and OWL. Ankit Tandon shared notes from the Ontology working group.

Cross-Project Themes

Release hygiene is now the bottleneck. Count the stopped votes: Iceberg Rust RC1 on a publish cycle, Iceberg Rust RC2 on a cross-engine encryption check, Polaris 1.8.0 on bundled NOTICE files, the Polaris Migrator on a bundled GPL file, Arrow Object Store RC1 on a license header, and sqlparser-rs on a NOTICE file and CDDL JavaScript. Add Danny Jones' pyiceberg-core wheel notices and Xuanwo's MIT attribution note on Arrow Rust 60. The pattern is clear. Rust crates, Python wheels, and fat jars all bundle third-party code, and every one of them has to carry the right legal text. Xuanwo, Dmitri Bourlatchkov, and Danny Jones are doing verification work that protects every downstream user. Projects that automate license reports in CI, the way Shawn Chang described for Paimon, will spend fewer weeks restarting votes.

Cross-implementation testing is the missing layer. The Rust encryption gap, the gc.enabled inconsistencies across five clients, and Parquet's reader version debate all describe one problem. A spec is only as reliable as the least careful implementation of it. The new iceberg-verification repository is Iceberg's answer. Parquet is choosing between a strict version gate and a softer contract. Ossie's converter bug fixes and agent field reports are the same problem appearing early, while the spec is still small enough to fix cheaply.

Integration code is moving to the engine. The iceberg-datafusion move, the DataFusion Variant discussion, and Andrew Lamb's LTS question all ask where integration code should live and how stable the host needs to be. The answer this week was to move integrations next to the APIs they depend on and give third parties a stable release line to build against.

Semantics are becoming catalog objects. Polaris is adding dedicated semantic model privileges. Ossie is simplifying its document shape while a major BI vendor plans a converter. Iceberg is designing FILE-type access delegation for multimodal inference services. The lakehouse catalog is growing from a table registry into the place where business meaning, access policy, and non-tabular data references all live together. That is good news for anyone building agents, which need exactly that combination of context and control.

Looking Ahead

Watch for Iceberg Java 1.12.0 RC1, which Neelesh Salian said is ready to cut, and for Iceberg Rust 0.11.0 RC3 once the encryption fix in PR #3236 merges. The Polaris 1.8.0 vote will turn on how the community resolves the NOTICE question, and the Catalog Migrator RC2 vote runs through the weekend. On the Parquet side, the reader versioning thread appears to be converging on option 1, and a formal decision will shape how the new modular footer ships.

The Iceberg virtual meetup on GSoC projects and commutative compaction runs Friday, September 18. The Iceberg constraint sync meets September 17, and the Polaris community sync the same day takes up consistent multi-object persistence. Further out, Lakehouse Day EU lands in Glasgow on October 10, and the Seattle Iceberg meetup follows on October 21.

Keep Learning

If you want to go deeper on Apache Iceberg, Apache Polaris, Apache Arrow, Apache Parquet, and the agentic lakehouse, I have written books that cover the architecture, the catalogs, and the practical engineering behind all of it. You can find every one of them at books.alexmerced.com.

AI Weekly: DeepSeek Cuts Prices as Agents Go Hosted

Alex Merced — Thu, 17 Sep 2026 20:07:12 +0000

Week of September 10 to 17, 2026

The first ten days of September belonged to the frontier labs. This week belonged to everyone else. DeepSeek shipped the month's only price cut, Sakana split its orchestrator into a cheap tier and a premium tier, and Shanghai AI Lab dropped a 744B open-weight research agent with almost no fanfare.

The tooling news moved in one direction. OpenAI, Anthropic, and GitHub all shipped features that run agents for you, measure what they cost, and control what they are allowed to do. And at the AI Infra Summit in Santa Clara, Intel CEO Lip-Bu Tan said the memory shortage behind all of this will get worse next year, not better.

Here is what happened, in the usual order: models, tooling, standards, and infrastructure.

Models: DeepSeek V4.1 Flash Resets the Cheap Tier

DeepSeek V4.1 Flash

DeepSeek released V4.1 Flash on September 10. It is the smallest model in a new architecture family, and it ships with native visual understanding. The API model name is deepseek-flash. Weights are on Hugging Face under the MIT license, along with a technical report.

The architecture is the headline. V4.1 Flash is a 552B-parameter mixture-of-experts model built on what DeepSeek calls a causal encoder-decoder design. It activates only 8B parameters when reading input and 16B when generating output. That split matters because agent workloads are input-heavy. A coding agent re-reads the same files and conversation history on every step, so most of its tokens are prefill, not decode.

The second headline is the KV cache. DeepSeek says V4.1 Flash needs one quarter of the high-bandwidth memory (HBM) and one eighth of the SSD storage that the previous generation used for its cache. DeepSeek ties that directly to price, noting that cache-hit charges often make up a large share of agent costs.

The rate card reflects the architecture. According to a tracker that reads each lab's pricing docs, peak rates are $0.30 per million input tokens and $1.20 per million output tokens, down from $0.44 and $1.32 for V4 Flash. The cache read dropped from $0.014 to $0.006, a 57 percent cut. Off-peak rates are half of peak. The same tracker lists a 1M-token context window and 384K maximum output. The official numbers live on the DeepSeek pricing page, and the new rates took effect at 04:00 UTC on September 10.

Benchmarks are vendor-reported. DeepSeek's launch post says V4.1 Flash beats flagship models, including its own V4 Pro, on its benchmark set. The tracker above lists DeepSeek-reported scores of 90.6 on Terminal-Bench 2.1, 74.2 on DeepSWE v1.1, 90.9 on GPQA Diamond, and a 3471 Codeforces rating. None of those have been independently reproduced yet. Terminal-Bench 2.1 is also two major versions behind the Terminal-Bench 4.0 board where GPT-6 Astra and Claude Fable 5.1 were measured earlier this month, so the numbers do not line up with that ranking.

There are breaking changes. DeepSeek retired V4 Flash and V4 Flash Vision Exp. The old model IDs deepseek-v4-flash and deepseek-v4-flash-vision-exp now route to V4.1 Flash for compatibility. DeepSeek's launch post also said that starting at 04:00 UTC on September 14, all deepseek-v4-pro requests were set to route to V4.1 Flash until V4.1 Pro launches. The tracker reports that DeepSeek's API changelog now says V4 Pro service continues past that date in response to user demand. If you pinned V4 Pro, check the changelog before you assume which model is answering.

What it changes for practitioners: The cheapest capable tier just got cheaper on exactly the line that agent loops burn. If you run long agentic jobs on a budget, test V4.1 Flash against your current model with your own evals. Two caveats apply. DeepSeek's peak window follows Beijing business hours, not yours. And a model ID that silently routes to a different model is convenient for migration but bad for reproducibility, so log the model that actually served each request.

Sakana Fugu Max and Fugu Ultra v2

Sakana AI released Fugu Max and Fugu Ultra v2 on September 11. Fugu is not a single foundation model. It is a trained orchestrator that reads a query, builds an agent scaffold on the fly, and routes the work across a pool of open-weight and specialist models, including NVIDIA Nemotron models. Both new versions share one architecture and differ in what they aim at.

Fugu Max targets the best result per dollar. It costs $2 per million input tokens and $6 per million output tokens at any context length, according to DataNorth's summary of Sakana's release post and OpenRouter listings. Fugu Ultra v2 targets the hardest multi-step work. It costs $5 input and $30 output per million tokens, rising to $10 and $45 above 272K tokens of context, per AI/TLDR. Sakana's announcement does not state a context window in the material I reviewed.

Sakana's benchmark claims are its own. Sakana reports that Fugu Ultra v2 scored 48.3 on Chartography, a visual reasoning and data interpretation benchmark, against 27.3 for Claude Opus 5 and 29.5 for Claude Fable 5. Reviewers also cite a vendor-reported 74.3 on DeepSWE for Ultra v2, reached without Fable 5, Fable 5.1, or GPT-6 Astra in its model pool. For Fugu Max, Sakana claims the best overall score on six of ten benchmarks against models in a similar price range, including Terminal Bench 2.1 and GPQA Diamond. DataNorth notes that it is unclear where Sakana sourced the competitor scores this time.

Availability is narrower than a typical API. Both models run through Sakana's OpenAI-compatible API. There are no open weights, and MarkTechPost reports that Sakana does not offer the service in the EU or EEA. Existing Fugu users switch tiers with a one-line parameter change.

What it changes for practitioners: Fugu is a bet that routing beats scale. Sakana also pitches it as insurance against vendor lock-in and revoked API access, since the model pool is swappable. That pitch has real appeal after a summer of access changes across the industry. The trade-off is observability. When an orchestrator picks the model, you need its traces to explain a bad answer, so ask for them before you put Fugu in a production path.

Atria Dawn Preview

The quietest big release of the week came from the Shanghai Artificial Intelligence Laboratory. Atria Dawn Preview is an agentic model built on the 744B-parameter mixture-of-experts GLM-5.2 foundation model. It targets research and engineering work that needs continuous tool use and multi-step execution. The model card describes a full loop: problem analysis, solution design, tool use, code, experiment runs, result analysis, and failure recovery.

The release order was unusual. AI Weekly reports that the Hugging Face repository went live on September 11, with an FP8 checkpoint on September 12, and no blog post or pricing at the time. The weights are MIT-licensed. A formal press release followed on September 15, describing Atria Dawn as an open-source model for long-horizon research agents that turns a published method into runnable experiments, reproducible metrics, and a report that traces conclusions to evidence. The model works inside a control framework and an experimental environment, checks whether its code runs and whether experiments hit their targets, and revises its plan from that feedback.

No first-party per-token price was published. Some routing gateways already list the model, so any price you see comes from a third party. I found no context window or independent benchmark score in the primary materials.

What it changes for practitioners: Atria Dawn is the second open research-agent release in two weeks built on a Chinese base model and published weights-first. The release-before-paper pattern shortens the window between a model existing and a model sitting in someone's pipeline. If your team pulls open weights into production, put an evaluation gate in front of new checkpoints that does not depend on the lab's own report.

GPT-Live-1 comes to the API

OpenAI brought GPT-Live-1 to the API on September 10. It is a full-duplex voice model that listens and speaks at the same time, so it handles interruptions, pauses, and backchannels without the handoff delays of a speech-to-text, LLM, text-to-speech chain. It delegates deeper reasoning and tool calls to a backend model you choose, such as GPT-6 Astra or a third-party model.

Pricing is $0.05 per minute for the voice layer, billed per second, with backend model and tool usage billed separately. OpenAI added 12 new real-time voices. The model provides native transcripts, keyword biasing, and explicit turn detection, and it connects over WebRTC, WebSockets, or telephony and SIP.

The benchmarks are OpenAI's. OpenAI's developer community post says GPT-Live-1 paired with GPT-6 Astra at medium reasoning effort completed 83.6 percent of Tau3 tasks on the first attempt, against 45.7 percent for GPT-Realtime-2.1. The same pairing scored 38.1 percent on TauBanking. OpenAI also claims a 30-point gain on Full Duplex Bench over GPT-Realtime-2.1. One customer quoted in the launch said switching from a cascaded build removed 23,000 lines of code.

What it changes for practitioners: Voice agents now follow the same split as coding agents: a fast front end for the conversation and a slower, smarter back end for the work. OpenAI notes that interrupting speech does not automatically cancel backend work, so your application owns task state and cancellation.

Retirements and deprecations to track

Two dated changes landed this week. OpenAI announced that GPT-5.5 retires from ChatGPT, ChatGPT Work, and Codex on October 14, 2026, across all plans. The API is not affected. Codex users who sign in with ChatGPT should switch to gpt-5.6-sol and update saved settings, custom agents, scheduled tasks, and scripts before that date. Separately, OpenAI deprecated gpt-5.4-cyber on September 11, with removal from the API on October 1, 2026, and gpt-5.6-cyber as the replacement.

OpenAI also retired automatic switching from Instant to Thinking for ChatGPT Plus and Pro users and removed the Higher intelligence setting on the web. Users can still pick a reasoning option manually.

No frontier lab shipped a new flagship this week. After Claude Fable 5.1, Gemini 3.8 Flash, Muse Spark 1.3, and GPT-6 Astra in the first three days of September, a quieter stretch was expected. Google's Gemini 3.5 Pro is still announced without a date.

This week's releases at a glance

Two patterns stand out in that table. First, the two open-weight releases both carry the MIT license, which puts almost no restriction on commercial use. Second, the two Sakana tiers show where the premium sits now. The capability tier costs five times as much on output as the cost tier, on the same API, and the only difference is how hard the orchestrator works. That is the same trade every agent platform is now exposing as a setting, whether it is called effort, tier, or mode.

A note on sourcing. Where a lab's own post did not state a price or context window, this issue cites independent trackers that read the providers' pricing pages. Check the provider documentation before you commit a budget, because several of these rate cards changed more than once in September.

Tooling: OpenAI Launches the Agents API

OpenAI Agents API and hosted sandboxes

OpenAI introduced the Agents API in public beta on September 10. It packages the agent loop behind Codex as a managed service. OpenAI runs the agent loop on its own infrastructure and handles orchestration, long-running sessions, and context management. You define the agent's capabilities and choose where it runs code and works with files.

There is no extra fee for the Agents API itself. You pay for the tokens and tools your agents use at standard rates.

Sandboxes are the other half of the launch. You can bring your own, run in your VPC, or use a partner. OpenAI lists first-class integrations with Blaxel AI, Cloudflare, Daytona, DigitalOcean, E2B, Modal, Oracle Cloud, Runloop AI, and Vercel. OpenAI also launched its own hosted sandboxes, where agents run code, work with files, and produce artifacts. You supply files, install packages, and add skills and plugins, and OpenAI provisions the environment.

The Codex Python SDK moved in step. Version 0.154.0, released September 11, adds max and ultra reasoning-effort values. It also adds ExternalMessage to synchronous and asynchronous run() and turn() calls, so external content can start a turn or join an active one with tool-level authority but without granting user authorization. The release adds include_turns on resume and fork, plus a per-turn service tier.

Check the migrations before you upgrade. HookMetadata now wraps its handler in .root, so hook.command becomes hook.root.command. Some notifications now have typed payloads. And turn handles that attach late only receive events from their attachment point, so collected results can be partial. Custom codex_bin overrides need CLI 0.151.0 or newer for the new features.

OpenAI's Data agent and ChatGPT for Financial Services

OpenAI also introduced a Data agent in ChatGPT Work on September 10. It connects to approved sources including Amazon Redshift, Datadog, Google BigQuery, ClickHouse, Databricks, MongoDB, and Snowflake. It pulls business definitions, metric logic, and relationships from semantic layers and trusted sources such as dbt, GitHub, Databricks Genie Ontology, Snowflake Horizon, and BI dashboards. Queries run with the connected account's existing permissions, including table, row, and column restrictions. The agent builds shareable interactive dashboards and can work inside Omni, Oracle BI, Power BI, Sigma, Tableau, and ThoughtSpot. OpenAI says nearly all of its own product team and over two-thirds of its go-to-market organization use data agents internally.

The same day, OpenAI launched ChatGPT for Financial Services, a tailored ChatGPT Work experience built with design partners Morgan Stanley and Evercore. It includes premium data from Daloopa, PitchBook, LSEG News, and Crunchbase, indexed and hosted by OpenAI, with granular citations back to tables and passages. OpenAI says it tuned the reliability of popular financial MCP connectors, including S&P Global and FactSet, through automated evaluation. Admins can publish Excel, Word, and PowerPoint templates. OpenAI reports GPT-6 Astra at 69.9 percent on OfficeQA Pro against 60.2 percent for GPT-5.6 Sol, a vendor-reported number.

Why this matters for data teams: The pattern here is worth noting. The Data agent does not ask analysts to move data. It connects to where data already lives, respects existing access controls, and reads meaning from the semantic layer. That makes the semantic layer and the governed catalog the most important parts of an agent-ready data stack. Teams with clean metric definitions will get good answers. Teams without them will get confident wrong ones.

Claude Code: plugin evals, effort caps, and gateway headers

Anthropic's Claude Code shipped versions 2.1.265 through 2.1.273 across the week, and the week 37 summary calls out two features. The first is claude plugin eval, added in 2.1.269. It runs a plugin against a suite of test cases, scores the results, and by default reruns each case without the plugin so you can see what the plugin actually contributes. claude plugin eval init interviews you about what a good result looks like, proposes test cases and checks, and writes the files. Anthropic notes that every run and every model-judged check is a real model call on your account.

The second feature lets you pop Claude Code Desktop panes, such as the diff or the terminal, into their own windows.

The changelog is where the admin controls live. Version 2.1.267 added a maxEffortLevel setting, at the top level or per model, that caps effort on every provider, including Bedrock, Vertex, and Foundry. Version 2.1.269 added CLAUDE_CODE_WORKFLOW_MAX_CONCURRENT_AGENTS, which accepts values from 1 to 256, and an option to tag OpenTelemetry metrics with repository attributes. Version 2.1.271 added per-command allowed_domains for Bash, PowerShell, and Monitor in sandboxed auto mode, so each command opens only the hosts it needs. It also added an omitClaudeMd flag for subagents and a modelPricing multiplier of up to 10 for internal chargeback rates.

Version 2.1.273 added opt-in request headers for LLM gateways, enabled with CLAUDE_CODE_GATEWAY_HINT_HEADERS=1. They tell a gateway the request class, agent type, previous tool durations, and whether context was compacted. It also changed auto mode on Bedrock, Vertex, and Foundry to use the local safety classifier by default, with CLAUDE_CODE_AUTO_MODE_SERVER=1 to opt back into the server-side classifier.

A long list of fixes targeted prompt-cache reuse. Several releases fixed cases where resuming a session, switching models, or reconnecting an MCP server rewrote the tool list or system prompt prefix and forced a full cache rewrite. With cache reads now priced far below fresh input across the industry, these fixes translate directly into lower bills for long sessions. Version 2.1.273 also fixed auto-compaction triggering at roughly half the real context window when advisor-tool turns were involved.

Claude Managed Agents and on-demand compaction

On the platform side, Anthropic's release notes list two changes. On September 10, Claude Managed Agents permission policies gained an auto mode. The server evaluates each agent or MCP tool call and runs it, denies it, or pauses for approval, and events now report how each call was evaluated. The ant CLI added ant beta:sessions connect, which attaches your terminal to a live Managed Agents session so you can follow it, send messages, and approve or deny waiting tool calls.

On September 14, the Messages API gained on-demand conversation compaction in beta, behind the compact-2026-09-04 header. You send a top-level compaction parameter, and the API returns a signed compaction block summarizing the messages you sent. On later requests, you send that block in place of those messages. You decide when to compact, the request can run in the background, and you can keep recent turns word for word. On models with preserved thinking, the thinking in kept turns stays valid.

Why this matters: Compaction used to be something each agent framework implemented its own way. A signed, server-generated summary block makes it a first-class API object that any client can store and replay. That is a quiet but real step toward portable agent state.

Anthropic's vertical plugins

Anthropic also shipped three vertical launches. Claude for Financial Advisors arrived September 14 as a Cowork plugin with connectors to Addepar, BlackRock, Charles Schwab, Envestnet, iCapital, Orion, SS&C Black Diamond, Wealthbox, Wealth.com, Vanguard, and Zocks, plus skills for meeting prep, rebalance review, estate and tax briefs, and compliance screening against the SEC Marketing Rule. On September 15, Claude for Small Business expanded to 43 workflows and 27 new integrations, including Shopify, Salesforce, Xero, Gusto, Square, Stripe, and Zapier. Anthropic says the plugin has been installed more than 900,000 times since May. Every workflow starts in approval mode. Also on September 15, Anthropic launched Salesforce in Claude in beta with 37 pre-built sales skills.

Anthropic also launched smart reports in beta for Claude Enterprise on September 10. They analyze how a team uses Claude, what the work costs, where sessions run into friction, and which repeated patterns are worth packaging as shared skills.

GitHub Copilot: cost tiers, ensemble review, and HydraFusion

GitHub had a busy week. On September 14, it added three tiers to Copilot's auto model selection: efficiency, balance, and intelligence. All three draw from the same model set. Auto still evaluates each prompt, so a simple docstring request can land on a small model even in intelligence mode. Billing follows the model auto selects, and paid subscribers keep a 10 percent discount on usage billed through auto. The tiers are rolling out in VS Code, Copilot CLI, and the GitHub Copilot app.

On September 11, GitHub updated Copilot code review. Copilot now resolves its own comments when a later commit addresses them, writes commit messages when you apply its suggestions, and validates code with the full set of shell tools from the Copilot SDK behind the agent firewall. The Lite effort level now uses an ensemble of agents. GitHub reports that the ensemble raised addressed comments per review by 47 percent for high-severity findings, 31 percent for medium, and 11 percent for low, while cutting review cost by about 8 percent.

The September 7 weekly release post, published September 10, introduced Project HydraFusion in the Copilot CLI's /experimental menu. HydraFusion routes each task between local, cloud, and compound models to balance performance, cost, and latency, and you select it like any other model. The same post covered Jira integration in the Copilot app, scheduled agent automations and an experimental voice mode in VS Code 1.137, and centrally managed sandbox policies for Copilot in JetBrains.

The Copilot CLI shipped several builds. Version 1.0.84-5 added session and memory import commands for a semantic JSONL interchange format, moved command parsing to a Rust grammar, and made /usage show per-model AI Credit consumption. Version 1.0.84-6 added a /config screen and network allow and deny rules in /sandbox, and fixed a bug where MCP tools with certain boolean schemas caused 400 errors on Gemini. It also fixed COPILOT_ALLOW_ALL so that falsey values disable automatic tool approval instead of enabling it. That last fix is worth a second look if you set that variable in CI.

GitHub also added VS Code Agents window metrics to Copilot usage reports and released REST APIs in public preview for enabling AI Scan for pull requests at the organization and repository levels.

The pattern across vendors

Put these launches side by side and the direction is plain. OpenAI's Agents API, Anthropic's Managed Agents auto policies, and GitHub's HydraFusion and auto tiers all move two decisions from the developer to the platform: where the agent runs and which model does each step. In exchange, every vendor is adding controls. Effort caps, per-command network allowlists, server-evaluated tool calls, usage reports by model, and plugin evals all exist so an administrator can see and bound what the platform decides.

That is the right trade for most teams. It also means your cost and quality now depend on routing logic you did not write. Measure it. Plugin evals, per-model usage reports, and gateway hint headers are the instruments. Use them before you trust the defaults.

Standards: A Quiet Week for Protocols, a Busy One for Formats

No new MCP or A2A release

There was no new Model Context Protocol specification release this week and no new Agent2Agent (A2A) release. The latest MCP spec is still the 2026-07-28 version, which made the protocol stateless, and the most recent MCP blog post is the updated roadmap from August 22. A2A's latest milestone was its acceptance as a Growth Stage project at the Agentic AI Foundation (AAIF) in late August. If you track these specs, the next big date is AGNTCon+MCPCon, which the AAIF lists for October 22 and 23 in San Jose.

A quiet spec week is not a quiet implementation week. MCP fixes showed up across this week's tooling releases. The Copilot CLI now notifies MCP servers when a tool call is cancelled, requests extra OAuth scopes when needed, and fixed a schema bug that broke some MCP tools on Gemini. Claude Code added a notification when an MCP server disconnects and reconnection gives up, and fixed MCP OAuth client registration bugs. OpenAI's financial services launch describes automated evaluation to raise the reliability of popular MCP connectors. The protocol is stable enough that the work has shifted to making clients and servers behave well at the edges. For teams running MCP servers in production, that is good news. Cancellation, re-authentication, and disconnect handling are where real deployments break, and the major clients are now fixing those paths release by release instead of waiting on the spec.

Portable agent state is becoming a format question

Two tooling changes this week point at a standards gap. GitHub's Copilot CLI added import commands for a semantic JSONL interchange format covering sessions and memory. Anthropic's Messages API now returns a signed compaction block that stands in for earlier conversation turns. OpenAI's Codex SDK added history selection on resume and fork.

Each vendor is defining how agent sessions, memory, and summaries get stored and replayed. None of these are shared formats yet. MCP standardized how agents reach tools. A2A standardized how agents talk to each other. The next interoperability fight is over how an agent's accumulated state moves between tools and vendors. Watch for proposals in that space.

Open semantic models: Apache Ossie

The most concrete standards news for data teams came from the Apache Ossie (incubating) dev list, which covers an open specification for semantic models. Microsoft's Power BI group asked the community for guidance on announcing its commitment to Ossie at the upcoming Fabric Community Conference. The group plans to build a converter between Power BI semantic models and Ossie and contribute it to the project.

Ossie also moved on its core format. Yufei Gu proposed that each Ossie document hold exactly one semantic model, with its fields at the root, and opened a follow-up PR to update the converters. Jean-Baptiste Onofré confirmed that the first release, 0.3.0, will focus on release mechanics and a full legal check. And Marco Ciavarella proposed a dedicated channel for field reports written by coding agents that build with the spec.

This connects directly to the tooling section. OpenAI's Data agent reads meaning from semantic layers. Every agent that answers business questions needs metric definitions it can trust. An open semantic model format that BI tools, catalogs, and agents all read is what makes those definitions portable instead of locked inside one vendor's product.

Data format standards: Parquet 2.14 and the versioning debate

Parquet had a significant standards week too. The Parquet Format 2.14.0 release vote passed on September 11, bringing the ALP floating point encoding and the FILE logical type into the spec. A separate vote approved Extended Precision Nanosecond Timestamps.

The larger debate is about how readers should handle unsupported format versions. Ryan Blue, Fokko Driesprong, Xiening Dai, and Kurtis Wright favor a strict rule: a reader must fail on a file written with a format version it does not support. Andrew Lamb and Will Edwards argued for letting readers attempt the read, as most do today. The strict option makes it easier to ship breaking improvements such as a lighter footer. It also means every engine in a stack has to upgrade its Parquet reader before writers turn on new features.

Parquet is the storage layer under most AI training data pipelines, feature stores, and lakehouse tables. How it versions is an AI infrastructure question as much as a data engineering one.

Iceberg REST catalog and multimodal access

On the Apache Iceberg list, Sung Yun opened a discussion about access delegation for the proposed FILE type, which lets a table column reference external objects such as images and documents. The use case is multimodal inference: a service outside the query engine needs to fetch those objects. The proposal adds client-requested pre-signed URLs to the REST catalog spec. Daniel Weeks and Prashant Singh pushed to align it with existing pre-signed URL work before anything lands. This is the open table format community designing for AI workloads directly, and it belongs on the radar of anyone building retrieval over files in a lakehouse.

Infrastructure: Memory Is Still the Wall

Intel says the memory crunch gets worse in 2027

The clearest infrastructure signal of the week came from the AI Infra Summit in Santa Clara, which runs September 15 to 17. Intel CEO Lip-Bu Tan warned there that the memory shortage will be worse in 2027 than in 2026. Fudzilla reports that Tan said memory prices have risen five to seven times, and that memory now makes up 70 to 80 percent of the component cost of some low-end phones and laptops. Invezz reports that Tan also flagged electricity and cooling as emerging constraints.

A separate report from Aroged, citing a Splunk broadcast appearance, says Tan acknowledged Intel cannot meet more than half of demand for its processors, does not expect memory to improve until 2028, and plans to start 14A production in the first quarter of 2027. These are secondhand accounts of spoken remarks, so treat the exact figures with some care. The direction is consistent across every report.

The cause is structural. Memory makers are steering wafer capacity toward HBM for AI accelerators, which leaves less for commodity DRAM and NAND. DIGITIMES' weekly roundup for September 7 to 13 describes HBM4 and server demand straining DRAM and NAND supply, alongside Intel CPU price increases.

What it means for AI teams: Memory is now the price floor for both training and inference. It shapes GPU availability, server costs, and even the laptops your developers use. Plan 2027 hardware budgets with higher memory costs baked in, and treat any software change that cuts memory use as a cost reduction.

Software is attacking the memory problem directly

That last point is exactly what DeepSeek did this week. V4.1 Flash's KV cache needs one quarter of the HBM and one eighth of the SSD storage of the previous generation. The KV cache holds the attention state for every token in context, and on million-token contexts it often consumes more accelerator memory than the model weights. Shrinking it lets a provider serve more concurrent requests on the same GPUs, which is how DeepSeek was able to cut prices during a memory shortage. DeepSeek also invited teams planning deployments of 2,000 GPUs plus a storage cluster to contact it directly.

The architecture choice works the same way. Activating 8B parameters for input and 16B for output means prefill, the input-heavy half of agent workloads, runs on a much smaller slice of the model. Expect other labs to publish similar input and output splits as agentic workloads dominate inference traffic.

OpenAI's GPT-Live-1 is a different angle on the same economics. A single full-duplex model replaces a three-stage speech pipeline, which cuts both latency and the number of models held in memory per conversation. At $0.05 per minute for the voice layer, it pushes the heavy reasoning to a backend model that only runs when the conversation needs it.

Columnar formats keep getting smaller

The storage side of AI infrastructure also moved. Apache Arrow Rust 60.0.0 passed its release vote on September 15. Release manager Andrew Lamb said arrow-rs is the first Parquet implementation to include the new ALP encoding for floating point data. The release also adds new Parquet PageIndex structures and many performance improvements. The crates are live on crates.io.

Float and double columns are everywhere in AI data: embeddings stored as arrays, model features, sensor readings, and evaluation metrics. General-purpose encodings compress them poorly. ALP targets exactly that data, and it now ships in the Rust parquet crate that DataFusion, iceberg-rust, and many other Rust engines depend on. Implementations in C++, Java, and Go are in review, according to the Parquet community sync notes. The same notes record progress on a vector logical type for Parquet, which targets embedding storage directly.

Accelerators at the AI Infra Summit

NVIDIA's AI Infra Summit event page promotes a keynote from Ian Buck, NVIDIA's VP of Hyperscale and HPC Computing, on infrastructure for agentic AI. The page describes Groq 3 LPX as an interactive inference accelerator that extends the Vera Rubin platform, aimed at fast token generation for responsive agent systems. Converge Digest's summit preview frames the event around scale-up interconnect, scale-out networking, and scale-across data center links, with Google Fellow Dave Patterson, Ian Buck, and Lip-Bu Tan on the main stage. The summit runs through September 17, so expect more detailed announcements to surface in next week's coverage.

Practitioner Takeaways

Run the cache math before you switch models

Rate cards hide the number that matters most for agents. Here is a worked example using DeepSeek's published peak rates. Take one long agent run that reads 10 million cached input tokens, 1 million fresh input tokens, and writes 200,000 output tokens.

On V4 Flash at $0.014 cached, $0.44 input, and $1.32 output per million, that run costs $0.14 plus $0.44 plus $0.264, or about $0.84. On V4.1 Flash at $0.006, $0.30, and $1.20, it costs $0.06 plus $0.30 plus $0.24, or $0.60. That is a 29 percent cut, and more than a third of it comes from the cache line alone.

Now change the shape. A chat workload with little caching and a lot of output saves far less, because output only dropped 9 percent. The lesson applies to every vendor: pull a week of real traffic, split it into cached input, fresh input, and output, and price that mix. Headline input prices tell you very little about an agent bill.

Five things to do this week

Audit pinned model IDs. DeepSeek now routes old IDs to a new model, and OpenAI has two retirement dates in the next four weeks. Log which model actually served each request so a silent swap shows up in your dashboards.
Set effort and concurrency caps. Claude Code's maxEffortLevel and workflow concurrency limit, and Codex's new max and ultra effort values, give you both more power and more ways to overspend. Decide your ceilings on purpose.
Evaluate plugins and skills like code. claude plugin eval runs each case with and without the plugin. Use that comparison to delete plugins that add tokens without adding quality.
Check your CI environment variables. The Copilot CLI fix for COPILOT_ALLOW_ALL changed how falsey values behave. If your pipelines set it, confirm the behavior you expect.
Invest in your semantic layer. OpenAI's Data agent, Apache Ossie, and Polaris' new semantic model privileges all point the same way. Agents answer business questions only as well as your metric definitions allow.

Where the week leaves the stack

Three layers moved at once. Models got cheaper per task through architecture, not just discounts. Platforms took over the agent loop and added the controls to supervise it. And the data layer kept adding the pieces agents need: compact float encodings, vector types, governed semantic models, and catalog-level access for multimodal files. None of those layers works well alone. The teams that get the most from this week's releases will be the ones that treat model choice, agent platform, and data architecture as one design problem.

What to Watch Next Week

DeepSeek V4 Pro routing. Confirm whether deepseek-v4-pro requests are served by V4 Pro or V4.1 Flash, and watch for V4.1 Pro.
Independent benchmarks. Look for third-party scores on DeepSeek V4.1 Flash, Fugu Ultra v2, and Atria Dawn Preview. Every number in this issue for those models is vendor-reported.
Gemini 3.5 Pro. Google has announced it with no date or price.
AI Infra Summit follow-ups. Detailed accelerator, memory, and networking announcements from Santa Clara.
Parquet versioning. A decision on strict reader version checks will shape how new footer and encoding features roll out.
Deadlines. gpt-5.4-cyber leaves the OpenAI API on October 1, and GPT-5.5 leaves ChatGPT and Codex on October 14.

Resources to Go Further

The tools change every week, but the fundamentals of data, lakehouse architecture, and agentic AI hold steady. I have written books on all of it, from Apache Iceberg and Apache Polaris to AI-assisted development and AI agents for data work. You can find every title at books.alexmerced.com.

The Open Lakehouse Explained, Then Built on Your Laptop with Dremio and MinIO

Alex Merced — Thu, 10 Sep 2026 19:56:45 +0000

Most people learn the open lakehouse backwards. They read five vendor pages, collect a stack of Apache project names, and still cannot answer a basic question: when I run a query, what does each piece actually do?

That gap is fixable in about an hour. The five projects that define the open lakehouse each own one layer of the problem, and once you see the layers separately, the architecture stops being a diagram and becomes obvious. Apache Parquet stores bytes on disk. Apache Iceberg turns a pile of those files into a table. Apache Polaris keeps track of which tables exist and who can touch them. Apache Arrow moves the data through memory and across the wire. Apache Ossie describes what the columns mean in business terms.

The second half of this article is a lab. You will run two containers on your laptop, one for object storage and one for a query engine, wire them together, and write a real Apache Iceberg table into an S3-compatible bucket. Then you will look at the files that landed and see the format layers with your own eyes. No cloud account, no credit card, no Spark cluster.

I work at Dremio, and Dremio is the query engine in the lab. I picked it because it runs in a single container and needs no external metastore, which keeps the exercise short. The concepts transfer to any engine.

Why the Lakehouse Split Into Layers at All

For thirty years a database was one product. Storage format, table metadata, catalog, query planner, and execution engine all shipped together and none of them were separable. That design has real advantages. The engine knows exactly how the bytes are laid out and can optimize against its own assumptions.

The cost shows up when you have more than one workload. Your BI tool wants SQL. Your data scientists want Python. Your machine learning pipeline wants to read raw files. Each of those tools has its own preferred engine, and a closed warehouse gives you exactly one door in, priced per query.

Teams solved this in the 2010s by dumping files into object storage and pointing many engines at the same directory. That worked for read-only analytics and fell apart everywhere else. Two writers touching the same directory produced corrupt results. A partially written job left half a dataset visible to readers. Renaming a column meant rewriting everything. There was no such thing as a transaction.

The lakehouse is the answer to that failure. Keep the cheap shared storage, then add back the guarantees a database gave you, defined as open specifications instead of product internals. Each guarantee lives in its own layer, and each layer has a written spec that anyone can implement.

The result is a stack you assemble rather than buy. Storage from one vendor, catalog from another, three engines reading the same tables at once, and no rewrite when you swap any single piece. That portability is the entire point, and it only works because the layers stay honest about their boundaries.

Apache Parquet: The File Layer

Parquet is a columnar file format. It is the oldest project in this group, donated to the Apache Software Foundation in 2015, and it is the default storage format for nearly every analytics system built since.

A CSV file stores data by row. All of order 1, then all of order 2. To sum one column across ten million rows, you read all ten million rows in full. Parquet flips that. It groups rows into chunks called row groups, and inside each row group it stores each column contiguously. To sum one column, you read that column and skip the rest.

Three properties fall out of that layout, and all three matter more than people expect.

Compression gets far better. A column holds one data type with repeated values, so run-length encoding and dictionary encoding do real work. A column of country codes with 200 distinct values compresses to almost nothing. Mixed row data never compresses that well.

Column pruning saves I/O. A query touching 3 columns out of 80 reads about 4 percent of the file. On object storage, where you pay per byte transferred and latency dominates, that is the difference between a two-second query and a two-minute one.

Predicate pushdown skips whole chunks. Each row group carries footer statistics with the minimum and maximum value per column. A query filtering on order_date > '2026-01-01' reads the footer, sees that a row group tops out at 2025-06-30, and skips it without decompressing a single page.

What Parquet does not give you is a table. A Parquet file knows its own schema and its own statistics. It has no idea that 4,000 sibling files in the same prefix belong to the same logical dataset, no notion of a transaction, and no way to express that three of those files were replaced by one this morning. That is the next layer's job.

Apache Iceberg: The Table Layer

Iceberg is a specification for describing a table as a set of files, plus enough metadata to make changes to that set atomic. It came out of Netflix, was donated to the ASF in 2018, and graduated to a top-level project in 2020. The current stable Java library line is 1.11, released in mid-2026, and it implements format version 3 of the specification.

The distinction between library version and format version confuses people constantly. The library version, 1.11, is a piece of software. The format version, 3, is the on-disk contract that any implementation in any language has to honor. When a vendor says they support Iceberg, ask which format version and which operations.

Here is the structure, from the bottom up.

Data files. Parquet files holding the rows. Iceberg supports ORC and Avro too, and Parquet is what nearly everyone uses.

Manifest files. An Avro file listing data files, each with its partition values, row count, and per-column min/max statistics. One manifest describes a batch of data files.

Manifest lists. An Avro file listing the manifests that make up one snapshot, with partition range summaries for each.

Metadata files. A JSON file holding the table schema, the partition specification, the sort order, table properties, and the full history of snapshots. Every write produces a new metadata file.

The catalog pointer. One tiny piece of mutable state: which metadata file is current. Everything below it is immutable.

That last point carries all the weight. Because every file below the pointer never changes, a writer builds a complete new version of the table off to the side, then swaps the pointer in one atomic operation. Readers that started before the swap keep reading the old snapshot and see a consistent view. Readers that start after see the new one. Nobody ever sees half a commit.

Once you have immutable snapshots, several features come free rather than being bolted on.

Time travel is just reading an older metadata file. SELECT * FROM sales AT SNAPSHOT '8234...' resolves a snapshot ID and reads the file list from that point in history.

Rollback is repointing the catalog at a previous metadata file. A bad load at 3 a.m. gets undone in one statement instead of a restore from backup.

Schema evolution works because Iceberg tracks columns by a unique integer ID, not by name or position. Rename a column and the ID stays the same, so old data files still resolve correctly. Add a column and old files report null for it. Drop a column and nothing gets rewritten.

Hidden partitioning removes the worst footgun of the Hive era. In Hive, a table partitioned by day required every query to filter on the partition column by name, and a query filtering on the raw timestamp scanned everything. Iceberg records the partition as a transform of a source column, so a filter on order_ts automatically prunes partitions defined as day(order_ts). Users stop needing to know the physical layout.

One thing Iceberg deliberately leaves out is where that catalog pointer lives. The specification defines the file formats and the commit protocol, then hands the pointer problem to a pluggable catalog. Implementations include the Hive Metastore, AWS Glue, a plain filesystem-based catalog, and the REST catalog protocol that Polaris implements. A table created through one catalog is not readable through another, because each catalog holds its own record of the current metadata file. Migrating between catalogs is a real project rather than a config change, which is why the catalog choice deserves more thought than the file format choice.

Format version 3 added deletion vectors, which replace v2 position delete files with compact bitmaps. Updating a handful of rows no longer forces a rewrite of the data files that contain them, and merge-on-read queries get faster. V3 also added a variant type for semi-structured JSON and mandatory row lineage tracking.

Apache Polaris: The Catalog Layer

The catalog is the smallest component in the stack and the one that decides how open your architecture really is.

An Iceberg catalog does one job well: map a table name to the location of its current metadata file, and swap that pointer atomically when a writer commits. Add a second job on top and you have a governance boundary, because every engine has to ask the catalog where a table lives before it can read the table.

Apache Polaris is an open source implementation of the Iceberg REST Catalog specification. It was co-created by Dremio and Snowflake, donated to the Apache Software Foundation, and graduated to a top-level project on February 18, 2026.

Two things make the REST catalog spec matter more than the older catalog options.

First, it standardizes the protocol instead of the implementation. Older catalog choices bound you to a client library. A Hive Metastore catalog meant every engine needed the Hive client and Thrift. A Glue catalog meant AWS SDK calls. The REST spec turns catalog access into HTTP with a documented JSON contract, so an engine written in Rust or Python talks to the same catalog as one written in Java without shipping anyone else's client.

Second, it moves commit logic to the server. In the older model, the client library built the new metadata and performed the atomic swap. Every engine had to implement that correctly, and subtle differences caused real corruption. With REST, the client sends the requested change and the server does the commit. One implementation of the hard part, shared by everyone.

Polaris adds credential vending on top of that, and this is the feature worth understanding even if you never deploy Polaris. Instead of giving each engine long-lived storage keys, the catalog holds the storage credentials. An engine asks for a table, the catalog checks the caller's permissions, and it returns a short-lived, narrowly scoped storage credential valid for that table's location only. Access control lives in one place instead of being duplicated across every engine's configuration and every bucket policy.

Polaris organizes objects into catalogs, namespaces, and tables, with role-based access control on each level. It runs internal catalogs, where Polaris manages the tables directly, and external catalogs, where it federates to another catalog implementation.

The lab below does not use Polaris. Running a catalog service, a database for its state, object storage, and an engine is four containers and a lot of configuration, which is the wrong shape for a first exercise. Dremio ships with an internal Iceberg catalog that handles the pointer for us, and the file layout you inspect at the end is identical either way.

Apache Arrow: The In-Memory Layer

Parquet is how data rests. Arrow is how data moves.

Apache Arrow defines a standard columnar layout for data in memory, plus a serialization format for sending that layout over a network without changing it. It was co-created by Jacques Nadeau, now the CTO at Dremio, and it has become the connective tissue between analytics tools.

The problem Arrow solves is serialization tax. Before Arrow, moving a result set from a Java engine to a Python client meant converting a Java object layout to a wire format, sending it, then parsing it into a pandas layout. On large results, that conversion cost more CPU than the query. Every hop between two systems paid the same tax.

Arrow removes the conversion by making the in-memory layout and the wire layout the same thing. A record batch on the server is copied to the socket, and the client points at the received bytes as a valid Arrow buffer. No parse step. Tools that both speak Arrow exchange data at close to memory bandwidth.

The layout is also built for modern CPUs. Values in a column sit in a contiguous buffer with a separate validity bitmap for nulls, which lets a query engine process 8 or 16 values per instruction using SIMD registers rather than one value per loop iteration. Vectorized execution is what makes columnar engines fast, and Arrow is the layout that makes vectorized execution straightforward to write.

Two adjacent pieces are worth naming because they show up in real deployments. Arrow Flight is a gRPC-based protocol for moving Arrow record batches between processes, and Arrow Flight SQL adds a database-style interface on top, so a client submits SQL and receives Arrow batches directly. Dremio exposes Flight SQL on port 32010, which is one of the ports you will map in the lab.

Parquet and Arrow are frequently confused because both are columnar. Parquet optimizes for size on disk with heavy encoding and compression. Arrow optimizes for CPU access speed with a fixed, predictable layout and no decoding step. An engine reads Parquet, decodes it into Arrow buffers, and works from there.

Apache Ossie: The Semantic Layer

The newest project in this group solves the problem that survives after all the technical layers work perfectly.

Apache Ossie is an open specification for semantic layers and ontologies. It started life as Open Semantic Interchange, was renamed to avoid a clash with the Open Source Initiative acronym, and entered the Apache Incubator on June 22, 2026. It is a specification project rather than a runtime, and it is still incubating, so treat it as a direction rather than a dependency.

The problem is definition drift. "Monthly Active Users" exists in the CRM, in the warehouse, and in three BI dashboards, and the four definitions disagree on whether a user who logged in through the mobile app on the last day of the month counts. Every organization past a certain size has this problem, and it never gets solved by fixing one dashboard, because the definitions are trapped inside whichever tool created them.

Ossie defines a vendor-neutral YAML format for expressing metrics, dimensions, relationships, and broader business concepts. A BI platform, a query engine, or an AI agent reads the same definition file and computes the same number. The definition moves with the data instead of living in a proprietary model file.

The agent angle is the reason this project got funded and staffed now rather than five years ago. A human analyst who gets a strange number investigates. An AI agent that gets a strange number writes it into a report with confidence. Giving agents a machine-readable, governed definition of what a metric means is the difference between an agent that helps and one that generates plausible nonsense at scale. Contributors include Snowflake, Salesforce, Databricks, dbt Labs, RelationalAI, GoodData, and Honeydew, which tells you the industry treats this as shared infrastructure rather than a competitive surface.

Ossie sits above the lab you are about to run. You will not configure it. Knowing the layer exists explains why "semantic layer" keeps appearing in lakehouse conversations that are otherwise about file formats.

How the Five Layers Cooperate in One Query

Walk a single query through the stack and the division of labor becomes concrete.

An analyst runs SELECT region, SUM(amount) FROM sales WHERE order_date >= DATE '2026-01-01' GROUP BY region.

The engine parses the SQL and asks the catalog for the table named sales. The catalog checks permissions and returns the location of the current metadata file, plus a scoped storage credential if it does credential vending.

The engine reads the Iceberg metadata file, follows it to the manifest list for the current snapshot, and reads the manifests. Each manifest entry carries partition values and column statistics. The engine drops every data file whose order_date maximum falls before 2026-01-01. A table with 40,000 files becomes a scan list of 900 without a single byte of table data being read.

For each surviving Parquet file, the engine reads the footer, drops row groups whose statistics fail the filter, and requests byte ranges covering only the region, amount, and order_date columns.

Those bytes get decoded into Arrow buffers in memory. Aggregation runs vectorized over those buffers, and partial results merge across threads and nodes in the same layout.

The final result travels to the client as Arrow record batches over Flight SQL, with no serialization step.

If the organization uses Ossie, the analyst never wrote that SQL. They asked for revenue by region, and a tool translated it using the governed metric definition, producing SQL that matches what every other tool in the company produces for the same question.

Five specifications, five jobs, zero overlap. That is the design.

Held side by side, the boundaries are easy to keep straight:

The useful test for any new tool you evaluate is which of these rows it replaces and which it merely reads. A product that reads Iceberg tables somebody else wrote sits in a different category from one that creates, writes, and commits through the standard catalog API. Both say "Iceberg support" on the website.

The Lab: What You Are Building

Two containers.

MinIO provides S3-compatible object storage on your laptop. It speaks the S3 API, so anything that talks to Amazon S3 talks to it with a changed endpoint. This is your data lake.

Dremio provides the SQL engine, the Iceberg write path, and the web console. Dremio Community Edition runs as a single container with an embedded coordinator, executor, and ZooKeeper.

The exercise has four steps. Start the stack. Create a bucket. Add MinIO to Dremio as a source configured for S3 compatibility. Write, query, and evolve an Iceberg table, then look at the files it produced.

Before the compose file, one piece of honesty about MinIO. The MinIO project archived its GitHub repository on April 25, 2026, and stopped publishing new container images in October 2025. The last image published to Docker Hub is tagged RELEASE.2025-09-07T16-13-09Z, and that is the tag pinned below. It works fine for a local exercise and receives no further security patches, so do not carry this compose file into production. For production S3-compatible storage, look at MinIO's commercial AIStor line, the community fork maintained at pgsty/minio, or alternatives such as Ceph RADOS Gateway, SeaweedFS, or Garage. The Dremio configuration below is identical for any of them, because the only thing that matters is the S3 API.

Requirements: Docker Desktop, roughly 8 GB of RAM available to Docker, and about 10 GB of disk.

The Compose File, Explained Line by Line

Create a directory, save this as docker-compose.yml, and read the explanation before running it.

services:
  minio:
    image: minio/minio:RELEASE.2025-09-07T16-13-09Z
    container_name: minio
    ports:
      - "9000:9000"
      - "9001:9001"
    environment:
      MINIO_ROOT_USER: admin
      MINIO_ROOT_PASSWORD: password
    command: server /data --console-address ":9001"
    volumes:
      - minio-data:/data
    networks:
      - lakehouse

  createbucket:
    image: minio/mc:latest
    container_name: createbucket
    depends_on:
      - minio
    entrypoint: >
      /bin/sh -c "
      until mc alias set local http://minio:9000 admin password; do sleep 2; done;
      mc mb --ignore-existing local/lakehouse;
      mc ls local;
      exit 0;
      "
    networks:
      - lakehouse

  dremio:
    image: dremio/dremio-oss:latest
    container_name: dremio
    ports:
      - "9047:9047"
      - "31010:31010"
      - "32010:32010"
      - "45678:45678"
    environment:
      DREMIO_JAVA_SERVER_EXTRA_OPTS: -Dpaths.dist=file:///opt/dremio/data/dist
    volumes:
      - dremio-data:/opt/dremio/data
    depends_on:
      - minio
    networks:
      - lakehouse

volumes:
  minio-data:
  dremio-data:

networks:
  lakehouse:

Now the details that matter.

The pinned MinIO tag. Pinning an exact release keeps the exercise reproducible. latest on an archived project is a moving target you have no control over.

Two MinIO ports. Port 9000 is the S3 API endpoint. Dremio talks to that one. Port 9001 is the web object browser, which you use to look at files. In the community build, that browser is a plain object viewer with the admin features removed, which is why bucket creation happens through the client container instead of the browser.

Root credentials as access keys. MINIO_ROOT_USER and MINIO_ROOT_PASSWORD become the S3 access key and secret key. In the Dremio source configuration, admin goes in the access key field and password goes in the secret key field. Real deployments create a scoped service account with mc admin user svcacct add and never hand out root keys.

The command line. server /data tells MinIO to serve the /data path as its storage backend. --console-address ":9001" pins the browser to a fixed port. Without it, MinIO picks a random port each start and your bookmark breaks.

The createbucket service. This container runs MinIO's command-line client once and exits. The until loop retries the alias command until MinIO answers, which handles the race where the client starts before the server is listening. mc alias set registers a connection named local. mc mb --ignore-existing local/lakehouse creates the bucket and stays quiet if it already exists, so restarting the stack is safe. mc ls local prints the bucket list to the container log so you have a visible confirmation. Compose will report this container as exited, and that is correct.

Dremio's four ports. 9047 is the web console and REST API. 31010 is the legacy ODBC/JDBC port. 32010 is Arrow Flight SQL, the fast path discussed earlier. 45678 is internal node-to-node communication, needed even in single-node mode.

The DREMIO_JAVA_SERVER_EXTRA_OPTS line. This sets Dremio's distributed storage path to a directory inside the container's data volume. Dremio uses distributed storage to hold job results, uploads, and Reflections. Without setting it, Reflections stay unavailable. This one variable is the difference between a container that starts cleanly and one that complains about missing distributed storage configuration.

Named volumes. minio-data keeps your Iceberg files across restarts. dremio-data keeps Dremio's own catalog, source definitions, and user accounts. Drop these and you start over from the signup screen.

The shared network. Both containers join the lakehouse network, so Dremio reaches MinIO at the hostname minio on port 9000. This is the single detail that trips up the most people, and the next section explains why.

Start it:

docker compose up -d
docker compose logs -f dremio

Dremio takes 60 to 120 seconds to become available on a laptop. Watch for the log line saying the Dremio Daemon started. Open http://localhost:9047, fill in the first-user signup form, and pick any credentials you can remember. That account is local to your container.

Adding MinIO as a Dremio Source

In the Dremio console, click Add Source in the lower left and choose Amazon S3. MinIO is not in the list by name because it does not need to be. It speaks the S3 API, and the S3 connector handles anything that does.

Fill in the General tab:

Name: minio
Authentication: AWS Access Key
AWS Access Key: admin
AWS Access Secret: password
Encrypt connection: unchecked

Unchecking encryption is required here. MinIO in this compose file serves plain HTTP with no certificate. Leaving the box checked makes Dremio attempt TLS against an HTTP endpoint, and the source fails to save with a connection error that does not name the real cause.

Now open Advanced Options:

Enable compatibility mode: checked
Default CTAS Format: ICEBERG
Root Path: /

Compatibility mode is what tells Dremio it is talking to an S3-compatible store rather than Amazon S3 itself. It changes request signing and endpoint handling. Without it, Dremio builds requests against AWS endpoints and never reaches your container.

Default CTAS Format deserves a pause. S3 sources in Dremio default to writing Parquet, not Iceberg. A CREATE TABLE AS against a source left on the default produces a folder of Parquet files with no table metadata at all, which looks like it worked and gives you none of Iceberg's guarantees. Setting this to ICEBERG is the single most important checkbox in this exercise.

Root Path of / exposes every bucket in MinIO as a top-level folder inside the source. Setting it to /lakehouse scopes the source to one bucket, which is closer to what you do in production.

Still in Advanced Options, find Connection Properties and add three:

The endpoint value has two rules that break people constantly. It cannot include the http:// or https:// prefix, and it cannot begin with the string s3. Write minio:9000 and nothing else.

Use minio, not localhost. Inside the Dremio container, localhost means the Dremio container. The hostname minio resolves through the Docker network to the MinIO container. This mistake produces a connection refused error that reads like a MinIO problem and is not one.

Path-style access matters because the default S3 addressing scheme is virtual-hosted style, which puts the bucket into the hostname as lakehouse.minio:9000. That hostname does not exist on your Docker network. Path style produces minio:9000/lakehouse, which does.

Click Save. The source appears in the left panel, and expanding it shows the lakehouse bucket. If it does not, jump to the troubleshooting section.

Writing and Reading Iceberg Tables

Open the SQL Runner and create a table.

CREATE TABLE minio.lakehouse.orders (
  order_id     INT,
  customer     VARCHAR,
  region       VARCHAR,
  amount       DOUBLE,
  order_date   DATE
) PARTITION BY (month(order_date));

The PARTITION BY (month(order_date)) clause is Iceberg's hidden partitioning at work. You are not adding a separate month column. Iceberg records a transform of order_date, and later queries that filter on order_date get partition pruning without knowing the layout exists.

Insert some rows:

INSERT INTO minio.lakehouse.orders VALUES
  (1, 'Acme Corp',    'East',  1200.50, DATE '2026-01-15'),
  (2, 'Globex',       'West',   890.00, DATE '2026-01-22'),
  (3, 'Initech',      'East',  2340.75, DATE '2026-02-03'),
  (4, 'Umbrella Ltd', 'North', 1500.00, DATE '2026-02-18'),
  (5, 'Soylent Inc',  'West',   675.25, DATE '2026-03-07');

Query it:

SELECT region, SUM(amount) AS total, COUNT(*) AS orders
FROM minio.lakehouse.orders
GROUP BY region
ORDER BY total DESC;

Five rows is not a performance test. The point is that the write path produced a real Iceberg table, and the next few statements prove it.

Look at the snapshot history:

SELECT * FROM TABLE(table_snapshot('minio.lakehouse.orders'));

You get one row per snapshot, with a snapshot ID, a commit timestamp, the operation, the manifest list path, and a summary. Two snapshots exist so far: one from the create, one from the insert.

Now mutate the table and watch history grow:

UPDATE minio.lakehouse.orders
SET amount = 1300.00
WHERE order_id = 1;

DELETE FROM minio.lakehouse.orders
WHERE region = 'North';

INSERT INTO minio.lakehouse.orders VALUES
  (6, 'Hooli', 'East', 4100.00, DATE '2026-03-22');

Re-run the snapshot query. Every statement added a snapshot. Copy the snapshot ID from the row right after your first insert and travel back to it:

SELECT * FROM minio.lakehouse.orders
AT SNAPSHOT '<paste_snapshot_id_here>'
ORDER BY order_id;

Five rows come back, with Acme at 1200.50 and Umbrella Ltd present. The current table has neither. Nothing was restored from a backup. The old metadata file still lists the old data files, and they were never deleted.

You can travel by time as well as by snapshot:

SELECT COUNT(*) FROM minio.lakehouse.orders
AT TIMESTAMP '2026-09-10 00:00:00';

Evolve the schema and confirm nothing breaks:

ALTER TABLE minio.lakehouse.orders ADD COLUMNS (sales_rep VARCHAR);

SELECT order_id, customer, sales_rep
FROM minio.lakehouse.orders
ORDER BY order_id;

Every existing row reports null for sales_rep, and not one data file was rewritten. Iceberg added a column ID to the schema in a new metadata file. Readers resolve missing IDs as null.

Finally, compact:

OPTIMIZE TABLE minio.lakehouse.orders;

Dremio rewrites small files into larger ones and clears delete files, then commits the result as another snapshot. On a five-row table the output is trivial. On a table fed by a streaming pipeline producing thousands of small files an hour, this is the single most valuable maintenance operation in the lakehouse.

Looking at the Files You Just Created

This is the part that makes the architecture stick. Open http://localhost:9001 and sign in with admin and password. Browse into the lakehouse bucket and then into the orders folder.

You see two directories.

metadata/ holds the Iceberg metadata. Look for files ending in .metadata.json, one per commit, numbered in sequence. Open the newest one in the browser. It is readable JSON containing the schema with integer column IDs, the partition spec with the month transform, a list of snapshots, and a current-snapshot-id field. Open the first one and compare. You can see the schema before sales_rep was added.

The same folder holds snap-*.avro files, which are the manifest lists, and other .avro files, which are the manifests. Those are binary, so you will not read them in a browser, and knowing they sit between the metadata and the data is enough.

data/ holds the Parquet files, organized into subfolders named for the partition values that Iceberg's month transform produced. Each .parquet file inside is a normal Parquet file that any Parquet reader opens without knowing anything about Iceberg.

Count the metadata JSON files. Each of your statements produced one. That is the commit protocol in physical form: write new files, write a new metadata file, swap the pointer.

Try one more thing. Prefix a Parquet file path in your browser and query it directly through Dremio's S3 source as a raw file. Dremio reads it and returns the rows in it. The file is data. The table is the metadata that tells you which files are current, and that separation is the whole idea.

Seeing the Same Thing From the Command Line

The browser view is friendly and slow. The client container gives you the whole tree at once.

docker run --rm --network <your_project>_lakehouse minio/mc:latest \
  /bin/sh -c "mc alias set local http://minio:9000 admin password && \
              mc ls --recursive local/lakehouse"

Replace <your_project> with your directory name, which Compose uses as the network prefix. Run docker network ls if you are unsure.

The output lists every object with its size and timestamp. Sort it mentally into three groups. The .metadata.json files are one per commit and grow slowly. The .avro files are manifests and manifest lists. The .parquet files under partition folders hold the actual rows and account for nearly all the bytes.

Run the same command again after an INSERT and diff the two listings. New Parquet files appear, new Avro files appear, and exactly one new metadata JSON appears. Nothing that existed before was modified. That immutability is what makes the atomic pointer swap safe, and seeing it in a file listing is more convincing than reading it in a specification.

What Breaks, and How to Recognize It

These are the failures worth knowing before you hit them.

The source saves but shows no buckets. Root Path is wrong or the credentials are wrong. Confirm the bucket exists by running docker compose logs createbucket and checking that mc ls printed lakehouse.

Connection refused or a timeout on save. The endpoint is set to localhost:9000 instead of minio:9000, or the containers are not on the same network. Run docker exec -it dremio curl -I http://minio:9000 and confirm you get an HTTP response.

An SSL or handshake error. Encrypt connection is still checked. Uncheck it.

A signature mismatch error. Path-style access is not set to true, or the access key and secret do not match your compose file exactly. Both are case sensitive.

CREATE TABLE produces plain Parquet with no metadata folder. Default CTAS Format is still Parquet. Change it in Advanced Options and recreate the table.

Dremio never becomes reachable on 9047. Almost always memory. Dremio's default heap plus direct memory settings need real headroom. Raise Docker Desktop's memory allocation to 8 GB and restart the stack.

A stale metadata error after writing from another tool. Dremio caches source metadata. Run ALTER TABLE minio.lakehouse.orders REFRESH METADATA to force a re-read.

Everything works, then breaks after docker compose down -v. The -v flag deletes named volumes, which takes your Dremio account and your Iceberg data with it. Use docker compose down without the flag to stop the stack and keep state.

Two operational notes that are not errors but bite later. Iceberg never deletes old files on its own, so a table with heavy writes accumulates orphaned data files and metadata files until you run expiration and orphan cleanup. And a table with thousands of tiny files spends more time in planning than in scanning, which is what OPTIMIZE exists to fix.

Where to Take This Next

The lab uses Dremio's internal catalog, which keeps the container count at two. The next step is separating the catalog out, and that is where the architecture gets interesting.

Add a Polaris container backed by Postgres, register the same MinIO bucket as its storage location, and connect Dremio to it as an Iceberg REST catalog source. Then start a Spark container with the Iceberg runtime, point it at the same Polaris endpoint, and write a table from Spark. Query that table from Dremio without any additional configuration. Write from Dremio and read from Spark. That single exercise demonstrates the multi-engine promise better than any diagram, and it is the reason the catalog layer exists.

From there the natural extensions are PyIceberg for reading tables directly from Python without an engine, a Flight SQL client on port 32010 to see Arrow move end to end, and a look at Ossie's YAML metric definitions once the specification stabilizes past incubation.

For production, none of this compose file survives contact. You need real storage with real durability, a catalog with an actual database behind it, scoped credentials instead of root keys, TLS on every endpoint, and a maintenance schedule for compaction, snapshot expiration, and orphan file cleanup. What does survive is the mental model. Files, tables, catalog, memory, meaning. Five layers, five specifications, and any piece replaceable without rewriting the others.

Conclusion

The open lakehouse is not one technology and it is not a product category. It is a set of specifications that split apart what a database used to bundle, so that storage, table semantics, governance, execution, and business meaning each stay open and each stay swappable.

Parquet makes the bytes small and skippable. Iceberg turns files into a transactional table with history. Polaris tracks which tables exist and controls who reaches them. Arrow moves the results without paying a serialization tax. Ossie is working on making the definitions themselves portable.

You now have all five in your head and three of them running on your laptop. Break the compose file on purpose. Set the endpoint wrong and read the error. Turn off compatibility mode and watch what fails. Write a table with the CTAS format on Parquet and compare the folder to the Iceberg one. An hour of deliberate breakage teaches more about this stack than a week of architecture diagrams.

Keep Going

If this piece was useful, I have written a lot more on lakehouse architecture and the open table format ecosystem. Apache Iceberg: The Definitive Guide covers the specification and the operational side in depth, and Architecting an Apache Iceberg Lakehouse walks through the design decisions behind a full platform build. You can find every book I have written, across lakehouse architecture, Apache Iceberg, Apache Polaris, and AI, at books.alexmerced.com.

Apache Data Lakehouse Weekly: September 3-9, 2026

Alex Merced — Thu, 10 Sep 2026 12:50:44 +0000

Three things ran through the dev lists this week. Projects kept redrawing their own boundaries, with the Iceberg Rust DataFusion integration voted out of Iceberg and into DataFusion on both lists at once. Catalogs kept turning informal metadata into spec text, with labels landing in the Iceberg REST spec and tags being argued down to the encoding of a query parameter in Polaris. And format governance got a lot more explicit, with Parquet debating what a version number in a footer actually promises a reader.

Six projects, one week. Here is what happened and why it matters.

Apache Iceberg

The largest thread of the week was not a spec argument at all. Sung Yun opened the call for the Iceberg Summit 2027 Selection Committee after the PMC approved the event, and the volunteer replies stacked up fast. Russell Spitzer, Ryan Blue, Kevin Liu, Dipankar Mazumdar, Neelesh Salian, Roy Hasson, John Zhuge, Talat Uyarer, Nathan Yee, Hongyue Zhang, Arnav Balyan and others put their names in within days. The committee will have 11 members, no more than one per company, and at least three from the Iceberg PMC. Volunteering closes Friday, September 11 at 11:59 PM PDT, and the PMC votes on the roster after that. The work is real but bounded, mostly program structure plus CFP review, which ran three to six hours last year.

That per-company cap is worth pausing on. Iceberg's conference program is now selected by a committee designed so no single vendor can shape the agenda, and the volunteer list already spans most of the major players in the ecosystem. For a project this commercially contested, that structure does more for trust than any code of conduct document.

On the spec side, the catalog surface kept expanding. Prashant Singh's finer grained read restrictions vote passed with binding +1s from Honah J., Kevin Liu and Amogh Jahagirdar, and he posted the result thread on September 3. The change lets a REST catalog hand back a restricted view of table data rather than an all-or-nothing load, which is the piece most enterprise deployments have been faking with a proxy in front of the catalog. Singh also pointed out a detail buried in the spec examples. The masked email address in the sample is iceberg16112018@apache.org, and 16112018 is the date Iceberg entered incubation.

Right behind it, Andrei Tserakhau restarted the vote on catalog-provided labels in the REST read path. The proposal is deliberately small, adding an optional flat key-value map to LoadTableResult and LoadViewResult, with structured tag entities and write APIs pushed to follow-ups. Daniel Weeks, Fokko Driesprong, Ryan Blue, Anoop Johnson, Amogh Jahagirdar, Kevin Liu, Sung Yun and Eduard Tudenhöfner all weighed in, and the result thread closed it out. Scoping the vote to the read path was the smart move here. Labels are the kind of feature that grows a governance model, a write API and a permissions story if you let the first version carry all of it, and splitting the spec change from the implementation PRs kept the discussion on one question at a time.

The view spec got the most interesting design debate of the week. Alex Stephen asked why creating a view with multiple engine representations is so hard in practice. The spec allows one view to carry Spark, Trino and Flink dialects, but there is no way to append a representation to an existing view, only to replace one, so a direct REST call is the only real path. He proposed a new view property, replace.append-dialect.allowed.

Ryan Blue's first question was whether the intent was for CREATE VIEW to append automatically. Stephen argued the real pain is the user journey, where a query fails because the engine picked the wrong representation and the only recourse is calling REST endpoints by hand. Daniel Weeks pushed the discussion toward ALTER VIEW rather than a property flag, and he made the argument that carried the thread. An engine should only add or update its own dialect, because updating another engine's representation would require parsing SQL it cannot parse. Weeks also objected to toggling behavior with properties, since that makes a declarative statement depend on session configuration. Prashant Singh backed that and pointed at StarRocks, which already has explicit ADD DIALECT and MODIFY DIALECT grammar.

Péter Váry raised the consistency question that has no clean answer. If a Flink user adds a Flink representation to a view created in Spark, how does anyone know the two return the same rows? His position is that the project accepts the user's authority on that. Talat Uyarer proposed a workflow that needs no spec change at all. Engines should display every representation with its dialect label in DESCRIBE EXTENDED, since the SQL strings are already in the metadata and showing another engine's text requires no translation. The user reads the Flink SQL, writes the Spark equivalent, and issuing ALTER VIEW is itself the consistency assertion. Váry still prefers defining both statements at creation time so the view is never partially defined. The thread has not settled, but it moved from "add a property" to "fix the presentation and use ALTER," which is a better place to be.

Two v3 and v4 discussions produced concrete decisions. Marco Kroll's thread on the _pos column in efficient column updates asked whether the spec has any precedent for storing data purely for debugging. He searched and found none, and Russell Spitzer agreed the check belongs at write time, since a writer that sees a mismatch between origin row position and actual position should just fail. Anurag Mantripragada closed it: the Efficient Column Updates sync decided _pos will not be required, and writing it will not be allowed. That is the right call. Every optional field in a spec becomes a compatibility question for every reader that follows.

Ryan Blue's field ID tracking for non-materialized columns in v4 went deeper into transform metadata. The design requires a result data type when the output cannot be derived, uses bound id-based references, and skips a special case for identity transforms. Russell Spitzer asked for sort orders to get the same treatment and raised the lineage question, whether a table now carries both a SchemaID and an ExpressionsID. Gianluca Graziadei pressed on Hilbert clustering over heterogeneous columns, where feeding raw values to the curve wastes bits and degrades clustering along a dimension. Blue's answer draws the line clearly. Whatever parameterization a function needs must be fully captured by the expression stored in metadata, so hilbert(zvalue(col1, 0, 1), zvalue(col2, 100, 50)) is fine and hidden inputs are not. Péter Váry connected it to his index work, where a Cluster spec already lists materialized and non-materialized fields, and suggested redefining sort order so the index can reuse it.

Russell Spitzer also restarted the File Type URI relativization debate after a long sync call, keeping the discussion on the list between meetings so people who could not attend can follow the argument. The open question is how the URI subfield of the new File type behaves, since Parquet allows more than one persistence form. EJ Song opened a related performance thread on row-granular concurrency for V3 deletion vectors. With one DV per data file, validateAddedDVs fails a commit at file granularity even when two operations deleted rows that do not overlap, and the resulting ValidationException is not retryable, so the whole operation is recomputed. Xiening Dai backed the idea and reminded the thread that a commit loop should retry until either a real conflict appears or the version conflict clears.

Releases moved on three fronts. Neelesh Salian posted the 1.12.0 status, grouped the remaining PRs by how close they are to merge, and said he wants RC0 cut in days, with a longer vote window because of the US long weekend. Danny Jones called the vote on Iceberg Rust 0.11.0 RC1, and the verification reports are worth reading as a template. L. C. Hsieh checked signatures, 489 license headers, 2,209 passing tests and the Python bindings on macOS arm64, and Anoop Johnson ran the full suite including the minio, REST, HMS and Spark integration tests on Ubuntu. The Terraform provider v0.1.0 RC3 vote also passed, giving teams a first supported path to manage Iceberg catalog resources as infrastructure code.

Smaller items still worth your attention. Rahul Mahadev's proposal to standardize a User-Agent format for REST clients got bumped by Micah Kornfield after it landed in his spam folder, which is its own small comment on how much good work gets lost to mail filters. Someone posted a 10x faster Z-order bit interleaving implementation using a lookup table and asked for review. And the European community keeps growing, with meetups announced for London and Warsaw in October.

Apache Polaris

Polaris spent the week on performance, protocol contracts and the health of its own review process.

Dmitri Bourlatchkov opened the most productive thread by questioning whether InMemoryEntityCache earns its keep. His observation was simple. With JDBC persistence, even a cache hit still issues a query to confirm the entity version, so what is the cache actually buying?

Prithvi S answered with numbers instead of opinion. He traced Resolver.resolveAll() with JDBC and compared a warm cache against a null cache on the same catalog, using a loadTable-shaped resolve down a two-level namespace path. On a warm cache the resolve issued a single SELECT against the ENTITIES table pulling id, catalog_id, entity_version and grant_records_version. His framing is the key point: InMemoryEntityCache is not a "skip the database" cache, it is a "skip the expensive load" cache, with the version check as the invalidation path. Yufei Gu backed that, noting the version check is what keeps a multi-pod Polaris deployment consistent, because one pod cannot see another pod's writes.

Robert Stupp then found the real problem hiding in the data. A cold resolve of that same simple path issued 23 SELECTs. He asked whether that is the expected cold-path shape or whether the hierarchy, grants and versions can be fetched in a bounded number of operations, and he proposed folding the version fence into the final conditional write for mutations, so an UPDATE with a version predicate that touches zero rows becomes the conflict signal. Jean-Baptiste Onofré traced the 23 queries to their source. AtomicOperationMetaStoreManager.loadResolvedEntityById() issues one or two grant-record queries per entity and gets called once per entity in a resolved path with no batching, which is a classic N+1 and has nothing to do with caching. Bourlatchkov added a second angle, asking whether grant record lookups are needed at all when an external authorizer like Ranger or OPA is in play, since those queries return empty results anyway.

That thread is a good model for how performance discussions should go on a dev list. A question, a traced measurement, a correction to the framing, and then a split into independent fixes.

The Iceberg Catalog Migrator release did not make it. Ajantha Bhat called the vote on 1.1.0 RC0 and collected careful verification from JB Onofré, Ayush Saxena, Prithvi S and Robert Stupp. Onofré caught that the binary distribution and CLI jar do not document jquery.jstree.js, an MIT-licensed dependency that needs its license inline. Bhat then cancelled the vote, and said he will fix the jars to include META-INF/LICENSE and META-INF/NOTICE and exclude webapps content from the Hadoop dependencies before cutting a new RC. A cancelled vote over a missing license file looks like friction from the outside. It is actually the ASF release process working exactly as designed.

Data sharing came back to life. Jean-Baptiste Onofré resumed the Open Sharing APIs proposal and said he would open a draft PR for an /api/shares/v1 endpoint to make the design concrete. Dennis Huo had a competing draft ready in PR 5446, favoring /api/management/ for share administration and /api/shares/ for the consumer data plane, plus a companion document walking through end-to-end user journeys for each design choice. Prithvi S read both and endorsed the shape: a first-class share with enumerated members, a restricted consumer, a listing that serves as the binding and audit unit, and a segregated read-only Iceberg REST surface with credential vending. Keeping stock Iceberg REST as the consumer protocol and leaving Flight and Polaris-to-Polaris federation out of v1 also drew agreement. Huo laid out the current mental model, where everything considered data plane lives under /api/catalog/ and everything administrative lives under /api/management/, while agreeing the privileges themselves should separate catalog, principal and sharing concerns.

The Tag Spec review turned into a lesson in API contract design. Robert Stupp raised four contract questions while the spec is still separate from the implementation, starting with the encoding of identifier elements in target query parameters. His point goes past the unit separator. Namespace elements and object names can contain ampersands, question marks, equals signs, plus signs and percent signs, and those need to survive rather than be read as query syntax. Prithvi S argued all four should be resolved before PR 5366 merges, since they are contract questions rather than storage layout questions. Dmitri Bourlatchkov went further and said Polaris should adopt a stricter namespace representation in its native APIs instead of inheriting the Iceberg REST convention, which has produced recurring issues on the Iceberg list. EJ Wang updated the spec doc with the clarifications and proposed keeping Iceberg's namespace query convention for v1, with explicit supported-name, encoding and decoding rules written into the spec, and a replacement codec handled separately since the issue affects existing APIs too.

Storage support keeps widening. Austen Tomek from Chicago Trading Company introduced himself and proposed Cloudflare R2 support with scoped credential vending, which is a first contribution done the right way, on the list before the PR. R2 speaks S3-compatible APIs but has no STS, so Cloudflare issues short-lived scoped credentials through locally signed JWTs while the server holds the parent API token. Yufei Gu read that as STS-style vending where Polaris performs issuance without calling out to the object store, and leaned toward treating R2 as S3-compatible storage rather than a new config type. Sushant Raikar pushed back on the middle ground and asked whether R2 should be fully first-class, with its own config, its own credential vending and its own R2FileIO, which would sidestep the one-FileIO-to-one-storage-type question entirely at the cost of a new FileIO to maintain. That trade is one every catalog faces as object stores multiply, and it is better settled once than per vendor.

Two more threads deserve a mention. Vignesh A's soft-roll authorization for the Iceberg REST /v1/config endpoint produced a clean distinction from Bourlatchkov: GET_CATALOG_CONFIG_PROPERTIES is an operation, not a privilege, because the SPI is written in terms of operations for non-native authorizers like Ranger, and OPA has no concept of privileges at all. And Yufei Gu's thread on PR review and committership named something every project is now living with. LLMs make it easy for contributors to submit large PRs, large PRs are harder to review well, and across many communities they sit unmerged while both authors and reviewers get frustrated. Onofré split the metric in two. Time to initial response matters a lot and should be fast, because it keeps contributors engaged. Time to merge matters much less than project quality and long-term maintainability, and iteration on a PR is normal committer work, not failure.

Elsewhere on the list, a GitHub discussion on export and apply for tables and views surfaced a real bootstrap gap. Users want to export a realm including tables, views and their privileges, rewrite the file paths, and apply the result in another environment. MonkeyCanCode flagged the hard parts, including agreeing on a table creation syntax without SQL, handling dialect-specific views, and the fact that an export full of filesystem paths cannot be reused across environments without rewriting. The community also announced a Bay Area meetup on September 30.

Apache Arrow

Arrow's week was about pruning the format and welcoming people.

The vote to informally deprecate Tensor and SparseTensor in IPC, started by Raúl Cumplido, passed with binding +1s from Weston Pace, Joris Van den Bossche and David Li plus support from Benjamin Kietzman, and Cumplido posted the result. Max Burke brought a useful pointer into the thread, an open flatbuffers proposal for a deprecated-readonly attribute that would keep generating getters while dropping setters, so existing data still reads and new data cannot be written with the field. Antoine Pitrou said it might help later, while noting Arrow can rely on conventions and let each implementation decide what it exposes. Informal deprecation is the pragmatic path here. The types stay readable, nobody's archived data breaks, and implementers get a clear signal to stop investing.

The more consequential format discussion was Mandukhai Alimaa and Rok Mihevc's proposal for a canonical extension type for the FILE type. Parquet recently added a FILE logical type, Parquet C++ reader and writer support is underway, and without a matching Arrow representation file fields lose their semantics crossing IPC, the C Data Interface, Flight and language bindings. Antoine Pitrou immediately questioned the name and the ownership. Should it be parquet.file rather than arrow.file, and is it Arrow's job to standardize a type that Parquet defined? Mihevc agreed the namespace should show where the spec came from and pointed at geoarrow as precedent for non-Arrow-namespaced types. Gang Wu and Neelesh Salian backed parquet.file, and Matt Topol noted the existing type shipped as arrow.parquet.variant, so symmetry argues for arrow.parquet.file. The naming is small. The principle is not. As formats borrow types from each other, the namespace is what tells a reader three years later which specification governs the semantics.

Andrew Lamb announced Kosta Tarasov as a new Arrow committer, and the congratulations thread ran long, with Neelesh Salian, Ruoxi Sun, Raúl Cumplido, Kevin Gurney and Jeffrey Vo among the repliers. Tarasov's name showed up on the Parquet list in the same week as a co-author of the merged Rust ALP implementation, which is a nice illustration of how contributions compound across sibling projects.

Matt Topol shipped a release the hard way. His Arrow Go 18.8.0 RC1 vote hit a bad signature error that David Li caught in verification, Topol re-signed and re-uploaded, and then the checks passed for Neelesh Salian, David Li on Ubuntu 25.04 with Go 1.27 and Raúl Cumplido on Debian 14 with Go 1.27.1. The result and the release announcement followed within a day.

The most interesting outside contribution came from Prateek Singh, who announced ArrowMetal 0.1.0, Arrow compute kernels running on Apple silicon GPUs through Metal. The design detail that matters is memory. Arrow buffers live in shared-storage Metal memory, so an array is a valid CPU Arrow buffer and a valid GPU buffer at the same time and nothing gets uploaded or downloaded. Data crosses through the C Data Interface, the C Stream Interface and the C Device Data Interface with ARROW_DEVICE_METAL, which means it works with pyarrow, Polars, DuckDB, pandas, arrow-rs, Arrow Go, Arrow JS, the R package and arrow-swift. Curt Hagenlocher's reaction captured the significance: this might breathe new life into the Device Data interface. Matt Topol agreed and said he will look at the Arrow Go issues, possibly saving some for the hackathon at Community Over Code in October. The zero-copy device interface has been in the spec for a while without a headline consumer. A laptop-class GPU backend that works across eight language bindings is exactly the kind of thing that pulls a dormant interface into use. The project also held its community meeting on September 9.

Apache Parquet

Parquet had the busiest technical week of the six projects, and nearly all of it circled one question: what does a format version promise?

The versioning proposal thread ran past 45 messages. Micah Kornfield pulled it back from mechanics to requirements, arguing the community should agree on the rules before arguing about what goes in the footer. His two anchors are worth quoting in spirit. A file written with a preview feature must be readable by any reader that supports the preview features used, and by a reader that supports the major version where the feature was fully adopted. A reader must never return incorrect data when it does not understand a preview feature, which puts the burden on writers to communicate preview usage in a way that makes silent misreads impossible.

That second rule is the one that shapes everything else. Parquet files outlive the software that wrote them, and a reader that quietly returns wrong values is worse than a reader that refuses to open the file. The related threads all inherit from it: whether to write the version number in the footer, how readers should behave when they meet an unsupported format version, what forward compatibility means when files get rewritten, and a useful comparison with how Arrow IPC handles the same problem.

One piece of that already shipped as spec text. Divjot Arora's vote on handling unrecognized logical and physical type combinations passed with binding +1s from Antoine Pitrou, Micah Kornfield, Fokko Driesprong and Gang Wu, plus non-binding support from Russell Spitzer and Matt Topol, and Arora posted the result. Pitrou described it as short and useful, which is the highest praise a spec clarification gets.

The build side got simpler. Divjot Arora's proposal to inline parquet.thrift into parquet-java drew a full options review. Ryan Blue approved the PR, which uses a local copy plus a script that pulls new copies from parquet-format by commit hash or ref and records the resolved version in a parquet-format.version file. Blue argued against writing scripts to diff and validate the local copy, since the file is in version control and git already does that well. Gang Wu floated a git submodule pointing at a commit hash, then clarified his reply was not blocking. Fokko Driesprong had the same thought and rejected it on ergonomics, calling submodules clunky, easy to leave stale and awkward in daily use, and he opened a PR to reinstate nightly parquet-format snapshots as an alternative. Arora documented all the considered options in the thread for posterity, and Russell Spitzer landed the closing argument: option one is not worth debating against a better solution until the simplest one is in place.

Releases are moving on both format and Java. Fokko Driesprong opened the 2.14.0 format release discussion in the spirit of releasing more often, carrying chronological ordering of INT96 timestamps, the FILE logical type with its self-reference follow-up, and Adaptive Lossless Floating-Point encoding. Arora asked to slip in his type-combination PR once its vote closed, Driesprong added it to the milestone, and Gang Wu and Andrew Lamb both backed the release, with Wu noting it unblocks a queue of waiting PRs. The 2.14.0 RC1 vote is now open, alongside the Parquet Java 1.18.1 RC1 vote.

Arora also drove the vote on extended precision nanosecond timestamps, which lets TimestampType annotate FIXED_LEN_BYTE_ARRAY(12) columns so the full SQL timestamp range fits in nanoseconds, with all three time units supported and reference implementations in Java and C++ plus a parquet-testing file. Daniel Weeks, Micah Kornfield and Ryan Blue voted binding +1, with Stevo Mitrić and Alkis Evlogimenos non-binding. Twelve bytes for a timestamp sounds extravagant until you have tried to store pre-1677 dates at nanosecond precision in 64 bits.

The vector type debate is the one to watch. Rok Mihevc summarized a focused call on the physical representation of a numeric vector type, convened because vector database storage needs differ from the fixed-size-list discussions so far, and because Iceberg needs a vector type too. The call converged on two options without picking one. Antoine Pitrou challenged the premise, asked for a real explanation of the different storage needs, called option A a short-term fix with severe encoding limitations, and said he is only lukewarm on option C while conceding it does not paint the project into a corner. Mihevc conceded the framing was too broad and restated the requirement as a contract rather than a layout: a fixed number of numeric elements, elements that cannot be null and are finite, and room for future vector-specific properties like normalization guarantees or specialized encodings. He noted Lance uses FixedSizeList with specialized physical encodings while Hudi stores a vector as a single FLBA.

Will Edwards made the strongest counterargument, that efficiency here is a software problem rather than a format problem, since a reader can expose flat typed memory and nothing in the format forces a List allocation per row. He reframed the real question as what SHOW CREATE TABLE should say when a Parquet file is the only source of schema. Daniel Weeks focused on semantics instead, arguing the point of a logical vector type is to differentiate it from a fixed-size list, so vectors should prohibit nulls, NaN and infinity as elements rather than merely detecting them in statistics. This is a genuinely hard design call, and the fact that Iceberg is waiting on the outcome makes it one of the most consequential decisions in the ecosystem right now.

Encoding work kept pace. Prateek Gaur's ALP encoding thread turned into a cross-language progress report. Andrew Lamb reported the Rust implementation from Kosta and Devan merged, Vinoo Ganesh is addressing Gang Wu's comments on the Java PR, the C++ PR is in another review round, and Gaur and Arnav Balyan are starting the Go implementation. With ALP riding in Parquet Format 2.14.0, floating-point columns are about to get materially cheaper across four language stacks at once. PFOR encoding and an extensible decimal floating-point type are moving behind it.

One security item needs action. Gidon Gershinsky published CVE-2026-73334, a moderate-severity issue in the org.apache.parquet.crypto.keytools package affecting parquet-hadoop 1.12 through 1.18.0. When a writer sets the optional KMS URL parameter, that URL is stored in the file, and a reader configured to trust file-controlled KMS URLs can forward it to a pluggable KmsClient that skips host validation. If you use Parquet envelope encryption with a custom KmsClient, read the advisory and check whether your reader configuration trusts file-controlled URLs. The community also kept its sync cadence, with notes from the September 9 sync posted the same day.

Apache DataFusion

DataFusion gained a repository and a maintainer roster this week.

The headline is the vote to accept the Iceberg DataFusion integration into the DataFusion project, opened by Andrew Lamb on September 9 with a proposed PR from Gabriel Musat to move the code into apache/datafusion-iceberg. Andy Grove and L. C. Hsieh voted binding +1, with Kevin Liu, Kumar Ujjawal and Shekhar Rajak non-binding, and Liu cross-linked the matching Iceberg vote so both communities are voting on the same move.

The discussion that led there ran 30 messages across both lists. Lamb's argument came from the field. At VLDB he spoke to at least three companies adding Iceberg support to their products, and every one of them had forked iceberg-rust for some reason. His conclusion is that giving the DataFusion integration access to maintainers who have deep DataFusion context helps everyone, and that the code should stay in the ASF under DataFusion governance rather than drift into a vendor repo. Kurtis Nusbaum, writing on the Iceberg list, agreed the DataFusion internals knowledge does not belong in an Iceberg-specific package, while flagging two honest counterarguments. Anyone forking to add a feature now has to fork in two places, and arrow-rs and parquet already show that one repo can hold two projects. He judged neither strong enough to outweigh the maintenance case. Kevin Liu agreed and removed the Iceberg Python binding that exported DataFusion's TableProvider, one less coupling to carry across the split. Renjie Liu and Gabriel Musat backed the plan, Musat volunteered to port the commit history, and Lamb filed for the new repository. The history port PR and a compile-and-test PR are already open.

Two releases are in flight. Tim Saucer's DataFusion 55.1.0 RC1 vote collected binding +1s from L. C. Hsieh, Adrian Garcia Badaracco, Andrew Lamb, Marko Milenković and Oleks V., verified across Apple silicon and macOS 15 with rustc 1.98.1. The sqlparser-rs 0.63.0 RC1 vote is moving in parallel, which matters well beyond DataFusion, since sqlparser-rs is the SQL front end for a long list of Rust data tools.

Andrew Lamb also announced Luca Cappelletti as a new DataFusion committer, with congratulations from Bruce Ritchie, Kumar Ujjawal, Bhargava Vadlamani and Jeffrey Vo. And Comet got its own meeting slot. The dedicated Comet weekly sync now runs Fridays at 10:30 AM Pacific, with Bhargava Vadlamani adding the meeting link to the Comet docs and Manu Zhang asking for recordings so contributors in other time zones are not shut out. The project is also crowdsourcing its September ASF board report.

Apache Ossie

Ossie, the incubating semantic layer interchange project, had its most substantive week yet, and it is the list to start reading if you have not.

Justin Talbot opened a PR adding a Relational Query Interface specification to core-spec, coming out of the expression language working group. It defines Layer 2 of the proposed layered query interface, covering how a semantic layer exposes an Ossie model to SQL-native BI and AI tools as related SQL relations, and what correctness guarantees hold when measures are queried through ordinary SQL with a MEASURE() extension. Some of the core ideas trace back to Hyde and Fremlin's Measures in SQL paper.

That paper's first author is on the list. Julian Hyde, in his introduction thread, laid out the position that will shape this spec if it holds. The query language must be closed, meaning query outputs have the same shape as their inputs, so queries can be composed on queries. If the language also subsumes relational algebra including joins and aggregation, and can define measures, then no separate modeling language is needed, because models are definable on base tables with the equivalent of CREATE VIEW. That is a strong claim, and it cuts against how most semantic layer products are built today, where the model is a YAML dialect sitting above SQL rather than an extension of it.

Jakub Moravec brought the equivalence problem in from a different angle. If Ossie converts a model from tool A to tool B, who guarantees the two are semantically equivalent? He argued at least one dimension of that is a lineage problem, drawing on his OpenLineage experience, and noted that evaluating equivalence by running queries depends on both implementations being correct and on the test data being complete. His related point in the how do we expect OSI to be used discussion is the one that should worry spec authors. In OpenLineage it is easy for a payload to be schema-valid and still useless for cross-vendor integration, because checking syntactic validity is simple while checking whether what was documented is sufficient is not.

Modeling proposals stacked up. A proposal for shared filters, shared dimensions and metric references drew a strong argument from wanggaohang for model-level filters. Enterprise filters get shared by many metrics, and inline-only definitions make every business-definition change expensive to propagate, while filters and metrics are often owned by different roles, with domain experts maintaining the definitions. Mario De Felipe argued in make relationship cardinality explicit that cardinality belongs in two places because it means two different things. At the ontology layer it is a business rule that holds regardless of how tables are laid out. At the dimensional and metric layer it describes the physical data, which is what join safety and path selection need. There are also live proposals for hierarchy support, where fabrice-etanchaud pointed at Mondrian's long-standing treatment as prior art, plus semantic filters and a dedicated thread for downstream agent field reports.

Vendors are showing up with code rather than opinions. Damian Waldron announced a bidirectional ThoughtSpot converter between TML and Ossie, ThoughtSpot's first code contribution to the project. Ossie datasets map to a ThoughtSpot model's tables plus a Table or SQL View document per dataset, fields and metrics map to model columns, relationships map to joins, and anything with no Ossie equivalent rides in custom_extensions under a vendor entry. Timextender also introduced itself and a converter it plans to contribute. Converters are how an interchange format proves it is real, and that custom_extensions escape hatch is both the pragmatic choice and the thing to watch, since every vendor-specific field that lands there is a piece of semantics the spec has not yet standardized.

The project is also learning to ship. The first release discussion settled the scope question quickly. Markus Weimer asked whether the first release needs to line up with the 1.0 spec and suggested a 0.3 instead, noting from past incubators that building the release muscle matters as much as the technical agreement. Yufei Gu confirmed the community sync reached the same conclusion, Jean-Baptiste Onofré said he is building the release machinery for a source-only 0.3.0 to verify plumbing and run a full legal check, and Russell Spitzer endorsed treating it as an exercise rather than loading it with meaning. Kurt Stirewalt asked the practical follow-up: the ontology working group wants PR 332 in the release, and is there a formal process for requesting that, or for deciding which in-review features make a given release? Governance questions like that are exactly what a first release surfaces. The project also welcomed Josh Klahr to the PPMC, and worked through build questions including Just versus Makefile and consolidating Python code under one uv workspace.

Cross-Project Themes

Code is moving to where the maintainers are. The Iceberg Rust DataFusion integration voting its way out of Iceberg and into DataFusion is the clearest case, but the same instinct shows up in Arrow debating whether a Parquet-defined type should carry a parquet namespace, and in Parquet inlining parquet.thrift into parquet-java instead of coordinating two repos on every change. Three years ago the ecosystem grew by adding integrations inside each project. Now it grows by putting each integration under the governance of the people who actually maintain that side of it. Andrew Lamb's VLDB observation, that every company adding Iceberg support had forked iceberg-rust, is the market signal behind the governance change.

Catalog metadata is becoming spec, and the arguments are about contracts rather than storage. Iceberg voted labels into the REST read path and passed finer-grained read restrictions. Polaris spent the week on tag spec encoding rules, on whether a config-endpoint check is an operation or a privilege, and on what a share is as an audit unit. In every one of those threads the productive move was the same: separate the wire contract from the implementation, resolve the contract first, and let the storage layout follow. Robert Stupp's insistence on a reversible encoding for namespace elements containing ampersands and percent signs is not pedantry. It is the difference between a spec that survives its second implementation and one that does not.

Version and compatibility semantics are the new center of gravity. Parquet's versioning thread, its unsupported-version reader behavior thread and its footer-version thread are all one question. Iceberg's decision to forbid writing _pos is the same question from the other side, refusing to add an optional field that every future reader would have to reason about. Arrow's informal deprecation of Tensor takes the same position, keeping old data readable while cutting off new writes. Formats that outlive their software have to be explicit about what a reader is allowed to assume, and three projects converged on that independently in one week.

Semantic modeling has an Apache home, and it is drawing the people who wrote the theory. Ossie has Julian Hyde arguing for a closed query language, ThoughtSpot and Timextender shipping converters, and OpenLineage veterans warning about schema-valid but useless payloads. Its numeric vector question is being decided in the Parquet list. Its query interface leans on Measures in SQL. If you build on the lakehouse and you have been treating the semantic layer as a vendor concern, that assumption is expiring.

Looking Ahead

Iceberg 1.12.0 RC0 should appear within days, with a longer vote window than usual. The Iceberg Summit 2027 Selection Committee call closes September 11 and the PMC vote on the 11 members follows. Both DataFusion and Iceberg votes on moving the DataFusion integration run at least seven days, so expect the apache/datafusion-iceberg repo to become real in mid-September. Parquet Format 2.14.0 RC1 and Parquet Java 1.18.1 RC1 are open, and the vector type physical representation is the discussion most likely to produce a decision with ecosystem-wide consequences. Polaris will cut a new Catalog Migrator RC once the license packaging fix merges, and the Open Sharing draft PRs are the ones to read. Ossie is building release machinery for a source-only 0.3.0. Community Over Code lands in October, and Matt Topol has already flagged the Arrow device interface work as hackathon material.

Want to go deeper on Apache Iceberg, Polaris, Arrow, Parquet, DataFusion and the rest of the open lakehouse stack? I write books on all of it, from beginner guides to architecture deep dives. You can browse the full catalog at books.alexmerced.com.

Alex Merced, Data Lakehouse and AI Evangelist