feat: declare derived columns on the column itself - #2517

Stack: 3/6 — stacked on #2516. Completes layer 1 of specs/derived-columns-and-system-one.md (§3.3, §3.5, §3.7).

Review just the last commit. The base is #2516.

Puts the provenance of a derived column next to the column, instead of in a schema-level parsers list:

class Tickets(pa.DataFrameModel):
    body: str
    n_words: int = pa.ParsedField(
        source="body",
        parser=lambda s: s.str.split().str.len(),
        ge=1,
    )

schema = pa.DataFrameSchema({
    "body": pa.Column(str),
    "n_words": pa.ParsedColumn(
        int,
        source="body",
        parser=lambda s: s.str.split().str.len(),
        checks=pa.Check.ge(1),
    ),
})

ParsedColumn is a Column subclass and ParsedField a FieldInfo subclass, so everything they accept keeps working — checks, coercion, nullability, description. Each desugars into the Parser(source=, target=) primitive from #2516.

Config.parser_source (and DataFrameSchema(parser_source=...)) is the schema-wide default for columns that don't name their own source. Deriving from a column the schema doesn't declare is a SchemaInitError at build time, not a KeyError at parse time.

@pa.parser generalized, not duplicated

Given a source, the named fields become what it produces, and the parser attaches to the schema rather than the column — a column-level parser can't create its own column:

@pa.parser("n_words", source="body")
def count_words(cls, s):
    """Number of whitespace-separated tokens."""
    return s.str.split().str.len()

Without source it keeps its original meaning. One decorator, one concept.

The ColumnParser protocol

parser= also accepts an object implementing:

class ColumnParser(Protocol):
    def bind(self, ctx: ParseContext) -> Callable: ...
    def batch_key(self, ctx: ParseContext) -> Hashable: ...

bind is called once at schema-build time with a ParseContext describing the target column — name, declared dtype, description, nullability, checks, resolved sources, and the schema. Two things follow, and they're the reason the protocol exists at all:

• A parser can read the column's declared type rather than being told what it's producing. This is what lets a later system_one.Choice() derive its options from the column's Enum without repeating them.
• A parser can refuse early. SchemaInitError from bind surfaces before any data is touched.

batch_key groups columns whose parsers can share work. Equal non-None keys go to the class's batch classmethod and are filled by a single call; None opts out. This is what lets a schema declare one derivation per column without paying for one pass per column — the mechanism the System One layer needs so per-column declarations cost one request, not N.

Two details worth a look

• Batched parsers always receive a DataFrame (Parser(frame_input=True)). Without it, a batch that happens to fill a single column would take the single-source/single-target Series -> Series shortcut and break. Found by a test, not by inspection.
• ParsedColumn needs its own backend registration. The registry keys on the exact schema class rather than walking the MRO, so a Column subclass doesn't inherit Column's backend.
• Worth knowing for reviewers: pandera deep-copies columns into the schema, so a ColumnParser instance handed to ParsedColumn is not the object that gets bound. The tests assert against schema.columns[...].parser, and stateful parsers can't rely on the caller's reference.

Testing

25 new tests in tests/pandas/test_parsed_columns.py: object and model APIs, schema-wide default and per-column override, chained derivation declared out of order, undeclared sources, the decorator in both modes, the protocol (bind receives the right context, errors surface at build time), and batching (merged, not merged, opted out, missing batch method).

Suites clean: pandas+io+base (2745 passed), polars+ibis (666 passed, plus the pre-existing test_ibis_backend_is_narwhals failure that also fails on main).

🤖 Generated with Claude Code