Repository navigation
Compliance suite has not gained a case since 2024, while the join engine was rewritten under it #320
Description
Activity
Summary
Verified against current
main: this issue's numbers hold up.test_joinintests/test_compliance/test_compliance_suite.py(lines ~914–1010) is parametrized overinlined: [True, False]against a single(source_object, target_object)pair — 2 combos total, exercising only theexpr-baseds1.id/s2.iddenormalization path viapopulated_from. Nothing in the compliance suite invokes the DuckDB engine selection path.One nuance worth adding
The join engine itself is not actually undertested — it's just untested in the artifact that gets published as the spec.
tests/test_transformer/test_join_engine.pyis 681 lines with 18+ cases, includingtest_sparse_miss_suppresses_object(the a19eb09 behavior called out in this issue),test_simple_subject_join_is_engine_capable/test_fk_chain_is_not_engine_capable(thecan_use_join_enginegating insrc/linkml_map/transformer/join_engine.py:108),test_to_many_join_does_not_explode_rows, andtest_empty_joined_file_degrades_to_null. There's alsotest_cross_class_lookup.py(254 lines) andtest_join_synthesis_completeness.py.So the real gap this issue identifies is precise: it's not "the join rewrite shipped without tests," it's "the join rewrite shipped without compliance-suite tests," and since
docs/specification/compliance.mdis what's linked under Specification in the nav, readers of that doc have no way to know the set-based engine, its per-row fallback, or sparse-miss suppression exist. That actually makes the fix cheaper than starting from scratch — the scenarios already exist intest_join_engine.pyand just need porting into the compliance suite'smap_object/scaffold style (pertests/README.md's pattern) rather than being invented fresh. Same logic likely applies to the enum/--unrestricted-evalgating point —object_transformer.py:592andcli.py:81confirm that gate is real and current, so it's a matter of surfacing existing behavior in a compliance case, not building new coverage.Related issues (same audit thread, all from @amc-corey-cox)
- The compliance doc can't be regenerated reproducibly, and the pandoc step is redundant #318 — explicitly linked here. Covers the build side (non-reproducible
Time_executed/Package:stamps, redundant pandoc step) of the same doc. Confirmed still open; this issue and The compliance doc can't be regenerated reproducibly, and the pandoc step is redundant #318 are correctly scoped as independent (neither blocks the other) but both need to land before a CI drift check is meaningful. - Makefile is half broken cookiecutter boilerplate; make deploy always fails #299 — the
make specificationtarget this doc depends on was broken until Retire the cookiecutter scaffolding in the Makefile and agent docs #317 fixed a path typo; worth knowing the doc-generation pipeline itself is only recently reachable at all. - Cut misfiled and mock-based tests (~1,600 lines) #298 — a test-suite complexity audit that explicitly flags
test_compliance_suite.pyas "the sole coverage forTransformationSpecificationInverter" and says it "stays" — reinforcing that this file is treated as load-bearing spec material, not disposable test cruft, which raises the stakes on it being current. - Alternative Implementation: Rust Port (linkml-map-rs) #262 — a third-party Rust port (
linkml-map-rs) claims "passes the ported upstream compliance suite (55/55 tests) and matches reference Python outputs byte-for-byte." Worth flagging to that author once this suite gains cases: their parity claim is against the 2024-frozen 55, so it says nothing about join-engine or gated-eval behavior yet.
No external papers or web resources apply here — this is an internal test-coverage/doc-fidelity issue.
- The compliance doc can't be regenerated reproducibly, and the pandoc step is redundant #318 — explicitly linked here. Covers the build side (non-reproducible
Regenerating
docs/specification/compliance.mdtoday produces the same content as the version stampedTime_executed: 2024-07-03— all 66 headings match one for one, differing only by the{#anchor}suffixes pandoc adds. The doc isn't out of date because nobody rebuilt it. It's identical because the suite has not gained a case in two years.Ten commits have touched
tests/test_compliance/since that date, and every one is mechanical:isort,Run ruff format,Automatic ruff lint fixes,Move "unit" dev group into the main group, a linkml version bump.What the 55 combos actually cover:
Seventeen of the 55 are unit conversion. Joins get two — against an engine that has since been replaced:
Enum handling also changed recently (7cffbf0, routing enum expression evaluation through the gated eval path), and
--unrestricted-evalgating is a behavioural boundary a compliance suite ought to pin down.This is the part that matters more than how the doc gets built: we publish this under Specification in the nav, so it reads as a statement of what LinkML-Map does. Right now it's a statement about mid-2024, and the areas we've changed most are the ones it covers least.
Intended outcome: the compliance suite exercises what the engine actually does now — set-based joins and sparse-join misses in particular — so the published doc is a description of current behaviour rather than a historical one.
Worth doing alongside #318, which covers why the doc can't currently be regenerated reproducibly. Neither blocks the other, but both need to land before a CI drift check could keep this honest on its own.