Skip to content

feat(generators): add --normalize-prefixes flag for well-known prefix names - #4

Open
jdsika wants to merge 1 commit into
mainfrom
fix/normalize-well-known-prefixes
Open

jdsika wants to merge 1 commit into
mainfrom
fix/normalize-well-known-prefixes

Conversation

@jdsika

@jdsika jdsika commented Mar 26, 2026 •

Copy link
Copy Markdown

Mirror of linkml#3446, same head. Review there; this PR tracks the fork stack.

2026-10-23, second push (review): (1) the hand-written override table is gone: where prefixmaps' names differ from the names rdflib binds by default (geosparql/geo, sc/schema, WGS84/wgs), rdflib's win, read from Graph(bind_namespaces="rdflib"); the resulting map is identical (16,440 entries). (2) The option was added to every generator's command line and silently ignored by most; it is now declared only on gen-owl, gen-shacl, gen-jsonld-context and gen-jsonld, which honour it, with tests that the others reject it. (3) The docs no longer call the mapping a static table. Still blocked on a prefixmaps release (below). test_generators + test_scripts + test_prefixes + test_sqlite.py: 2093 passed, 54 skipped; the one local failure, test_datadict_personinfo (metamodel version line), fails on main locally too.

2026-10-23: rebased onto main (a1593d16c) as a single commit again (the GitHub merge commit ff4883929 is gone). Conflicts in packages/linkml/pyproject.toml (the pydantic and pyjsg bumps next to the prefixmaps git pin) and uv.lock, which was regenerated from main's lock with uv lock; only the prefixmaps entries differ from main. No code change. The Docker job fails on this branch because the image has no git for the prefixmaps pin; it passes once prefixmaps publishes the fix and the pin becomes a version bound.

Warning

Blocked: depends on an unreleased prefixmaps, and this breaks the Docker build.

packages/linkml/pyproject.toml currently carries a git pin:

"prefixmaps @ git+https://github.com/linkml/prefixmaps@7543515",

plus [tool.hatch.metadata] allow-direct-references = true to permit it.
The fix it depends on (linkml/prefixmaps#82 — the gx prefix mapping to a
dead Graphite/Synaptica namespace instead of Gaia-X) is merged in the
prefixmaps main branch but has not been released: the latest version on
PyPI is still 0.2.6 (2024-10-17).

This is not just a policy problem. The git pin is baked into the built wheel's
metadata, so the Docker image build fails: pip install has to fetch the
dependency from GitHub, and the base image has no git.

ERROR: Cannot find command 'git' - do you have 'git' installed and in your PATH?
ERROR: process "/bin/sh -c pip install /tmp/linkml-whl/*.whl" did not complete
       successfully: exit code: 1

That is the one failing check on this PR; every other job is green.

Do not merge until prefixmaps cuts a release. Then the pin becomes
prefixmaps >= 0.2.8, allow-direct-references is removed, and uv.lock is
regenerated — both spots are marked TODO(prefixmaps-0.2.8). The Docker
failure resolves itself at that point, since the dependency comes from PyPI.


Summary

Adds an opt-in --normalize-prefixes flag to gen-owl, gen-shacl, gen-jsonld-context and gen-jsonld, which forwards it to its inline context. The flag renames non-standard prefix aliases for well-known namespaces to their conventional names, for example sdo: → schema: and dce: → dc:. Namespace IRIs never change; only prefix labels do.

Where the names come from

well_known_prefix_map() derives the names from prefixmaps: its curated linked_data context takes precedence over its broader merged context. Where rdflib binds a namespace by default, its name wins, because rdflib serializes the output: geo, schema and wgs where prefixmaps says geosparql, sc and WGS84. There is no hand-written table. Both http://schema.org/ (from prefixmaps) and https://schema.org/ (from rdflib) map to schema. The pin explained above is needed for the gx correction in prefixmaps#82.

Behaviour

  • OWL and SHACL share normalize_graph_prefixes(). It renames schema-declared aliases first, then runtime-injected bindings, and never overwrites an HTTPS binding with its HTTP variant.
  • A conventional name that is already declared for a different namespace is left alone, with a warning.
  • The JSON-LD context renames both its prefix declarations and the CURIEs it emits.
  • The flag defaults to off, so existing output is unchanged.
  • Only the generators that honour it offer it. gen-rdf writes the schema's class_uri and prefix declarations as literal data, and gen-shex's RDF already uses rdflib's names, so neither emits an alias to rename.

How was this tested?

test_normalize_prefixes.py (27 tests) and three tests in test_jsonldcontextgen.py cover each generator, the HTTP/HTTPS variants, collisions, and agreement between the generators. The full compiler suite passes after rebasing onto main (12,421 passed).

test_csv_limit now restores csv.field_size_limit. It left the limit at 20, so prefixmaps' CSV loading failed in any later test on the same pytest-xdist worker. That produced intermittent failures, reproducible by running test_csv_limit[20-1] before these tests.

Checklist

  • My code follows the contributor guidelines
  • I have added tests that prove my fix/feature works
  • Existing tests pass locally with my changes

@jdsika
jdsika force-pushed the fix/normalize-well-known-prefixes branch 3 times, most recently from 42e007b to 71db97f Compare March 26, 2026 17:16
@jdsika jdsika changed the title fix(generators): normalize well-known prefixes in output fix(generators): add --normalize-prefixes flag for well-known prefix names Mar 26, 2026
@jdsika
jdsika force-pushed the fix/normalize-well-known-prefixes branch 8 times, most recently from 60f62c4 to 1b36dc0 Compare April 2, 2026 09:38
@jdsika

jdsika commented Apr 2, 2026

Copy link
Copy Markdown
Author

🔍 Adversarial Review — PR #4

Summary

Solid feature with thorough test coverage, but contains a confirmed race condition between Phase 1 and Phase 2 of normalize_graph_prefixes() that can silently swap namespace URIs (HTTPS → HTTP) for schema.org. The cross-PR conflict with PR #1 (well_known_prefix_map() defined differently in both) also needs a merge strategy.

🐛 Bugs & Issues

1. Phase 2 overwrites Phase 1 normalization for HTTP/HTTPS variants (Confirmed)

When a schema declares sdo: https://schema.org/ and the metamodel runtime injects schema: http://schema.org/, the following sequence occurs:

  • Phase 1 correctly rebinds sdo → schema: https://schema.org/ (user's HTTPS)
  • After Phase 1, the graph still contains schema1: http://schema.org/ (orphaned runtime HTTP binding)
  • Phase 2 sees schema1: http://schema.org/, looks up wk["http://schema.org/"] = "schema", checks collision ("schema" not in schema_prefixes — user declared sdo, not schema), and rebinds schema: http://schema.org/ with override=True, replace=True
  • This silently replaces the Phase 1 HTTPS binding with the HTTP variant

Reproduced locally with rdflib 7.x:

from rdflib import Graph, Namespace, URIRef
g = Graph()
g.bind('schema', Namespace('http://schema.org/'))      # metamodel
g.namespace_manager.bind('sdo', URIRef('https://schema.org/'))  # user

# Phase 1
g.bind('schema', Namespace('https://schema.org/'), override=True, replace=True)
# → schema: https://schema.org/ ✅, schema1: http://schema.org/ (orphan)

# Phase 2
g.bind('schema', Namespace('http://schema.org/'), override=True, replace=True)
# → schema: http://schema.org/ ❌  (HTTPS binding silently lost!)

Fix: Phase 2 should check whether std_pfx is already bound in the graph (not just schema_prefixes) before rebinding:

# In Phase 2, before rebinding:
current_ns_for_std = dict(graph.namespaces()).get(std_pfx)
if current_ns_for_std and str(current_ns_for_std) != ns_str:
    continue  # std_pfx already claimed by Phase 1 for a different URI

2. test_sdo_normalised_to_schema may be passing for the wrong reason

The test asserts pfx["schema"] == "https://schema.org/", but if Phase 2 overwrites this with HTTP, the test should fail. If it passes, it may be because the metamodel's emit_prefixes don't include schema for the test schema, meaning the HTTP binding is never injected — hiding the bug in production schemas that DO trigger both bindings. The test should explicitly set up both HTTP and HTTPS bindings to exercise Phase 2 properly.

⚠️ Concerns

3. Cross-PR conflict with PR #1 — well_known_prefix_map() defined differently

PR #1 PR #4
Approach Dynamic: {str(ns): str(pfx) for pfx, ns in RdfGraph().namespaces()} Static: MappingProxyType({...}) with ~30 entries
rdflib dependency Changes with rdflib version Frozen, version-independent
HTTP/HTTPS variants Only what rdflib provides Explicitly includes both

PR #4's static approach is explicitly designed to solve a problem with PR #1's dynamic approach (rdflib defaults changed between 6.x and 7.x). These will conflict on merge. Recommendation: merge PR #4 first, or rebase PR #1 to use PR #4's static map.

4. Cross-PR conflict — flag forwarding in jsonldgen.py

PR #4 forwards both normalize_prefixes AND deterministic to the embedded ContextGenerator:

for flag in ("normalize_prefixes", "deterministic"):
    if hasattr(self, flag):
        context_kwargs.setdefault(flag, getattr(self, flag))

PR #1 adds deterministic handling in a different location (post-serialization in end_schema). PR #4's forwarding of deterministic creates a dependency on PR #1's field existing. The hasattr guard prevents a crash, but semantically PR #4 shouldn't forward a flag it doesn't define. Either:

5. PR title says fix: but commit says feat: — which is it?

The commit message uses feat(generators): while the PR title uses fix(generators):. This is a new opt-in flag, which is a feature — feat: is correct per conventional commits.

6. Static map maintenance burden

The _WELL_KNOWN_PREFIX_MAP needs manual updates when rdflib adds new curated defaults. The test_matches_rdflib_defaults test catches additions but not removals. Consider adding a reverse check: entries in the static map that are NOT in rdflib's defaults could indicate stale entries.

🧪 Test Coverage Assessment

Strong coverage (18 tests across 7 classes):

  • ✅ OWL, SHACL, JSON-LD context normalization (sdo→schema, dce→dc)
  • ✅ HTTP/HTTPS schema.org variants
  • ✅ Collision detection with warnings (caplog)
  • ✅ Cross-generator consistency
  • ✅ Static map integrity and copy safety
  • ✅ JSONLDGenerator flag forwarding
  • ✅ Flag-off backward compatibility

Gaps:

  • ❌ No test for normalize_prefixes + exclude_imports interaction
  • ❌ No test for empty schema (zero prefixes) — is _prefix_remap.clear() safe?
  • ❌ No test that explicitly sets up both HTTP and HTTPS runtime bindings to exercise Phase 2 overwrite (the scenario from Bug feat(generators): add --deterministic flag with hybrid RDFC-1.0 + rdflib serialization #1)
  • ❌ No test for add_prefix being called before __post_init__ completes (defensive, but _prefix_remap access would crash)
  • ❌ CURIE remapping test (test_normalize_prefixes_curie_remapping) only checks that Person @id starts with schema: — doesn't verify full_name slot's @id uses schema:name instead of sdo:name

📋 Fix Plan

  1. Fix Phase 2 overwrite bug — add a graph-binding check before rebinding in Phase 2 to prevent overwriting Phase 1's normalization
  2. Add regression test — explicitly set up a graph with both schema: http://schema.org/ and sdo: https://schema.org/, run normalize, and assert the HTTPS binding survives
  3. Remove "deterministic" from forwarding loop — or gate it behind hasattr only (already done) but add a comment explaining the cross-PR dependency
  4. Fix PR title — change fix: to feat: for consistency with the commit
  5. Add empty-schema test — verify no crash when schema has zero prefixes
  6. Add exclude_imports interaction test — verify normalization works when imports are excluded
  7. Strengthen CURIE remapping test — assert that slot URIs (not just class URIs) are remapped in the context

✅ What's Good

  • Excellent documentation: the 3-case handling in visit_schema is well-commented with case numbers
  • Defensive _prefix_remap initialization: correctly placed before super().__post_init__() with clear comment explaining why
  • Collision detection: warning on skip is the right behavior — no silent data loss
  • MappingProxyType for the static map: prevents accidental mutation
  • well_known_prefix_map() returns a copy: callers can't corrupt the canonical map
  • Phase 2 concept is sound: cleaning up runtime-injected orphan bindings is the right idea, it just needs the overwrite guard
  • Cross-generator consistency test: ensures OWL, SHACL, and JSON-LD agree — excellent
  • Backward compatibility: flag defaults to off, existing behavior preserved

@jdsika
jdsika force-pushed the fix/normalize-well-known-prefixes branch from d9fb738 to dd0ce0c Compare April 2, 2026 15:24
@jdsika
jdsika force-pushed the fix/normalize-well-known-prefixes branch 3 times, most recently from 8d098e9 to ef47b47 Compare April 24, 2026 20:26
@jdsika
jdsika force-pushed the fix/normalize-well-known-prefixes branch 11 times, most recently from 8dc3ece to 260a97e Compare May 11, 2026 15:49
@jdsika
jdsika force-pushed the fix/normalize-well-known-prefixes branch from 260a97e to e56ca65 Compare May 11, 2026 16:15
@jdsika
jdsika force-pushed the fix/normalize-well-known-prefixes branch from e56ca65 to 5f17e19 Compare June 9, 2026 15:21
@rmessaou
rmessaou force-pushed the fix/normalize-well-known-prefixes branch 2 times, most recently from ce55ccb to 909eb7d Compare July 8, 2026 08:49
@jdsika jdsika self-assigned this Jul 11, 2026
@jdsika
jdsika force-pushed the fix/normalize-well-known-prefixes branch from a4b329c to 32821d0 Compare September 11, 2026 12:33
@jdsika
jdsika force-pushed the fix/normalize-well-known-prefixes branch from 32821d0 to 0342283 Compare October 2, 2026 11:48
@jdsika jdsika changed the title fix(generators): add --normalize-prefixes flag for well-known prefix names feat(generators): add --normalize-prefixes flag for well-known prefix names Oct 13, 2026
jdsika added a commit that referenced this pull request Oct 14, 2026
Review update from #4: test_csv_limit left the process-wide CSV field
size limit at 20, so later tests on the same xdist worker that read
prefixmaps' CSV data failed intermittently.

Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
@jdsika
jdsika force-pushed the fix/normalize-well-known-prefixes branch from 0342283 to 316b430 Compare October 14, 2026 07:34
jdsika added a commit that referenced this pull request Oct 14, 2026
Review update from #4: test_csv_limit left the process-wide CSV field
size limit at 20, so later tests on the same xdist worker that read
prefixmaps' CSV data failed intermittently.

Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
@jdsika
jdsika force-pushed the fix/normalize-well-known-prefixes branch from ff48839 to 9ede52d Compare October 23, 2026 08:49
… names

Add an opt-in --normalize-prefixes flag to the OWL, SHACL, JSON-LD
context and JSON-LD generators that renames non-standard prefix aliases
to their well-known names. The names come from the prefixmaps
linked_data context (curated) layered over its merged context, and where
rdflib binds a namespace by default, from rdflib, which serializes the
output (geo, schema, wgs where prefixmaps says geosparql, sc, WGS84).

- Both http://schema.org/ and https://schema.org/ map to 'schema'.
- OWL and SHACL share normalize_graph_prefixes(): phase 1 renames
  schema-declared prefixes, phase 2 cleans up runtime-injected bindings
  without overwriting HTTPS bindings with HTTP variants.
- A standard name already declared for another namespace is skipped
  with a warning.

The flag defaults to off, preserving existing behaviour, and only the
generators that honour it offer it: gen-rdf writes the schema's class_uri
and prefix declarations as literal data and gen-shex's RDF uses rdflib's
names already, so neither emits an alias to rename. The documentation
for the flag is in docs/generators/owl.rst.

test_csv_limit set csv.field_size_limit(20) without restoring it, so any
later test on the same worker that read prefixmaps' CSV data failed; it
now restores the limit.

Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
@jdsika
jdsika force-pushed the fix/normalize-well-known-prefixes branch from 9ede52d to 29098c8 Compare October 23, 2026 11:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant