Skip to content

feat(generators): add --normalize-prefixes flag for well-known prefix names - #3446

Draft
jdsika wants to merge 1 commit into
linkml:mainfrom
ASCS-eV:fix/normalize-well-known-prefixes
Draft

jdsika wants to merge 1 commit into
linkml:mainfrom
ASCS-eV:fix/normalize-well-known-prefixes

Conversation

@jdsika

@jdsika jdsika commented Apr 24, 2026 •

Copy link
Copy Markdown
Contributor

2026-10-23, second push (review): (1) the hand-written override table is gone: where prefixmaps' names differ from the names rdflib binds by default (geosparql/geo, sc/schema, WGS84/wgs), rdflib's win, read from Graph(bind_namespaces="rdflib"); the resulting map is identical (16,440 entries). (2) The option was added to every generator's command line and silently ignored by most; it is now declared only on gen-owl, gen-shacl, gen-jsonld-context and gen-jsonld, which honour it, with tests that the others reject it. (3) The docs no longer call the mapping a static table. Still blocked on a prefixmaps release (below). test_generators + test_scripts + test_prefixes + test_sqlite.py: 2093 passed, 54 skipped; the one local failure, test_datadict_personinfo (metamodel version line), fails on main locally too.

2026-10-23: rebased onto main (a1593d16c) as a single commit again (the GitHub merge commit ff4883929 is gone). Conflicts in packages/linkml/pyproject.toml (the pydantic and pyjsg bumps next to the prefixmaps git pin) and uv.lock, which was regenerated from main's lock with uv lock; only the prefixmaps entries differ from main. No code change. The Docker job fails on this branch because the image has no git for the prefixmaps pin; it passes once prefixmaps publishes the fix and the pin becomes a version bound.

Warning

Blocked: depends on an unreleased prefixmaps, and this breaks the Docker build.

packages/linkml/pyproject.toml currently carries a git pin:

"prefixmaps @ git+https://github.com/linkml/prefixmaps@7543515",

plus [tool.hatch.metadata] allow-direct-references = true to permit it.
The fix it depends on (linkml/prefixmaps#82 — the gx prefix mapping to a
dead Graphite/Synaptica namespace instead of Gaia-X) is merged in the
prefixmaps main branch but has not been released: the latest version on
PyPI is still 0.2.6 (2024-10-17).

This is not just a policy problem. The git pin is baked into the built wheel's
metadata, so the Docker image build fails: pip install has to fetch the
dependency from GitHub, and the base image has no git.

ERROR: Cannot find command 'git' - do you have 'git' installed and in your PATH?
ERROR: process "/bin/sh -c pip install /tmp/linkml-whl/*.whl" did not complete
       successfully: exit code: 1

That is the one failing check on this PR; every other job is green.

Do not merge until prefixmaps cuts a release. Then the pin becomes
prefixmaps >= 0.2.8, allow-direct-references is removed, and uv.lock is
regenerated — both spots are marked TODO(prefixmaps-0.2.8). The Docker
failure resolves itself at that point, since the dependency comes from PyPI.


Summary

Adds an opt-in --normalize-prefixes flag to gen-owl, gen-shacl, gen-jsonld-context and gen-jsonld, which forwards it to its inline context. The flag renames non-standard prefix aliases for well-known namespaces to their conventional names, for example sdo: → schema: and dce: → dc:. Namespace IRIs never change; only prefix labels do.

Where the names come from

well_known_prefix_map() derives the names from prefixmaps: its curated linked_data context takes precedence over its broader merged context. Where rdflib binds a namespace by default, its name wins, because rdflib serializes the output: geo, schema and wgs where prefixmaps says geosparql, sc and WGS84. There is no hand-written table. Both http://schema.org/ (from prefixmaps) and https://schema.org/ (from rdflib) map to schema. The pin explained above is needed for the gx correction in prefixmaps#82.

Behaviour

  • OWL and SHACL share normalize_graph_prefixes(). It renames schema-declared aliases first, then runtime-injected bindings, and never overwrites an HTTPS binding with its HTTP variant.
  • A conventional name that is already declared for a different namespace is left alone, with a warning.
  • The JSON-LD context renames both its prefix declarations and the CURIEs it emits.
  • The flag defaults to off, so existing output is unchanged.
  • Only the generators that honour it offer it. gen-rdf writes the schema's class_uri and prefix declarations as literal data, and gen-shex's RDF already uses rdflib's names, so neither emits an alias to rename.

How was this tested?

test_normalize_prefixes.py (27 tests) and three tests in test_jsonldcontextgen.py cover each generator, the HTTP/HTTPS variants, collisions, and agreement between the generators. The full compiler suite passes after rebasing onto main (12,421 passed).

test_csv_limit now restores csv.field_size_limit. It left the limit at 20, so prefixmaps' CSV loading failed in any later test on the same pytest-xdist worker. That produced intermittent failures, reproducible by running test_csv_limit[20-1] before these tests.

Checklist

  • My code follows the contributor guidelines
  • I have added tests that prove my fix/feature works
  • Existing tests pass locally with my changes

@codecov

codecov Bot commented Apr 24, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 94.94949% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.89%. Comparing base (a1593d1) to head (29098c8).

Files with missing lines Patch % Lines
packages/linkml/src/linkml/utils/generator.py 93.47% 1 Missing and 2 partials ⚠️
...s/linkml/src/linkml/generators/jsonldcontextgen.py 94.44% 0 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3446      +/-   ##
==========================================
+ Coverage   81.62%   84.89%   +3.27%     
==========================================
  Files         162      162              
  Lines       20518    20612      +94     
  Branches     4366     4390      +24     
==========================================
+ Hits        16748    17499     +751     
+ Misses       2876     2171     -705     
- Partials      894      942      +48     
Flag Coverage Δ
linkml 81.67% <94.94%> (+0.05%) ⬆️
runtime 81.65% <94.94%> (+0.05%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Comment thread packages/linkml/src/linkml/utils/generator.py Outdated
@jdsika

jdsika commented Apr 29, 2026

Copy link
Copy Markdown
Contributor Author

Status: Draft — awaiting upstream dependency

This PR is moved back to draft pending the merge of an upstream fix:

Once that lands and a new prefixmaps release is cut, we can evaluate whether to:

  1. Replace the hardcoded _WELL_KNOWN_PREFIX_MAP with a runtime call to load_context("merged").as_inverted_dict()
  2. Or keep the static map (rdflib 7.x defaults) but document prefixmaps as the canonical upstream source

Additionally:

The implementation and tests in this PR are complete and passing — no code changes needed beyond the upstream resolution.

@jdsika
jdsika force-pushed the fix/normalize-well-known-prefixes branch from ef47b47 to 3a71506 Compare May 7, 2026 13:55
@jdsika
jdsika force-pushed the fix/normalize-well-known-prefixes branch 2 times, most recently from 076b00d to 1116533 Compare May 7, 2026 17:12
@jdsika
jdsika force-pushed the fix/normalize-well-known-prefixes branch 9 times, most recently from 260a97e to e56ca65 Compare May 11, 2026 16:15
@jdsika
jdsika force-pushed the fix/normalize-well-known-prefixes branch from e56ca65 to 5f17e19 Compare June 9, 2026 15:21
@rmessaou
rmessaou force-pushed the fix/normalize-well-known-prefixes branch 2 times, most recently from ce55ccb to 909eb7d Compare July 8, 2026 08:49
@jdsika

jdsika commented Sep 8, 2026

Copy link
Copy Markdown
Contributor Author

@kevinschaper do you think we are ready to release prefixmaps == 0.2.8

Is there anything still needed for the prefixmaps repo?

@jdsika
jdsika force-pushed the fix/normalize-well-known-prefixes branch from a4b329c to 32821d0 Compare September 11, 2026 12:33
@jdsika
jdsika force-pushed the fix/normalize-well-known-prefixes branch from 32821d0 to 0342283 Compare October 2, 2026 11:48
@jdsika
jdsika force-pushed the fix/normalize-well-known-prefixes branch from 0342283 to 316b430 Compare October 14, 2026 07:34
@jdsika
jdsika force-pushed the fix/normalize-well-known-prefixes branch from ff48839 to 9ede52d Compare October 23, 2026 08:49
… names

Add an opt-in --normalize-prefixes flag to the OWL, SHACL, JSON-LD
context and JSON-LD generators that renames non-standard prefix aliases
to their well-known names. The names come from the prefixmaps
linked_data context (curated) layered over its merged context, and where
rdflib binds a namespace by default, from rdflib, which serializes the
output (geo, schema, wgs where prefixmaps says geosparql, sc, WGS84).

- Both http://schema.org/ and https://schema.org/ map to 'schema'.
- OWL and SHACL share normalize_graph_prefixes(): phase 1 renames
  schema-declared prefixes, phase 2 cleans up runtime-injected bindings
  without overwriting HTTPS bindings with HTTP variants.
- A standard name already declared for another namespace is skipped
  with a warning.

The flag defaults to off, preserving existing behaviour, and only the
generators that honour it offer it: gen-rdf writes the schema's class_uri
and prefix declarations as literal data and gen-shex's RDF uses rdflib's
names already, so neither emits an alias to rename. The documentation
for the flag is in docs/generators/owl.rst.

test_csv_limit set csv.field_size_limit(20) without restoring it, so any
later test on the same worker that read prefixmaps' CSV data failed; it
now restores the limit.

Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants