Dedicated store and pipeline for citation links detected by the Sefaria linker, kept as stable, delta-friendly, ref-based artifacts that the SeforimLibrary build turns into clickable in-app links.
Full design: SeforimLibrary/LINKER_DELTA_PLAN.md and
SeforimLibrary/LINKER_IMPLEMENTATION_STAGES.md.
A link has two ends:
- source — a span in a book (line + char range of the citation phrase),
- target — a Sefaria ref (
"Yoma 55a:5"), a stable logical id.
We store the ref, not a resolved target line. The build resolves it to a line
every time (via resolveRefs). Because Sefaria line ids are keyed on REF:$heRef,
a Sefaria content update keeps the same line id → the link survives with zero
delta churn. The expensive step (NER) runs only on source lines that are new or
changed. An exact release-only line baseline reuses prior artifacts for identical
lines, including lines that moved inside an edited book. A book with "שם" (ibid) is
re-NERed by line only when the edit cannot move its ibid history: every changed line
is free of NER entities, before and after, and the entity lines keep their order.
Otherwise the whole book is linked again.
artifacts/<source_name>/<canonical_he_title>.jsonl # one file per source book
baseline/snapshot_hashes.json # per-book content hash of the snapshot last linked
line-baseline/ # release-only line fingerprints, NER-entity flags, artifact digests
meta.json # lineage of the last run
schema/artifact.schema.json # JSON Schema for one record
src/linker_artifact.py # THE format contract (shared code)
tests/ # contract tests
examples/ # a validated sample artifact
artifacts/ and line-baseline/ are generated release payload state and are not
committed. baseline/ and meta.json are committed only after the immutable
payload is published.
One JSON object per line (see schema/artifact.schema.json):
{"book_key": {"source_name": "MoreBooks", "canonical_he_title": "חזון איש"},
"line_index": 28, "line_index_base": 0,
"start": 37, "end": 48,
"target_ref": "Psalms 16:8"}book_key— the source book identity, identical to the generator'sBookKey(sourceName, canonicalHeTitle). Derived from the DB as(source.name, COALESCE(book.heRef, book.title)). Collision-free (verified across the full corpus).line_index— 0-based, equalsline.lineIndexin the DB.start/end— offsets of the citation span into the stored line content (line.content, HTML included). The build maps raw→visible viacountVisibleChars.target_ref— canonical English ref (Ref.normal()). The only thing the build needs to resolve the target.- Ambiguous citations are dropped at detection time — artifacts hold only unambiguous links.
src/linker_artifact.py is the single source of truth for this format and the
artifacts/<source>/<title>.jsonl filename scheme. Import it — never re-implement.
python3 -m unittest discover -s tests -v