Neural typography. Reusable semantic cues. A serious experiment in memory.

Examples · Run locally · Architecture · Research protocol · Licensing

Synesthesia AI turns ordinary text into a richer typographic reading surface. The words stay intact; their color, weight, width, slant and arrangement carry an additional visual code. Optional motion and tonal sound add another layer.

The ambition is simple and deliberately large: read a book once and remember everything. That is the research target, not a promise. The project builds both the reader and the delayed-recall experiments needed to find out whether the idea works.

The following images are captures of the running application using real TRIBE predictions. The passage is an original fictional example, not a testimonial or a memory result.

The neural views are designed mappings of model features, not a reconstruction of someone's personal synesthesia. The stable semantic view is a separate, explicit policy. Example provenance and the recorded neural response accompany the images.

• Import TXT, Markdown, DRM-free EPUB and text-layer PDF.
• Navigate source sections with character-offset tracking.
• Compare with the original without substituting a summary.
• See warnings for omitted graphics, unsupported structure and extraction limits.
• Keep reading plain text when neural inference is unavailable.

Slow, bounded text motion is optional and respects reduced-motion preferences. Sound is a user-started sequence of tonal cues, not narration. Both stop when appropriate, including source/original changes and hidden tabs.

The system keeps three things distinct:

1. Evidence and response models. TRIBE v2 predicts population-level cortical responses. ROAMM, ZuCo and DERCo support reading-signal baselines. A pinned local MiniLM encoder supplies semantic embeddings.
2. Presentation policies. Fixed projections, variable fonts and a stable cue grammar turn those signals into typography. A model prediction does not establish the best intervention.
3. Outcome measurement. Randomized reading conditions, delayed assessments and book-held-out analysis test what the presentation actually does for recall.

Source, prediction, alignment and presentation hashes support reproducibility. They check internal integrity; they are not proof of scientific efficacy or authentication of a remote server.

The stable reader can run without the large TRIBE model. You need Python 3.11+, Node.js 20.19+, uv and npm.

git clone https://github.com/lucadominguez/synesthesia-ai.git
cd synesthesia-ai
make bootstrap
make build
uv run --all-packages python scripts/install-semantic-model.py

export SYNESTHESIA_REPO_ROOT="$PWD"
export SYNESTHESIA_STATIC_ROOT="$PWD/apps/web/dist"
uv run --all-packages uvicorn synesthesia_api.runtime:create_runtime_app \
  --factory --host 127.0.0.1 --port 8000

Open http://127.0.0.1:8000/research, import a source and keep Stable semantic code selected. The installer downloads pinned public semantic weights and verifies their checksums.

Continuous and spatial neural views need the full research stack. They do not silently substitute random styles when TRIBE or its baselines are missing. Full setup, resource requirements and verification.

The memory workflow is built around delayed outcomes rather than an attractive screenshot:

• Separate strict one-pass, trained one-pass and assisted tracks.
• Randomized plain, continuous, spatial and stable presentations with matched exposure budgets.
• One selected delayed horizon per source: 1, 7 or 30 days.
• Unaided responses locked before cloze cues are shown.
• Exact cloze scores kept separate from self-rated understanding, application and unsupported details.
• Exploratory personalization gated on sufficient outcomes and whole-book holdouts.

Motion and sound are off in controlled trials. Their memory effects cannot be inferred from a typography-only result. The live learner stays untrained until eligible human outcomes exist and the fitting gates are satisfied.

For a first-exposure experiment, enroll an unfamiliar source before opening it for casual reading. Protocol, eligibility gates and limitations.

uv run --all-packages python scripts/install-semantic-model.py
make test lint build
npm audit --include=dev --audit-level=moderate

The repository includes Python, JavaScript and desktop/mobile browser tests. Software fixtures test error paths and future assessment states; they are not presented as participant data. See the verification guide for clean-clone browser tests and optional real-model checks.

This is a research prototype, not a proven memory aid. One-pass whole-book recall, improved retention and induction of clinical synesthesia have not been demonstrated by this project. English semantic families, text extraction, CPU latency and model licensing impose practical limits.

TRIBE v2 weights are CC BY-NC 4.0. MiniLM weights are Apache-2.0. Bundled fonts retain their SIL Open Font License notices. There is currently no blanket software license for project-authored code; publishing this repository does not relicense third-party components. Full notices and scope.

The book-and-ribbons cover and architecture graphic are documentation illustrations. The reader examples and motion capture are actual application output. Asset sources and reproduction.