Skip to content

Proposal: adopt schema-driven map/archetype sources and compiled runtime caches #166

Closed as not planned
2 of 5 issues completed
Closed as not planned

Description

@zoeyrose

Summary

Replace the ad hoc line-oriented map/archetype authoring syntax with one
schema-driven content model, then compile that model into versioned binary
runtime artifacts.

Recommended split:

  • Authored source: a constrained YAML 1.2 subset (*.map.yaml and
    *.arc.yaml) validated against checked-in JSON Schemas.
  • Tool/API interchange: deterministic JSON using the same logical schema.
  • Server runtime: explicit, bounded binary archetype and per-map formats;
    never a dump of C structs.
  • Development fallback: compile a missing/stale map artifact on first use
    into a content-addressed on-disk cache.
  • Production/release path: compile and validate every map during content
    collection/packaging, so a normal server never parses authored YAML in the
    game process.

This is more than a syntax conversion. The durable improvement is a typed
intermediate representation and schema shared by the server compiler, native
editor/content CLI from seraphinastudios/tilewright-editor#13, checkers, audits, and smaller tools. YAML/JSON
alone would still leave the current duplicated and inconsistent parsers in
place, and parsing JSON directly in the server would not deliver the desired
startup/first-entry performance.

What exists today

The current source tree has two related but distinct paths.

Archetypes

  • Nested arch/**/*.arc files are authored source.
  • tools/compilers/__init__.py:ArchetypesCompiler merely concatenates them
    into the generated arch/archetypes / server/lib/archetypes aggregate.
  • server/src/server/arch.c:arch_init() loads the complete aggregate during
    server startup. It parses the file twice: arch_pass_first() constructs and
    hashes archetypes, then arch_pass_second() resolves cross-archetype and
    treasure/artifact relationships.
  • Keeping archetypes resident is intentional today: maps clone them by
    pointer, many subsystems cache required archetype pointers, and startup code
    iterates the complete table. Archetypes therefore need a faster compiled
    bundle, not a map-style eviction policy.

Maps

  • Authored maps are extensionless files beginning with arch map.
  • server/src/loaders/map_header.l parses the header and
    server/src/loaders/object.l parses each nested arch ... end object.
  • ready_map_name() / load_original_map() already load authored maps lazily
    on first visit. Server startup does not parse every map.
  • load_objects() does important runtime work after decoding: archetype
    cloning, callbacks, connection registration, speed initialization,
    auto-apply, treasure generation, map insertion, monster setup, animation,
    and lighting. A compiler must not pre-execute or accidentally bypass these
    semantics.
  • check_active_maps() already evicts inactive instantiated maps. Before
    eviction, swap_map() writes mutable runtime state with new_save_map() to
    server/data/tmp, and load_temporary_map() later reloads it. These mutable
    snapshots are not the same thing as a cache of immutable authored content.
  • Player-unique maps, random maps, style maps, unique-item .v00 files, and
    dynamic maps exercise additional load/save modes that a replacement must
    preserve explicitly.

The current corpus is large enough for parser duplication and map-entry
latency to matter. A scan of this tree found:

  • 3,631 authored maps, about 59.4 MB total;
  • about 2.04 million arch object records in those maps;
  • 1,351 .arc files, about 593 KB total; and
  • 3,781 top-level Object archetype definitions.

Parsing is duplicated across the server Flex loaders, old and Qt map checkers,
mapset, worldviewer, world_content_audit.py, farming_analyzer.py, and
bot/navigation tooling. Those consumers implement different subsets of the
grammar. The catch-all rule in object.l also turns unknown fields into custom
key/value data, which is useful for scripts but makes misspelled engine fields
indistinguishable from intentional extensions.

Proposed decision

1. Define one typed content model before changing syntax

Add checked-in, versioned schemas and a common semantic model for:

  • map headers, flags, regions, tiling, entry coordinates, messages, and source
    metadata;
  • archetype definitions, multipart parts, inheritance/clone references, and
    default inventory;
  • placed map objects, typed overrides, nested inventory, coordinates, and
    authored order;
  • named faces, animations, treasures, artifacts, object types, attacks,
    protections, flags, layers/surfaces, and other cross-resource references;
    and
  • explicitly namespaced custom properties for scripts/plugins.

Suggested locations (exact names can be refined with seraphinastudios/tilewright-editor#13):

schemas/content/atrinik-map-v1.schema.json
schemas/content/atrinik-archetypes-v1.schema.json
schemas/content/atrinik-diagnostics-v1.schema.json

Generate field metadata, documentation, editor property descriptions, and
loader/compiler tables from one source where practical. Do not hand-maintain a
new schema independently from the several hundred rules in object.l.

Unknown standard fields should be errors. Dynamic script/plugin data should
remain possible only under an explicit custom object/map so a typo such as
dificulty cannot silently become a key/value property.

The semantic model must retain source spans and comments for editor
diagnostics and minimal edits. In-memory APIs should expose typed values and
stable logical node identities without exposing YAML-library nodes to every
consumer.

2. Use constrained YAML for authored source

YAML is recommended over plain JSON for repository-authored maps because the
files are frequently reviewed or changed by humans, comments are valuable,
and two million object entries would make JSON punctuation noisy. JSON remains
the stable machine/CLI interchange format because JSON Schema, automation,
and cross-language tooling support are stronger there.

Use a deliberately small YAML 1.2 profile:

  • UTF-8, deterministic formatting, string keys, and JSON-compatible scalar
    types only;
  • reject anchors, aliases, merge keys, custom tags, duplicate keys, non-string
    keys, and implementation-specific implicit types;
  • bound nesting, collection sizes, scalar lengths, aliases (zero), total
    decoded bytes, object counts, and diagnostic counts;
  • keep comments and source spans in the content model; and
  • require a top-level format identity such as format: atrinik-map/v1.

Illustrative map source:

format: atrinik-map/v1
map:
  name: Emergency Map
  size: [1, 1]
  difficulty: 1
  darkness: -1
  flags: [no_save]
  region: creation
  note: |
    Created: 2009-09-22 Zoey Rose
    Modified: 2015-04-19 Zoey Rose
objects:
  - archetype: floor_ostone1
    at: [0, 0]
  - archetype: exit
    at: [0, 0]
    properties:
      destination: /shattered_islands/world_2_68
      destination_position: [3, 7]

Illustrative archetype source:

format: atrinik-archetypes/v1
archetypes:
  - id: quill_pen
    name: quill
    type: misc_object
    face: quill_pen.101
    render:
      layer: item
    flags: [no_pick]
    inventory: []
    parts: []

The actual schema must be derived from a complete grammar inventory and
corpus, not these simplified examples. Named enums/flags and grouped concepts
are preferred to unexplained integer fields where a stable semantic name
exists, but migration must preserve every current value exactly.

Before committing to YAML, implement a proof of concept over a representative
corpus and verify comment/source-span retention, deterministic output,
cross-language parsing, and bounded hostile-input behavior. If available YAML
libraries cannot meet those requirements without maintaining another parser,
use JSONC as the authored syntax while retaining the same schema and binary
plan. The content model, not the surface syntax, is authoritative.

3. Compile archetypes into one binary bundle

Replace concatenation with a real collection/compiler step that parses every
authored archetype, validates all fields/references, and emits one immutable
bundle, for example server/lib/archetypes.abin.

The bundle should contain normalized typed records, a deduplicated string
table, multipart/default-inventory relationships, and stable indices for
resolved references. It should have:

  • magic, format version, byte-order marker, header length, total length, and
    bounded record/table counts;
  • schema/compiler identity and a content-bundle digest;
  • explicit integer widths and length-delimited tables;
  • internal offsets/indices validated before allocation or pointer linking; and
  • a digest over the complete artifact.

It must never contain raw object, archetype_t, pointers, padding, hash-table
state, or other compiler/ABI-dependent memory images.

arch_init() should instantiate and link the full in-memory archetype table
from this bundle. The goal is to remove text tokenization and the second source
parse, not to make archetypes individually lazy before measurements and
call-site redesign show that to be useful.

4. Compile each immutable map independently

Define a separate binary map artifact, for example .amap, containing:

  • magic/version/lengths and strict upper bounds;
  • the logical map path and normalized typed header;
  • a deduplicated string/reference table;
  • a deterministic pre-order object table with parent/child relationships,
    authored sequence, archetype reference, coordinates, and typed override
    records;
  • the source digest, schema/compiler identity, and required content-bundle
    digest; and
  • a complete artifact digest.

Use explicit, portable encoding with validated offsets/lengths. Do not serialize
C layouts. Compression should be optional and measurement-driven; compact
tables plus the OS page cache may be faster and simpler than decompressing
every visit. If compression is added, use a bounded existing codec and defend
against expansion bombs.

The binary decoder should produce the same typed intermediate map as the
source compiler. A single instantiation path then applies current load-mode
semantics (MAP_ORIGINAL, MAP_STYLE, MAP_PLAYER_UNIQUE, etc.) and performs
the runtime initialization currently embedded in load_object_buffer() and
load_objects().

This separation is important: parsing/validation/reference normalization can
happen once, but randomized speed, treasure generation, callbacks, auto-apply,
connections, lighting, and other live behavior still happen for each fresh
map instance as appropriate.

Recommended AMAP execution model: structured tables, not a free-form VM

A structured binary representation and bytecode are closely related: both
replace text tokenization and string-to-field dispatch with numeric, typed load
operations. The useful distinction is that bytecode describes commands in
sequence, while structured binary tables describe the desired map data and
relationships
declaratively.

A pure opcode stream might look like:

BEGIN_MAP width=24 height=24
BEGIN_OBJECT archetype=418 x=3 y=7
SET_INT field=HP value=20
SET_STRING_ID field=DESTINATION value=12
BEGIN_INVENTORY
BEGIN_OBJECT archetype=91
END_OBJECT
END_INVENTORY
END_OBJECT
END_MAP

The equivalent structured representation separates records:

Map header table
  width=24, height=24, object_count=2

String/reference table
  12="/shattered_islands/world_2_68"

Object table
  0: archetype=418, parent=NONE, x=3, y=7, properties=0..2
  1: archetype=91,  parent=0,    x=0, y=0, properties=2..2

Property table
  0: field=HP,          type=INT,       value=20
  1: field=DESTINATION, type=STRING_ID, value=12

For Atrinik, use a hybrid table-oriented AMAP format. Its tables are
effectively declarative load instructions, but they can be completely
preflighted before constructing or mutating live server objects. A recommended
section layout is:

AMAP header and section directory
content/schema/compiler identities and artifact digest
deduplicated UTF-8 string/reference table
typed map-header record
object table in authored/pre-order sequence
typed property/override table
optional length-delimited source/debug-span section

Each object record should contain bounded explicit-width values such as
archetype ID, parent index, child range or pre-order depth, authored sequence,
coordinates, and property range. Each property record should be a tagged union
of stable schema-generated field_id, value_type, and inline value or table
index. IDs must not be C pointers, offsetof() values, raw enum layout assumed
to be ABI-stable, or arbitrary executable callbacks.

The loader should operate in explicit stages:

  1. Validate the complete header, section directory, lengths, counts, digest,
    UTF-8 strings, object relationships, property types, field IDs, indices,
    coordinates, nesting, archetype/resource identity, and trailing bytes.
  2. Allocate the map and a bounded object-pointer vector.
  3. Walk object records sequentially, clone the referenced archetype, and apply
    typed properties through generated/shared field setters.
  4. Link inventory children and insert top-level objects in preserved authored
    order.
  5. Run the existing load-mode-specific finalization: callbacks, randomized
    speed, connections, auto-apply, treasure generation, animation, monster
    setup, lighting, and other live semantics.
  6. Release or unmap the AMAP bytes after instantiation unless a measured,
    bounded open-blob cache benefits repeated loads.

This gets the important bytecode benefit: a cache hit performs no Flex scan,
tokenization, key comparison, numeric conversion, or YAML/JSON parsing, and it
does not need to materialize a second generic document tree before loading the
map. The server walks compact records and performs already-resolved typed load
operations directly.

Structured tables are preferred over a pure stateful opcode VM here because
they make it simpler to:

  • validate the entire untrusted/corrupt artifact before any callback or
    partially instantiated map state exists;
  • check every relationship and range without simulating a begin/end stack;
  • skip unknown optional length-delimited sections while rejecting unknown
    required ones;
  • inspect, test, memory-map, index, and diagnose individual records;
  • preserve authored order and nested inventory explicitly; and
  • evolve sections without defining control flow or a general-purpose virtual
    machine.

A streaming opcode format could use slightly less temporary state, but that is
not a meaningful advantage if the table decoder walks records sequentially and
keeps only the object-pointer vector required to resolve parents. Instantiated
object state will dominate the temporary table-view cost. Benchmark both if
necessary, but the default design should remain declarative and bounded.

It is reasonable to describe these tables informally as “map-loading
bytecode,” provided the specification makes clear that they are data-only,
contain no arbitrary control flow, and are validated transactionally before
execution. No reverse AMAP-to-YAML serializer is required: YAML is the
authoritative source and AMAP is disposable compiled output. Mutable swapped
maps continue to use the separately versioned snapshot path described below.

Acceptance for this part requires demonstrating that a valid AMAP can be
validated and instantiated without invoking either Flex loader or building a
YAML/JSON DOM, while every malformed table/reference case fails before
callbacks, treasure generation, map insertion, or other observable runtime
side effects.

5. Add a two-mode compiled-map cache

Collected/release mode

Extend tools/collect.py or its replacement in seraphinastudios/tilewright-editor#13 to compile all maps and
write a small manifest mapping logical map path to artifact digest, source
digest, and required content-bundle digest. Release/server packages should ship
the manifest and compiled artifacts. Authored YAML need not be present in a
production image.

This compilation is the authoritative whole-world validation step and should
be parallelizable and incremental. It moves failures to CI/package creation
instead of the first player entering a map.

Development lazy mode

When a developer changes a map without running full collection:

  1. ready_map_name() resolves the logical path.
  2. Look up the expected artifact in the collected manifest or local cache.
  3. Verify format/compiler/content-bundle identity and source freshness.
  4. If missing or stale and source compilation is enabled, take a per-key lock,
    parse/validate/compile into a same-directory temporary file, fsync as
    appropriate, and atomically rename it into the cache.
  5. Re-check after acquiring the lock so concurrent requests/processes do not
    compile the same map twice.
  6. Decode/instantiate the artifact. Never fall back to a stale artifact after
    a compile or validation failure.

Use content-addressed keys covering at least:

map source digest
+ schema/compiler version
+ archetype/resource bundle digest
+ binary format version

Place runtime-generated artifacts under a dedicated mutable cache such as
server/data/cache/content/, never beside authored maps, collected
server/lib inputs, server/data/tmp, player data, or unique-item state.

Maintain a bounded on-disk LRU/size budget if measurements justify it. It is
safe to prune only verified cache artifacts because they are reproducible.
Instantiated mapstruct memory should continue to use the existing player/
timeout-aware check_active_maps() lifecycle; a cache eviction policy must not
evict a live map or change reset behavior. A bounded LRU of open/mapped binary
blobs is optional—the OS page cache may already provide the desired behavior.

Lazy compilation is primarily a development convenience. It can introduce a
first-entry hitch, so normal production startup should consume precompiled
artifacts. If runtime compilation remains supported in production, perform
source parsing/encoding away from the game loop and commit only the bounded,
validated result on the authoritative thread.

6. Keep immutable cache and mutable snapshots separate

The current text map format also persists mutated inactive maps. A compiled
source cache cannot replace this without losing doors, spawned/removed items,
containers, script changes, and reset timing.

Initially, the existing temporary-map and unique-item readers/writers may
remain while immutable authored maps switch to .amap. The general object
Flex loader is also used by player/runtime persistence, so it must not be
deleted merely because archetype and authored-map loading no longer needs it.

As a follow-up phase, define a separately versioned mutable snapshot format
(for example ASNP) using the same typed field codec but different lifecycle
and identity rules:

  • atomic writes and crash recovery;
  • map generation/source identity plus reset metadata;
  • complete preservation of runtime-created and changed objects;
  • explicit limits and corruption diagnostics; and
  • a migration/inspection tool for existing local runtime data before any old
    reader is removed.

Snapshot invalidation must never be coupled to compiled-cache pruning. Random,
player-unique, dynamically created, and unique-item state are authoritative
mutable data, not reproducible cache entries.

Integration points

Server

  • Split decoding from runtime instantiation in server/src/loaders/object.l
    and server/src/server/map.c; do not duplicate the post-load semantics in a
    binary-only path.
  • Replace authored-map header/object parsing in load_original_map() with the
    compiled artifact resolver/decoder.
  • Replace arch_load() text passes with the archetype bundle decoder/linker.
  • Add settings for collected manifest location, development source fallback,
    cache directory, cache size, and strict production mode.
  • Add timing/counters for artifact hit/miss/stale/corrupt/rebuild, compile and
    decode time, instantiated object count, bytes read/mapped, cache bytes, and
    map first-entry latency.
  • Preserve all current flags and special paths: style maps, world maker,
    MAP_NO_DYNAMIC, player-unique templates, map flush/reset, dynamic tiled
    neighbors, random maps, and originator-derived maps.

Content tooling

  • Make the lossless content core and schema work from Replace Gridarta with the standalone MIT Rust editor seraphinastudios/tilewright-editor#13 the sole source
    parser/model.
  • Move map checker, Qt checker/editor replacement, mapset generation,
    world-content audit, farming analyzer, bot navigation/tasks, and worldviewer
    onto that model or its versioned JSON API.
  • Provide convert, format, validate, compile, inspect, diff, and
    cache verify/prune commands with deterministic JSON diagnostics.
  • Keep map logical paths stable even though source filenames gain a suffix;
    exits and tile links should refer to logical map IDs, not filesystem
    extensions.
  • Refuse writes to generated aggregates, server/lib, build/package output,
    and mutable server/data by default.

Build, collection, and packaging

  • Add incremental dependency tracking so a map recompiles when its source or
    relevant content bundle changes, without rebuilding unrelated maps.
  • Compile all maps in CI and release packaging; support a narrow single-map
    compile for editor/playtest iteration.
  • Keep generated artifacts under build/ or the collected runtime output, not
    checked in beside authored source unless a separate reproducibility decision
    explicitly chooses that.
  • Produce byte-identical artifacts from identical input across Linux and
    Windows, or document any platform-neutral digest boundary if packaged bytes
    differ for a justified reason.

Migration plan

Phase 0: grammar, corpus, and measurements

  • Inventory every map_header.l and object.l field, fallback/custom key,
    nesting/multipart rule, load flag, and current producer/consumer.
  • Build a representative and adversarial corpus covering comments,
    messages, inventories to the current maximum depth, multipart objects,
    custom properties, styles, unique/random maps, tiled/stacked maps, exits,
    malformed input, and unknown references.
  • Measure current archetype startup time/RSS and cold/warm first-entry,
    swap/reload, and checker/collector performance on small, median, largest, and
    object-dense maps.
  • Prototype constrained YAML versus JSONC and select the authored syntax
    using measured lossless editing, diagnostics, implementation complexity,
    hostile-input behavior, and diff quality.

Phase 1: schema and common content core

  • Define the versioned logical schemas, diagnostics, source model, strict
    custom-property boundary, and deterministic formatter/JSON API.
  • Parse the complete current corpus losslessly and compare semantic output
    with the current server/checkers.
  • Generate field metadata/tables from the schema or a shared declarative
    definition.

Phase 2: binary formats and loaders

  • Specify ABIN/AMAP framing, limits, byte order, identities, and
    malformed-input behavior in a repository design document.
  • Implement deterministic encoders and bounded transactional decoders.
  • Refactor current object/map post-decode behavior into a shared
    instantiation path and prove text-versus-binary semantic equivalence.
  • Switch archetype startup and authored-map loading to compiled artifacts;
    add manifest and development lazy cache.

Phase 3: tool and source migration

  • Add a one-shot converter and semantic-diff report for every .arc and
    map source.
  • Convert in reviewable groups, preserving comments, authored object
    order, identifiers, references, nesting, custom values, and attribution.
  • Move every in-repository parser/writer to the common model/JSON API.
  • After all current consumers move, remove the superseded authored-source path
    rather than maintaining two permanent authoring formats. Retain only the
    runtime persistence pieces that still have current writers/readers.

Phase 4: mutable snapshot format and cleanup

  • Design and implement the separate bounded mutable snapshot codec.
  • Add inspection/migration/recovery for existing local runtime state and
    update temp-map, player-unique, and unique-item workflows as explicitly
    scoped.
  • Remove obsolete Flex paths only when no current persistence consumer
    remains.
  • Update README, INSTALL, ADS/map naming documentation,
    docs/ARCHITECTURE.md, component AGENTS.md files, and content/runtime
    skills.

Validation and acceptance criteria

  • Every current authored map and archetype converts with zero unexplained
    fields and validates against the versioned schema.
  • Source-to-new-to-IR semantic comparison preserves every header,
    archetype, object override, nested inventory, multipart relationship,
    comment/note, custom property, cross-reference, coordinate, and authored
    order.
  • Unchanged source formatting is deterministic; editor no-op and targeted
    edit behavior meets Replace Gridarta with the standalone MIT Rust editor seraphinastudios/tilewright-editor#13's lossless/minimal-diff requirements.
  • Compiled archetype/map bytes are deterministic and portable; decoder
    round trips match compiler IR exactly at minimum/maximum values.
  • Binary decoders reject bad magic/version/endianness, truncation at every
    field, integer overflow, overlapping/out-of-range offsets, duplicate IDs,
    invalid UTF-8, impossible nesting/counts/coordinates, excessive lengths,
    stale bundle identities, digest mismatch, trailing data, and compression
    bombs without partially instantiating content.
  • Fresh authored-map binary loads produce equivalent live state to the old
    loader for ordinary, style, tiled/stacked, multipart, inventory, auto-apply,
    treasure, animation, connection, monster, and lighting fixtures.
  • Map reset, flush, inactive swap/reload, unique objects, player-unique
    maps, random maps, and dynamic/originator maps retain their current
    authoritative behavior; compiled-cache pruning cannot remove mutable state.
  • A cache hit does no YAML/text tokenization. A missing/stale development
    entry recompiles once, commits atomically, and is reused after restart.
    Corrupt/stale entries fail closed or rebuild from available source; they are
    never silently accepted.
  • Release/server packaging compiles the whole world and can run with no
    authored YAML present. Invalid content fails collection/CI before packaging.
  • Cache disk/open-blob memory is bounded and observable. Active map memory
    remains governed by player presence and current swap/reset semantics.
  • Benchmarks demonstrate a material improvement in archetype startup and
    warm map first-entry/decode time without increasing steady-state map RSS.
    Exact targets should be set from Phase 0 baselines rather than guessed.
  • Map checker, content audit, bot navigation, farming analyzer, mapset, and
    editor/content CLI consume one schema/model or its versioned JSON API; no
    independent partial map/archetype parser remains.
  • Linux and portable MinGW builds pass with warnings as errors, server
    checks pass, whole-world validation passes, and converter/compiler/decoder
    tests run in CI.

Non-goals

  • Parsing YAML/JSON directly in the server on every load.
  • Preloading or instantiating every map at startup.
  • Serializing raw C structs or pointers for speed.
  • Treating mutable swapped/player/unique state as a disposable cache.
  • Making archetypes lazy before profiling and redesigning their pervasive
    pointer/index consumers.
  • Keeping two permanent authored formats solely for hypothetical old tooling.
  • Replacing artifacts, treasures, factions, interfaces, or every other content
    format in the first delivery; the schema/compiler should leave a coherent
    extension path for them.
  • Bundling the complete native editor, renderer extraction, or automation API
    from Replace Gridarta with the standalone MIT Rust editor seraphinastudios/tilewright-editor#13 into this issue.

Relationships

The desired end state is one understandable authored model, one validated
compiler boundary, fast bounded runtime artifacts, lazy map instantiation, and
clear separation between reproducible content caches and authoritative mutable
world state.

Child issues

These GitHub sub-issues are the executable delivery units tracked by #168. Each belongs to one roadmap milestone; this parent remains open until all are complete.

M1

M2

M3

M5

Activity

  1. zoeyrose commented on Aug 7, 2026

    @zoeyrose
    MemberAuthor

    The proposal is now decomposed into focused atrinik/content delivery issues: content#8, content#11, content#15, content#16, and content#17. Those component-owned issues are the active implementation trackers, so this cross-repository duplicate can close.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions