Skip to content

nockawa/Typhon

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

182 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Typhon

.NET

⚠️ The engine is in active development and not usable as a library or product. ⚠️

There is no reference documentation, no stable API, no NuGet package, and no support.

A microsecond-latency ACID data engine combining ECS archetype storage with tick-based parallel execution.

Typhon is an embedded data engine for real-time workloads like game servers, simulations, and stateful dataflow.
It pairs a microsecond-latency ACID store — MVCC snapshot isolation, configurable durability, source-generated ECS archetype accessors — with a tick-based parallel runtime that dispatches fine-grained system chunks across worker threads, each operating directly on the store.
The runtime doesn't sit on top of the database — it runs inside it.

📖 Start here:


Key Features

  • Microsecond Operations — Optimized for µs-level latency with pinned memory, SIMD, and lock-free reads
  • ACID Transactions — Full transactional semantics with optimistic concurrency control
  • MVCC Snapshot Isolation — Readers never block writers; each transaction sees a consistent snapshot
  • ECS Archetype System — Entities are typed by archetype; components are blittable structs with source-generated accessors and archetype inheritance
  • Larger-Than-RAM Storage — The database is a memory-mapped, paged file on disk, virtualized through a page cache that holds only the hot working set — so resident memory is bounded by the cache, not the database. Manipulate a 100 GiB store on a machine with a fraction of that RAM; component data, indexes, and the entity map all page to/from disk on demand (in-memory ECS frameworks, by contrast, must fit the entire world in RAM)
  • Workbench Dev UI — A local, browser-based companion (ASP.NET Core + React/Vite) — DataGrip for Typhon, not just a profiler. Open a recorded .typhon-trace, attach to a live engine over TCP, or open a .typhon database file directly. Performance: per-tick profiler timeline with CPU-sample + off-CPU overlays, Top Spans, Call Tree, Critical Path, System DAG. Data & schema: component/archetype browsers, Schema Inspector, a zoomable Hilbert-curve Database File Map, live Resource Tree. Queries: catalog, plan tree, per-execution inspector — all in a dockable, persisted workspace
  • Entity Clusters — SoA batched storage co-locating up to 64 entities per cluster, with cluster-native SIMD predicate evaluation (AVX-512 / AVX2 / scalar dispatch), zone-map pruning, and k-way sorted merge
  • Query Engine — Typed queries with Where, With/Without filters, polymorphic iteration, OR logic, navigation joins, and a cluster execution path that routes filtered scans through SIMD gather-compare on cluster SoA columns
  • Views & Change Tracking — Incremental entity-set monitoring across transactions (added/removed/modified detection) with ring-buffer delta streaming
  • B+Tree Indexes — Cache-aligned 128-byte nodes with optimistic lock coupling and specialized key-size variants
  • Spatial Indexing — Page-backed wide R-Tree for AABB / Radius / Ray / Frustum / kNN queries, plus a per-cell cluster broadphase (SpatialGrid) with BMI2 Morton-encoded cell keys and multi-resolution SimTier dispatch for near/far/coarse simulation budgets
  • Write-Ahead Logging — WAL with configurable durability modes (Deferred, GroupCommit, Immediate) and coalesced tick-boundary snapshots via TickFence / ClusterTickFence chunks for the runtime tick loop
  • Page Integrity — CRC32C checksums, full-page images, and checkpoint-based recovery
  • Schema Versioning — Component revisions with field-level migration support
  • Three Storage Modes — Versioned (full MVCC + WAL), SingleVersion (last-writer-wins, tick-fence WAL), Transient (heap-only, no persistence)
  • Configurable Durability — Choose per-UnitOfWork whether data is deferred, group-committed, or immediately fsynced
  • Game Server Runtime — Tick-based micro-task DagScheduler with any-worker parallel dispatch, per-system change filters, typed MPSC event queues, side-transactions, cluster dormancy with staggered heartbeat wake, checkerboard Red/Black dispatch, and a 4-level overload response hierarchy (tick-rate modulation, player shedding)
  • Subscription Server + Client SDK — TCP delta streaming of published views with MemoryPack serialization, per-client incremental sync, backpressure-driven resync, and a zero-engine-dependency Typhon.Client assembly for game clients
  • Observability — Runtime telemetry, metrics, and diagnostics with zero-cost JIT-eliminated toggles
  • Deep-Trace Profiler — Per-tick Gantt and flame-graph visualization via a .typhon-trace binary format, live TCP streaming to a React/Vite viewer, and optional dotnet-trace CPU sampling correlation for full managed-stack profiles
  • Interactive Shell (tsh) — Database REPL for inspection and debugging
  • Public/Internal API split — Two namespaces per assembly (Typhon.Engine / Typhon.Engine.Internals), each subsystem folder split into public/ + internals/, enforced at compile time by the TYPHON008 Roslyn analyzer
  • Roslyn Analyzers — Custom analyzers detecting undisposed engine resources at compile time + the TYPHON008 internal-API leak detector

Quick Start

A taste — declare a component and an archetype, spawn an entity, read it back:

// A component is a plain struct; an archetype is the fixed set of components an entity has.
[Component("Game.Health", 1, StorageMode = StorageMode.Versioned)]
public struct Health { public int Current, Max; }

[Archetype(1)]
public sealed partial class Unit : Archetype<Unit>
{
    public static readonly Comp<Health> Health = Register<Health>();
}

// ...build the engine + register the schema (see the Guide), then:
using var tx = dbe.CreateQuickTransaction();
var id = tx.Spawn<Unit>(Unit.Health.Set(new Health { Current = 100, Max = 100 }));
var hp = tx.Open(id).Read(Unit.Health);   // hp.Current == 100
tx.Commit();

Illustrative — engine build + schema registration are elided. The User Guide has the full, runnable version (doc/guide/example).

Architecture

Typhon Architecture Layers

Development Status

Typhon is in active development targeting an alpha release. Current state:

  • Core transaction engine with MVCC
  • B+Tree indexes with optimistic lock coupling
  • Component-level durability options
  • Write-Ahead Logging with configurable durability modes
  • Page integrity (CRC32C, full-page images, checkpoints)
  • Query engine with filtering, sorting, and navigation
  • Views and incremental change tracking
  • ECS archetype system with source-generated accessors
  • Entity cluster storage (SoA batched, SIMD predicate evaluation, zone-map pruning, k-way sorted merge)
  • Dual page store abstraction (PersistentStore + TransientStore)
  • Schema versioning and evolution
  • Spatial indexing (page-backed wide R-Tree: AABB / Radius / Ray / Frustum / kNN)
  • Spatial grid with per-cell cluster broadphase and BMI2 Morton encoding
  • Multi-resolution simulation (SimTier dispatch, cluster dormancy, checkerboard Red/Black)
  • Game server runtime (DagScheduler, parallel system dispatch, overload management, event queues)
  • Subscription server and Typhon.Client SDK (TCP delta streaming, MemoryPack wire protocol)
  • Deep-trace runtime profiler (.typhon-trace format, live TCP streaming, React viewer)
  • Interactive shell (tsh)
  • Observability and monitoring stack
  • HashMap collections with key-size specialization
  • Workbench dev UI (data browsing, profiler viewer, System DAG view, Data Flow timeline, Access Matrix, internal data API)
  • Public/Internal API namespace split + TYPHON008 analyzer
  • Crash recovery testing
  • Backup and restore

Project Structure

Each engine subsystem folder splits into public/ (consumer surface, namespace Typhon.Engine) and internals/ (implementation, namespace Typhon.Engine.Internals).

Typhon/
├── src/
│   ├── Typhon.Engine/              # Main database engine — 17-feature tree
│   │   ├── Foundation/             # Cross-cutting primitives
│   │   │   ├── Collections/        # HashMaps, bitmaps, lock-free arrays
│   │   │   ├── Concurrency/        # Latches, AccessControl, epoch system, deadlines, timer service
│   │   │   └── Memory/             # Memory allocator + block primitives
│   │   ├── Ecs/                    # ECS engine, archetypes, entity clusters, accessors
│   │   ├── Indexing/               # B+Tree variants (key-size specialized, OLC)
│   │   ├── Querying/               # Query engine, views, navigation joins, predicates
│   │   ├── Schema/                 # Component & archetype schema, validation, evolution
│   │   ├── Spatial/                # R-Tree, spatial grid, tier dispatch, trigger zones
│   │   ├── Storage/                # Pages, cache, segments, PagedMMF / IPageStore / PersistentStore / TransientStore
│   │   ├── Transactions/           # MVCC transactions, UnitOfWork, change capture
│   │   ├── Revision/               # Revision chains, deferred cleanup
│   │   ├── Durability/             # WAL, checkpointing, recovery, FPI capture
│   │   ├── Runtime/                # DagScheduler, tick loop, systems, overload management, queues
│   │   ├── Subscriptions/          # TCP delta streaming server, published views
│   │   ├── Profiler/               # In-engine typed-event profiler (codecs in Typhon.Profiler/)
│   │   ├── Observability/          # Telemetry config, metrics, diagnostics
│   │   ├── Resources/              # Resource graph, lifecycle, options
│   │   ├── Errors/                 # Exception hierarchy, deadline propagation
│   │   └── Hosting/                # DI extensions, service collection helpers
│   ├── Typhon.Analyzers/           # Roslyn analyzers (dispose detection + TYPHON008 internal-API leak)
│   ├── Typhon.Client/              # External client SDK (TCP subscriptions, zero engine deps)
│   ├── Typhon.Generators/          # Source generators (archetype accessors, traced-event encoders, source location)
│   ├── Typhon.Profiler/            # Trace file format, readers/writers, sidecar cache, Chrome Trace exporter
│   ├── Typhon.Protocol/            # MemoryPack wire-format types (TickDeltaMessage, etc.)
│   ├── Typhon.Schema.Definition/   # Component & archetype attributes
│   ├── Typhon.Shell/               # Interactive database shell (tsh)
│   └── Typhon.Shell.Extensibility/ # Shell extension points
├── test/
│   ├── Typhon.Engine.Tests/        # NUnit test suite (3500+ tests)
│   ├── Typhon.Client.Tests/        # Client SDK tests
│   ├── Typhon.Workbench.Tests/     # Workbench server-side tests
│   ├── Typhon.Benchmark/           # BenchmarkDotNet performance tests
│   ├── Typhon.ARPG.Schema/         # Example ARPG game schema
│   ├── Typhon.ARPG.Shell/          # Shell demo with ARPG data
│   ├── Typhon.MonitoringDemo/      # Observability demo
│   ├── Typhon.IOProfileRunner/     # Storage I/O profiling sandbox
│   ├── Typhon.Scheduler.POC/       # DagScheduler POC harness
│   ├── Typhon.SqliteBenchmark/     # Comparative SQLite benchmark
│   └── AntHill/                    # Godot-based ant-colony demo (runtime + clusters + spatial tiers)
├── tools/
│   ├── Typhon.Workbench/           # Local dev UI (ASP.NET Core 10 + React 19/Vite) — data browsing, schema, profiler, System DAG, Data Flow timeline, Access Matrix
│   ├── Typhon.Workbench.Fixtures/  # Dev-only test fixture generators consumed by the Workbench DEBUG tabs
│   └── CheckDurations/             # Helper utility for tick-duration analysis
├── claude/                         # Architecture docs, ADRs, design specs (separate nested git repo)
└── benchmark/                      # Benchmark results

History

This project has had quite a journey:

  • 2015 — Initial bootstrap with a different design, quickly shelved
  • 2020 — COVID resurrection as a POC: "Can we build a µs-latency ACID database for persistent games?" Promising results, then shelved again
  • 2025 — Third resurrection with firm intention to reach alpha stage
  • 2025-2026 — Rapid progress: WAL & durability, query engine, ECS archetype system with source generators, schema evolution, observability stack, and interactive shell delivered
  • Q2 2026 alpha push — Game-server runtime (DagScheduler, parallel system dispatch, overload management), entity clusters with cluster-native SIMD query execution, dual page store abstraction (PersistentStore + TransientStore), spatial indexing (page-backed R-Tree + spatial grid cluster broadphase), multi-resolution SimTier dispatch with cluster dormancy and checkerboard, TCP subscription server with zero-engine-dependency client SDK, and a deep-trace runtime profiler with live-streaming viewer
  • Q2 2026 Workbench buildout — Internal Data API + System DAG view (#306, #314, #322), static schema export and Schema Inspector (#326), Data Flow Timeline + Access Matrix panels with cross-panel selection and hover linking (#327)
  • Q2 2026 API hardening — Public/Internal namespace migration, accessibility flips, friend-list audit, and leak-tightening pass: ~430 internal types now in Typhon.Engine.Internals enforced by the TYPHON008 analyzer (#329)

About

A microsecond-latency ACID database engine with a native Entity-Component-System data model, built for real-time systems.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

 
 
 

Contributors