Scepter

Composable Rust primitives for large-scale time-series routing, indexing, aggregation, and query planning.

scepter is a small library for building observability, metrics, stream processing, and distributed query systems. It packages the kinds of low-level building blocks described in the public Monarch paper into independent Rust APIs: ordered keys, range assignment, field-hint indexes, histogram-like distributions, collection aggregation, ingest routing, pushdown planning, and standing-query sharding.

The crate is inspired by Monarch-style system design, but it is not Google Monarch and does not depend on any Google service or internal API. The name keeps the theme without claiming ownership: a scepter is the control instrument of a monarch; this crate focuses on control-plane and data-plane primitives for distributed time-series systems.

Use Cases

• Route metric writes to shards by lexicographic target ranges.
• Prune distributed query fanout with compact field-hint indexes.
• Represent and merge distribution-valued metrics with exemplars.
• Turn cumulative points into delta windows.
• Aggregate ingestion streams before storage.
• Finalize bucketed delta aggregates behind an admission window.
• Select query replicas by coverage, density, completeness, and recovery state.
• Attach partial-result health metadata to degraded distributed queries.
• Export hot-path data as Apache Arrow batches behind an optional feature.
• Use Roaring-backed numeric field hints for compressed postings.
• Encode compact CBOR/Zstd wire payloads behind optional features.
• Split logical query plans into leaf, zone, and root execution fragments.
• Shard periodic standing queries across evaluator workers.

Install

cargo add scepter

Or add it manually:

[dependencies]
scepter = "0.1"

Quick Example

use scepter::{FieldHintIndex, FieldPredicate, RangeAssigner};

let mut ranges = RangeAssigner::new();
ranges.assign(b"a".to_vec()..b"m".to_vec(), "leaf-1")?;
ranges.assign(b"m".to_vec()..b"z".to_vec(), "leaf-2")?;

assert_eq!(ranges.worker_for("latency"), Some(&"leaf-1"));

let mut index = FieldHintIndex::new();
index.insert_value("ComputeTask", "job", "monarch", "leaf-1");

let candidates = index.candidates(
    "ComputeTask",
    "job",
    &FieldPredicate::Equals("monarch".to_owned()),
);
assert!(candidates.contains("leaf-1"));

# Ok::<(), scepter::ShardError>(())

Distribution Metrics

use scepter::{BucketLayout, Distribution};

let layout = BucketLayout::fixed_width(0.0, 10.0, 3)?;
let mut latency = Distribution::<()>::from_layout(&layout);

latency.record(4.2, 1)?;
latency.record(18.0, 2)?;

assert_eq!(latency.total_count(), 3);
assert!(latency.percentile(99.0)?.is_some());

# Ok::<(), scepter::DistributionError>(())

Collection Aggregation

use scepter::{AdmissionWindow, BucketedAggregator, DeltaPoint, SumReducer};

let mut iops =
    BucketedAggregator::with_reducer(60, AdmissionWindow::new(10), SumReducer)?;

iops.ingest(DeltaPoint {
    key: ("cluster-a", "storage-user"),
    end_time: 12,
    value: 3_000_u64,
})?;
iops.ingest(DeltaPoint {
    key: ("cluster-a", "storage-user"),
    end_time: 38,
    value: 2_000_u64,
})?;

let finalized = iops.advance_to(70);

assert_eq!(finalized[0].start, 0);
assert_eq!(finalized[0].end, 60);
assert_eq!(finalized[0].value, 5_000);

# Ok::<(), scepter::CollectionError>(())

Query Reliability

use scepter::{
    IssueKind, QueryHealth, ReplicaCandidate, ReplicaQuality, ReplicaResolver,
    ReplicaState,
};

let resolved = ReplicaResolver::with_max_fallbacks(1).resolve(vec![
    ReplicaCandidate::new(
        b"a".to_vec()..b"m".to_vec(),
        "leaf-a",
        ReplicaQuality::new(0, 60, 60, 60, true, ReplicaState::Available),
    ),
    ReplicaCandidate::new(
        b"a".to_vec()..b"m".to_vec(),
        "leaf-b",
        ReplicaQuality::new(0, 60, 55, 60, true, ReplicaState::Recovering),
    ),
]);

assert_eq!(resolved[0].primary, "leaf-a");
assert_eq!(resolved[0].fallbacks, vec!["leaf-b"]);

let mut health = QueryHealth::with_expected_children(2);
health.record_completed();
health.push_issue("zone-west", IssueKind::PrunedZone, "soft deadline elapsed");

assert!(health.is_partial());
assert_eq!(health.completeness(), 0.5);

Primitive Families

scepter is organized by concern:

• key: ordered key encoding with LexicographicKey and KeyEncoder.
• model: schema-rich time-series types such as TargetSchema, MetricSchema, TimeSeriesKey, MetricKind, and LocationResolver.
• shard: range assignment, range load scoring, worker lookup, reassignment, and range splitting.
• hint: compact field-hint indexing with trigrams, n-grams, full-string excerpts, predicates, intersections, and unions.
• distribution: bucket layouts, distribution values, exemplars, cumulative points, percentile estimates, and delta windows.
• aggregate: distributed aggregation traits, Sum, and CollectionAggregator.
• collect: bucketed delta aggregation with admission windows, finalized buckets, load-smearing offsets, and pluggable reducers.
• ingest: write envelopes, stale-write drop policy, and range-based ingest routing.
• query: logical plans, execution levels, fanout plans, and pushdown fragments.
• reliability: replica quality ranking, primary/fallback replica resolution, and partial-result query health metadata.
• standing: periodic standing queries and stable evaluator sharding.

Performance Features

The default crate has no runtime dependencies. Optional features enable interchange and compression formats for high-volume paths:

[dependencies]
scepter = { version = "0.1", features = ["arrow", "compressed-postings", "wire"] }

• arrow: Apache Arrow RecordBatch exporters for finalized buckets, replica candidates, and query-health issues.
• compressed-postings: NumericFieldHintIndex, a Roaring-backed field-hint index for stable u32 child ordinals and dense set operations. The generic FieldHintIndex remains faster for tiny singleton-style candidate lookups.
• cbor: compact CBOR helpers for wire payloads.
• zstd: Zstandard compression helpers.
• wire: convenience feature enabling cbor and zstd together.

Design Principles

• No runtime dependencies for the library itself.
• Recoverable invalid input is returned as Result, not exposed as public API panics.
• unsafe is forbidden.
• Public API documentation is required by #![deny(missing_docs)].
• Public error types implement std::error::Error.
• APIs are small and composable instead of tied to one database runtime.
• Tests include unit tests, property tests, and mutation-regression tests.

What This Crate Is Not

scepter is not a complete time-series database. It does not provide storage, replication, networking, persistence, query parsing, or a hosted service. It is the reusable primitive layer you can use while building those systems.

Test Strategy

Scepter uses three layers of tests:

• Example-sized unit tests live beside each module.
• Property tests in tests/properties.rs check invariants over generated inputs.
• Mutation-regression tests in tests/mutation_regression.rs lock down behavior that mutation testing has proven easy to under-specify.

The current mutation suite catches every viable mutant reported by cargo-mutants.

Quality gates used before publishing:

cargo fmt --check
cargo test
cargo clippy --all-targets -- -D warnings
cargo bench --bench critical_paths
cargo audit
cargo deny check advisories bans licenses sources
cargo mutants --timeout 120 --jobs 2
cargo publish --dry-run

Benchmarking

Critical-path benchmarks live in benches/critical_paths.rs and use Criterion. They cover:

• sortable key encoding
• range assignment lookup and splitting
• field-hint indexing and candidate pruning
• distribution record, merge, percentile, and delta operations
• collection aggregation and bucketed delta finalization
• query pushdown planning
• replica resolution
• standing-query evaluator sharding

Run them with:

cargo bench --bench critical_paths

The first benchmark pass exposed a slow field-hint candidate path. Switching candidate intersection to start from the smallest posting list improved exact trigram lookup from millisecond-scale to low-microsecond-scale in the 10k-value benchmark.

Worked Example

The mini_engine example wires the primitives together into a small observability-engine flow: route an encoded write, index field hints, prune query fanout, split a logical query into pushdown fragments, and merge distribution-valued leaf results.

Run it with:

cargo run --example mini_engine

Book

The mdBook guide is published at https://copyleftdev.github.io/scepter/.

Local docs workflow:

just book
just book-serve

License

Licensed under either of:

• Apache License, Version 2.0
• MIT license

at your option.