Skip to content

benchmark: add BEAM evaluation #355

Description

@jinliyl

Background

BEAM evaluates long-term conversational memory at 128K, 500K, 1M, and 10M-token scales across diverse domains and memory abilities. ReMe needs a scalable BEAM harness to characterize memory quality and resource behavior beyond the context sizes covered by smaller conversational benchmarks.

Changes

  • Add a BEAM benchmark workflow under benchmark/beam
  • Import the official conversations, questions, annotations, and evaluation assets without committing downloaded or generated data
  • Ingest conversations incrementally and preserve turn order, timestamps, domain, scale, and evidence identifiers
  • Support representative smoke-test subsets and full 128K, 500K, 1M, and 10M-token tiers
  • Reproduce the official evaluation protocol and report results by memory ability, domain, and context scale
  • Measure retrieval evidence quality separately from end-to-end answer quality
  • Track ingestion time, index size, query latency, model and embedding usage, token usage, and cost
  • Make long runs resumable and add configuration documentation, result schemas, and focused fixture-based tests

Activity

  1. moved this from Todo to In Progress in ReMeon Jul 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

  • Status
    In Progress

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions