Skip to content

Historical run metadata for the 2026-06-25 logic_with_navigation scores #540

Open

Description

@anthesiallc

Hello, I'm working on a local benchmark comparison tool and want to preserve
the provenance and comparability of published scores.

I'm reviewing the 28 logic_with_navigation results in
table_2026_06_25.csv at site commit 19e766a5de4de07d672ed5bf9f0a69ceed1d39bf.
I can identify the table's model variants and current model configurations, but
cannot establish which historical run settings produced these particular cells.

Is there an existing manifest or metadata export tying these results to:

  • Original run/result identifiers, or your recommended stable result-identity
    convention when individual IDs are not published.
  • The executed harness commit and question-set revision, including sample counts,
    exclusions and whether each score covers the same complete task set.
  • Actual provider/API model IDs, generation limits/settings, reasoning settings
    and overrides, and any tool use.
  • Which variants were evaluated under the same protocol, and whether a complete
    cohort under that protocol is available separately if this table mixes them.

Existing links or a stripped metadata export would be sufficient; no prompts,
answers, credentials or new evaluations are requested. Run dates would also help
if already available. Unknown fields can remain explicitly unknown.

Could you also clarify whether the
datasheet's dataset-distribution statement
covers the published numeric CSV summaries for local private storage and derived
rankings with attribution? The intended scope here is local private use.

Thank you for any pointers. If this release's historical metadata is unavailable,
knowing that would also help establish the limits of the comparison.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions