Add a run_dir admission manifest for candidate acceptance/rejection audits #381

Activity

1. OtotaO commented on Jun 22, 2026

   

   OtotaO

   on Jun 22, 2026

   More actions

   +1 on this. The part that makes it compelling is that the record is replayable for free: every field
   can be reconstructed from facts GEPA already holds, so a verifier can recompute the manifest and diff
   it against what was written without rerunning the optimizer.

   Concretely, the verify side is just:

   def verify_record(record, candidate, parents):
       assert record["candidate_hash"] == _candidate_hash(candidate)
       assert record["parent_candidate_hashes"] == [_candidate_hash(p) for p in parents]
       # subsample + decision fields are exact replays of what the engine had in hand

   I went and traced the seams rather than guess, so here's the plan I'd implement. Corrections very
   welcome where I've misread something.

   Reflective accept/reject. The decision lives in GEPAEngine._accept_reflective_proposal: the
   boolean is self.acceptance_criterion.should_accept(proposal, state), and at that point the proposal
   (a CandidateProposal) plus state give me everything for the identity/lineage half of the record
   without any extra work. candidate_hash comes from the module-level
   _candidate_hash(proposal.candidate) in state.py, parent_candidate_hashes from
   _candidate_hash(state.program_candidates[i]) over proposal.parent_program_ids, the subsample scores
   from the subsample_scores_before/after the method already reduces to old_sum/new_sum,
   proposal_kind from proposal.tag, and the reject reason on the reject branch.

   The one boundary I want to flag honestly: the val-level fields (val_average_score,
   evaluated_val_ids, is_best_program, the assigned candidate index) are not in scope at the
   decision. They're produced later inside _run_full_eval_and_add (the _evaluate_on_valset →
   update_state_with_new_program → ValsetEvaluatedEvent path), which only runs on the accept branch.
   So the record is naturally two-phase: write identity/lineage/subsample/decision at the seam, then
   backfill the val fields at the full-val step (and on a reject they stay pending, with candidate_idx == -1, matching what the existing proposals table already does).

   components_updated is the one field with no first-class home. I'd recover it by parsing the
   prompt:/raw_lm_output: keys off proposal.metadata, the same way _log_proposal_lm_calls does
   today, unless you'd rather I add a dedicated field upstream.

   Merges. You're right that they're handled inline in run(): there's no _accept_merge_proposal
   analog, the call is just new_sum >= max(parent_sums) under the tag == "merge" guard. The nice thing
   is both parents are fully reachable there. proposal.parent_program_ids == [id1, id2] index into
   state.program_candidates, so I can hash both parent dicts and the merged proposal.candidate right
   at the site. Reflective and merge both funnel their accepts through _run_full_eval_and_add +
   CandidateAcceptedEvent, so for accepted records what I'd do is stamp them from a small shared helper
   at that common chokepoint (threading proposal_kind + parent hashes through), rather than duplicate
   the write. That keeps the two paths symmetric and stops the manifest drifting between them. The
   reject records don't share a seam (reflective rejects at the CandidateRejectedEvent branch, merges
   at their own inline < max(parent_sums) branch), so I'd write those at each decision point. If you'd
   rather it all open-coded to mirror the current inline style, easy to flip.

   Budget fields. total_metric_calls / per-discovery cost / num_full_val_evals all come straight
   off state.total_num_evals and friends (already surfaced in GEPAResult), so those are real.
   cache_hits/cache_misses I'd leave null: as far as I can tell EvaluationCache keeps only _cache
   with no cumulative hit/miss counter, so there's nothing to read without adding counters upstream (happy
   to PR those separately if you want them).

   For persistence I'd match GEPAState._atomic_write_json (tmp-write + os.replace, json.dump(..., indent=2, default=json_default)), the same helper save() uses for run_log.json /
   candidates.json, so the manifest lands with the same atomic-write guarantees.

   One thing that's genuinely your call: the opt-in surface. My default would be a
   write_admission_manifest: bool = False field on EngineConfig (mirrored as a flat kwarg in
   api.py's optimize(), since the two entry points are parallel), which means a small third edit to
   add the param to GEPAEngine.__init__ and forward it from both build sites. But if you'd rather just
   infer it from run_dir being set and avoid a new flag entirely, that's an equally clean surface. Tell
   me which you prefer and I'll wire it that way.

   Separately, and very much subordinate to the above: is tamper-evidence of the manifest itself in
   scope for you, or out? The hashes already make each record self-checking against the candidate it
   names, but they don't chain records together or carry a signature an external reviewer could check
   offline. If that's ever interesting I can sketch it: I maintain a small signed-receipt library
   (SUM) built on exactly that shape (Ed25519 over JCS-canonical bytes,
   detached JWS), so it'd be a thin, opt-in add rather than new infrastructure. Otherwise I'll happily
   leave it out and keep this a purely replayable audit log.
2. sunghunkwag commented on Jun 22, 2026

   

   sunghunkwag

   on Jun 22, 2026

   Last edited by sunghunkwag

   Author

   More actions

   Thanks for tracing this against the current code. Your read of the reflective and merge seams matches mine, and "replayable for free" is the core property I was aiming for.

   For a v1, my preferences would be:

   • Make it explicitly opt-in: write_admission_manifest: bool = False in EngineConfig, mirrored in api.py::optimize(), and require run_dir when enabled. I would not infer it from run_dir alone because existing users should not start emitting a potentially sensitive artifact unexpectedly.
   • Keep the manifest as JSONL, but do not backfill a line in place. Build the decision snapshot at _accept_reflective_proposal; for an accepted proposal, finalize the validation fields after _run_full_eval_and_add and append the complete record once. Rejects can append once immediately with validation fields null/empty and candidate_idx: null. The shared finalization helper for reflective and merge accepts sounds right.
   • Make the phase explicit: the recorded decision is the subsample acceptance decision; validation fields are post-decision enrichment, not a second admission gate.
   • I would derive a deterministic components_changed from the candidate-versus-parent data rather than parse prompt: / raw_lm_output: metadata. Metadata may be absent for adapters or custom proposers, while a candidate diff remains replayable. If ?components selected for mutation? is also needed, that should be a separate optional first-class field.
   • Leave cache hit/miss counters null or omit them in v1. Adding counters can be a separate change.
   • Use a dedicated one-record JSONL append helper rather than _atomic_write_json, which atomically replaces a whole JSON document. Serialize one complete line, append and flush it, and test handling of a truncated final line.
   • A minimal replay verifier/test that recomputes candidate and parent hashes and checks the recorded decision inputs would be valuable.

   I checked SUM; the Ed25519/JCS/detached-JWS design is relevant and well aligned with this kind of receipt. However, I would keep signatures, record chaining, and tamper-evidence out of #381 and treat them as a separate opt-in follow-up after the base schema is stable.
3. OtotaO commented on Jun 22, 2026

   

   OtotaO

   on Jun 22, 2026

   More actions

   This is a great spec, thanks. I'll open a PR that implements exactly this:

   • write_admission_manifest: bool = False on EngineConfig, mirrored in optimize(), and required
     run_dir when enabled. Agreed on not inferring it from run_dir alone, existing runs shouldn't start
     emitting a new artifact unexpectedly.
   • Append-once, no in-place backfill. Build the decision snapshot at _accept_reflective_proposal; for
     an accepted proposal, finalize the validation fields after _run_full_eval_and_add and append the
     complete record once; rejects append immediately with validation fields null/empty and candidate_idx: null. One shared finalization helper across the reflective and merge accept paths.
   • Phase made explicit in the schema: the recorded decision is the subsample acceptance decision, and
     the validation fields are post-decision enrichment, not a second admission gate.
   • components_changed derived from a candidate-versus-parent diff rather than parsing
     prompt:/raw_lm_output: metadata. You're right that metadata can be absent for custom
     adapters/proposers, while a diff stays replayable. If "components selected for mutation" is also wanted
     later, I'll add it as a separate optional first-class field.
   • cache hit/miss left null/omitted in v1; counters can be their own change.
   • A dedicated single-record JSONL append helper (serialize one complete line, append and flush) rather
     than _atomic_write_json, plus a test that a truncated final line is handled gracefully.
   • A minimal replay test that recomputes the candidate and parent hashes and re-checks the recorded
     decision inputs.

   Agreed on keeping signatures, record chaining, and tamper-evidence out of this PR. I'll keep the base
   schema clean and we can treat that as a separate opt-in follow-up once it's stable. I'll get a draft up
   and tag you.
4. aranya-chatterjee commented on Jun 25, 2026

   

   aranya-chatterjee

   on Jun 25, 2026

   More actions

   is this still up for grabs or has @Ototoo already opened a PR?
5. OtotaO commented on Jun 25, 2026

   

   OtotaO

   on Jun 25, 2026

   More actions

   @aranya-chatterjee yep, the PR is up: #382. It implements the full v1 spec from this thread (opt-in
   write_admission_manifest, append-once JSONL with no in-place backfill, one shared finalize helper
   across the reflective and merge accept paths, components_changed from a candidate-vs-parent diff, a
   dedicated single-record append helper, and a replay verifier test). It's marked ready for review and CI
   is green, so I'm just waiting on @sunghunkwag to take a look.

   Happy to have another set of eyes on it if you're interested.
6. OtotaO commented on Jul 16, 2026

   

   OtotaO

   on Jul 16, 2026

   More actions

   The PR implementing this (#382) is up and green whenever it is useful to look
   at. It follows the v1 spec from this thread: an opt-in write_admission_manifest
   flag with a required run_dir, append-once records with one finalize helper
   shared by the reflective and merge accept paths, components_changed derived
   from a candidate-versus-parent diff rather than parsed metadata, and a replay
   test that recomputes the candidate and parent hashes. No rush at all on my end.
   Happy to rebase or adjust it whenever it fits your review cadence.