[Nightshift] Migrate Semantic Memory - #291288

Relates to elastic/nightshift-program#1549.

Closes elastic/nightshift-program#1614

Summary

Brings Deductive-style Semantic Memory

Storage and retrieval

• Plugin-owned, hidden index nightshift-semantic-memory (context.semantic is the ELSER recall lane), reached only through a lazy internal client. No raw index route is exposed and workflow users get no index privileges.
• Space-scoped like Cortex: IDs are ${space_id}:memory_<slug> and every query filters attributes.space_id. Only the Nightshift investigator reads or writes Memory; the agent id is not stored on memory documents.
• Retrieval combines lexical and semantic RRF, read-time exponential decay, and Thompson sampling for browse mode. Hydration returns the top 15.
• All Memory operations wait on one cached index/template readiness promise. A failed init fails Memory clearly without blocking plugin startup.

What the agent sees

• Pages are written to /workspace/memories/<slug>.md and contain only the memory text: no front matter, ids, or scores.
• /workspace/memories/.index.json is an additive catalog of { path, updated_at }. Nothing unrelated is deleted.
• Only pages newly written this turn are announced ("Potentially relevant memories retrieved this turn:", with path and last-updated date). Cortex materializes its full wiki but announces nothing.
• The model generates one title per memory and the slug is derived from it.

Round context

• A final compose step returns a memory-only <system_update> as model_context and workflow_context["nightshift.semantic_memory.recall"] = { version: 1, data: { recalled_ids } }. Agent Builder persists both on the triggering round. The UI shows only the user's message, and later turns replay the model context verbatim until compaction.
• Post-round optimize reads the recalled IDs from that round, so it does not depend on sandbox state or race with concurrent turns.
• On HITL resume (round_execution_index > 0), materialization and decision-tree hydration are skipped. Other beforeAgent hooks still run.

Optimize

Three LLM calls run after each round, plus a compaction call when a written memory is over budget. Each gets the user task and the investigation in order: agent notes and each evidence tool call with a result excerpt, with ERROR on failed calls. Only the critique also gets the final answer.

• Evidence only: all three calls drop calls (with their results) that are not evidence about the customer's environment: the agent's own process tools (progress reports, todos, list_files, skill loading, sleep, conversation metadata, user questions, sandbox file writes and edits) and any read of pre-seeded knowledge (paths under /workspace/cortex/ or /workspace/decision-trees/, /workspace/elastic.md, /workspace/connectors.md, by file_path/path or named in a bash command). Memory reads under /workspace/memories/ and unknown tools are kept, so the critique still sees which memories were opened.
• Tool results: the post-round hook's tool_results input (added by [Nightshift] Give Cortex optimize the investigator's tool results #294383) is passed to both Cortex and Memory. Memory attaches each result to its call by tool_call_id. It first tries to read the persisted round, which also has the agent's notes; that read still fails from the workflow's request (Agent Builder reports the conversation as not found), so in practice the transcript is built from the hook's calls and results. Without results, the critique falls back to tool-call parameters, and extraction is skipped unless at least one evidence call returned output: a memory must be backed by a tool result, not by the final answer alone.
• Critique labels recalled memories useful or harmful (ranked, one evidence budget of 128,000 estimated tokens shared by the recalled pages). Unused means no label, opening a memory is not using it, and harmful means its content was wrong or misleading.
• Extraction is a classifier: it proposes up to three topics. Each entry has only a title, keywords (stored as tags) and replaces (recalled memory ids it supersedes: none for a new memory, one for an update, several for a merge). It writes no content. Recalled memories are shown with id, title, last-updated time and content. Its scope: organizational context, component behavior and environment topology that a tool result shows; never generic knowledge, how the run investigated, absence, secrets, container internals, or facts only relevant to this request.
• Writer (nightshift_memory_write) writes the content of every entry, new ones included, from the topic and keywords, the replaced memories (if any, integrating their facts that still hold), the evidence transcript, the run time and the other entries' titles (so facts stay on their own topic). It has the same scope rules, adds no advice, dates facts that can change, writes a stopped problem in the past tense, and returns empty content, which writes nothing, when the run establishes nothing durable about the topic. It has a 4,000-character budget. Content over it goes to a compaction call (nightshift_memory_compact) with the writer's instructions, its input, the draft and a word target set below the budget; content still over the budget, or whose compaction call fails, is truncated and stored (one attempt, no retry).
• Recall key: a written memory's context is this round's user task. Recall matches the next round's task against it, so a similar future task lands close to it.
• Which memories an entry replaces: the ones it names in replaces, plus a live memory already stored at the entry's id even if it wasn't recalled (extraction only sees recalled memories). An archived memory at that id is written over. There is no lexical or catalog overlap matching. A memory already replaced by an earlier entry in the round skips the later entry.
• Harmful memories are archived up front with reason harmful and never reach the writer, even when an entry replaces them. That entry is written from only the live memories it replaces, or as a new memory.
• Writes: one exact-slug create-or-merge flow. An existing live slug is merged in place, a missing slug is created, and a create 409 fetches the winner and merges. Source pages are archived (reason merged) only after the canonical write succeeds; the merge revalidates its sources around synthesis and resynthesizes on change. An id that belongs to an archived memory is reused: the new memory is written over the archived document under optimistic concurrency, with counters reset and merge history kept.
• Impression/conversion counters use same-index optimistic concurrency (versioned read, JS decay, partial update guarded by _seq_no/_primary_term, bounded retries). This avoids scripts, which Elasticsearch forbids on indices with semantic_text.
• Memory optimize resolves its model through the Nightshift resolver from [Nightshift] Remove model selection from Model Settings #293894 (@kbn/nightshift-ai), given the step's connector_id and the triggering round's round_connector_id, like Cortex. Both share one telemetry tag (significant_events_investigation), so spend is attributed to Nightshift. elastic/nightshift-program#1644 renames the tag.

Safety

• Non-canonical memory IDs are rejected before building sandbox paths or notifications. Slug canonicalization is idempotent (every leading memory- prefix stripped, cut to 80 characters with no trailing hyphen), so every written id is canonical.
• Secret detection covers the proposed title, keywords, the task, and the written content.
• Materialize and optimize both skip unless the agent is the Nightshift investigator, while storage stays Space-scoped.
• Memory steps stop with the workflow: cancellation or the step timeout aborts every Memory LLM call, and no later call or write starts.
• Partial sandbox writes count as materialization failures. Optimize still runs when the sandbox is unavailable.
• Persistent workflow logs never contain page IDs or slugs, connector IDs, prompts, or raw model errors.
• Memory persistence failures are reported as failed optimizations.

Telemetry

• Aggregate EBT events nightshift-semantic-memory-materialized and nightshift-semantic-memory-optimized carry correlation IDs, bounded outcomes, and counts only. Model, duration, and token details come from existing Agent Builder and workflow telemetry.

Managed workflows

• system-nightshift-sandbox-materialize-workspace: obtain sandbox, then Cortex and Memory in parallel (single attempt, 45s fail-fast), then compose model and workflow context.
• system-nightshift-agent-optimize: obtain sandbox, then Cortex and Memory optimize in settled parallel branches. It replaces the standalone Cortex optimize workflow, declares tool_calls and tool_results (Agent Builder only sends tool_results to workflows that declare it), and passes both to Cortex and Memory.
• system-nightshift-decision-tree-hydrate: restores stored decision trees once per round for the investigation and reinforcement agents.

Known limitations

• The writer integrates the replaced memories, so a line copied from Cortex before the evidence filter can persist through later updates. The filter stops new seeded content; it doesn't clean old memories.
• An active topic is rewritten most rounds (refreshed counts and "last confirmed" dates), and a merge can archive a memory a later round would have recalled. Tracked for evals.
• The writer sometimes keeps an unverified inference from a replaced memory or a stale present-tense sentence. These persist with tool results, so they are prompt-following issues, tracked for evals.
• The seeded-path filter is a heuristic: it matches the known roots and docs by path parameter or bash command, not every way a tool could read them.

Test plan

• CI: pending on cde9c6ad51d (after merging main and regenerating the workflow step schemas).
• Nightshift plugin: 894 tests, run in-band on cde9c6ad51d. Covers the evidence-only filter (process tools, seeded paths by parameter and bash command, memory reads kept, critique and extraction inputs), extraction skipped without evidence output (empty, reasoning-only, empty-result and seeded-only rounds), the writer budget and compaction, slug idempotency and the truncation boundary, Space isolation, counters, budgets, merge and create-race recovery, OCC retries, source preservation, round-context handoff, telemetry, managed workflow wiring, the Memory transcript renderer (ordering, results, error marking, budgets, fallback, building from hook results), topic entries (new, update, merge, a live memory at the entry's id joining unnamed), the writer writing every entry and skipping empty content, the task as recall key, archived-id reuse, and harmful-memory handling.
• Managed workflow definitions: 1,723 tests on cde9c6ad51d. This includes real-YAML checks that resume-only steps run at index 0 and not at 1 or 2, that both strict post-execution schemas declare the inputs Agent Builder sends (now including tool_results), and that both optimize branches receive tool_results as an array.
• Real Elasticsearch integration (2 tests): the production mapping and hidden setting, lexical and browse retrieval, Space isolation, concurrent counter increments, and archiving.
• Static checks: Nightshift type-check and lint pass; workflow step schemas regenerated.
• Agent Builder: the generic round-context persistence, replay, HITL, and compaction behavior is covered in [Agent Builder] Pass model and workflow context through workflow hooks #292787 (merged) and not duplicated here.
• LLM task audit (local, gitignored, reads the Phoenix trace of the optimizer's LLM calls, plus a written AI judgment of every call) on 5cf304cf8f1: the critique and extraction got every evidence call's result (6 of 19 calls were evidence) and none of the dropped calls; no Cortex, decision-tree or environment-doc content reached any Memory call; extraction and the writer got no final answer. The judgments found the limitations above.
• Live end-to-end with a real Claude Sonnet connector, driving the Agent Builder API and managed workflows the UI uses (local, gitignored probes). It validates wiring and persistence, not answer quality.
  • On 3692d5ea962: round 1 26/26; the critique and extraction got only the 7 evidence calls of 20, the writer's prompt carried the budget and it wrote 2,822 characters (no compaction needed).
  • On 5cf304cf8f1: round 1 26/26; round 2 40/44, where the 4 failures are seed-specific checks after round 1 correctly merged the seed into a new memory. On f94bcc3629d (after the [Nightshift] Remove model selection from Model Settings #293894 merge): release checks 16/16 (optimize gets the round connector, spend tagged) and HITL 7/7.
  • On d548aa2f53c: two rounds, 26/26 each, both optimize branches completed. One round updated two memories in place; the other wrote a new memory through the writer. Every written memory's context was that round's task.
  • On a3e6f2fd159: two rounds, 27/27 and 25/25 assertions. The second round updated a memory in place with a recurrence found only in a query result.
  • On f74c1e369a6 (earlier head): two ordinary turns, 44/44 assertions (memory retrieved and materialized, announced on the first turn only, recalled IDs persisted per round, user message unchanged, hidden context replayed verbatim, no memory IDs in workflow logs, counters persisted); every optimize LLM call carried feature_id=significant_events_investigation (9/9); HITL resume 7/7 (materialize and tree hydrate ran only at index 0, optimize and reinforce each ran once with the round's recalled IDs and connector).