fix(security): html-escape memory context summaries rendered into the injection prompt - #4119

Review blockers (vendored DeerMem):
- bytedance#4044 restore _escape_memory_for_prompt (current_memory blob in
  MEMORY_UPDATE_PROMPT) - prevents </current_memory> breakout
- bytedance#4028 html.escape staleness-section cat/content in _build_staleness_section
- bytedance#4119 add _escape_summary for injection-path summaries (Work/Personal/
  Current Focus/Recent/Earlier/Background)
- default-model silent no-op: factory injects host default chat model via a
  new host_llm slot (create_chat_model(name=None)); DeerMem prefers host_llm
  over build_llm(model). Zero-config extraction works out of the box again
- MemoryConfigResponse: fix stale docstring (backend-agnostic shape; DeerMem
  knobs live under backend_config, not top-level - restoring flat would
  re-couple the API to DeerMem). Frontend audited: does not read /memory/config
- _host_default_tracing_callback: restore langfuse assistant_id/environment
- search: push category onto the ABC signature; DeerMem filters BEFORE the
  top_k slice (was filtered client-side after slicing -> starved results)
- _do_update_memory_sync: split into wrapper+impl; bind trace_id into the
  request-trace ContextVar on the Timer/executor worker via a new
  trace_context_manager host hook (None trace_id left unbound - no fabrication)
- client.py fact-CRUD now passes user_id (was writing to the global bucket
  while get_memory reads per-user)
- _resolve_manager_class: fail-fast (raise ValueError) on an unresolved
  explicit manager_class instead of silently falling back to DeerMem (memory is
  persistent state - a wrong store is a silent data-integrity footgun)

Upstream memory fixes dropped by the host->vendored rename conflict, re-ported
to backends/deermem/deermem/core/ (+ deer_mem.py):
- bytedance#4073 queue busy-timer-spin -> _reprocess_pending flag (core/queue.py)
- bytedance#4074 null source.confidence in staleness -> _coerce_source_confidence
  (core/updater.py: _build_staleness_section + _apply_updates stale sort)
- bytedance#4075 factsToRemove is optional (drop from _REQUIRED_MEMORY_UPDATE_TOP_LEVEL_KEYS)
- bytedance#4076 null confidence in search ranking -> _coerce_source_confidence
  (deer_mem.py DeerMem.search)

host_llm + trace_context_manager are host-injected via backend_config (factory
in manager.py), keeping backends/deermem/ at exactly one `from deerflow` line
(the ABC contract) - portability test preserved.

Co-Authored-By: Claude <noreply@anthropic.com>

Brings in 4fd521e..2bd0f56: sandbox fixes (bytedance#4051/bytedance#4053/bytedance#4058/bytedance#4078/bytedance#4079/bytedance#4110),
CVE-2026-33128/bytedance#4089/bytedance#4093/bytedance#4106, subagents (bytedance#4056/bytedance#4059/bytedance#4088), skillscan (bytedance#4087),
channels (bytedance#4055/bytedance#4069/bytedance#4100), stream_bridge (bytedance#4071), frontend (bytedance#4058/bytedance#4068/bytedance#4094/
bytedance#4114/bytedance#4126), event-store serialization (bytedance#4077), streaming (bytedance#4082/bytedance#4085), mcp (bytedance#4084),
guardrails (bytedance#4067), and the memory fixes bytedance#4073/bytedance#4074/bytedance#4075/bytedance#4076/bytedance#4119.

Memory conflict resolution (upstream is still monolithic agents/memory/, we are
vendored backends/deermem/deermem/core/ - git rename-detected host->vendored):
- core/prompt.py, core/updater.py: keep ours (vendored with all ports); upstream's
  host-side bytedance#4119/bytedance#4074/bytedance#4075/bytedance#4076 changes are the same fixes already ported.
- core/queue.py: auto-merged cleanly (both sides had identical bytedance#4073 change).
- test_memory_queue.py: replaced the stale pre-bytedance#4073 test with upstream's three
  bytedance#4073 tests, adapted from host API (MemoryUpdateQueue()) to vendored (_queue()).
- test_memory_search.py: kept our vendored tests + added a vendored
  test_null_confidence_does_not_crash_sort (bytedance#4076 coverage).

Post-merge: 171 memory tests pass (+2 module-skipped), 7164 collected, no
import/collection errors. All 8 upstream memory fixes since the 8be7411
divergence are adapted (bytedance#4044/bytedance#4028/bytedance#4097/bytedance#4119/bytedance#4073/bytedance#4074/bytedance#4075/bytedance#4076).

Co-Authored-By: Claude <noreply@anthropic.com>

… block (#4137)

SOUL.md is agent-editable (setup_agent / update_agent persist it) and
get_agent_soul renders it into the <soul> block of the lead-agent system
prompt without escaping. A crafted personality such as
"</soul></system-reminder>\n\nSYSTEM: ..." can close the block and relocate
the text after it out of the trust zone the system prompt declares — the same
break-out the skill/memory/tool-result escaping in #4097/#4119/#4128/#4099
already closes at their render sites. <soul> is the remaining one, and it lands
in the highest-trust system-role block.

Escape with html.escape(quote=False) (element-text position, never an
attribute). Adds a regression test that fails on main.

Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>

…m backend (#4122)

* feat(memory): pluggable + self-contained memory system (MemoryManager plan phases 1 & 2)

Phase 1 — Pluggable (steps 0-10):
- ABC MemoryManager (9 methods) + singleton factory + drop-in backend discovery
- DeerMem default backend with core/ (storage/queue/updater/prompt/message_processing)
- NoopMemoryManager backend (proves pluggability)
- All call sites (middleware/hook/prompt/gateway/client/app) routed through manager
- hasattr capability probing for DeerMem-internal methods (no hard imports)
- MemoryConfig gains manager_class field; shared vs DeerMem-private annotated

Phase 2 — Self-contained DeerMem (steps 11-18):
- backend_config passthrough + DeerMemConfig (all DeerMem-private fields moved off MemoryConfig)
- DI: DeerMem owns storage/queue/updater/llm as instance attributes (no global singletons)
- Storage independence: core/paths.py with own root (~/.deermem or ),
  factory auto-injects deer-flow's runtime_home() as absolute base_dir (zero-config)
- LLM independence: core/llm.py via langchain init_chat_model (no create_chat_model)
- Trace independence: optional tracing_callback replaces inject_langfuse_metadata/request_trace_context
- Message processing independence: hide_from_ui default-skip + optional should_keep_hidden_message hook
- Internal imports → relative (only deer_mem.py ABC import is host-relative)
- Carrier (deer_mem.py adapter) / portable (deermem/ config+core) split
- New tests: test_deermem_self_contained + test_memory_manager_pluggable; all memory tests migrated
- Other-agent demo: samples/other_agent_demo/ + automated portability test
- config.example.yaml memory section updated to phase-2 schema

* feat(memory): port consolidation + staleness fix into self-contained DeerMem; phase-2 host hooks

Port upstream #3996 (memory consolidation) and #3993 (staleness KeyError fix)
from origin/MemoryManager into the pluggable, self-contained DeerMem structure
(backends/deermem/deermem/), adapted to the DI MemoryUpdater (config injected,
not get_memory_config globals):

- DeerMemConfig: add consolidation_enabled (opt-in, default false) /
  consolidation_min_facts / consolidation_max_groups_per_cycle /
  consolidation_max_sources
- prompt.py: factsToConsolidate JSON field + {consolidation_section} placeholder
  + CONSOLIDATION_PROMPT constant
- updater.py: _coerce_source_confidence / _select_consolidation_candidates /
  _build_consolidation_section module helpers (matching the existing
  _select_stale_candidates style); consolidation normalization in
  _normalize_memory_update_data; consolidation apply in _apply_updates (after
  max_facts trim, with apply-time guardrails mirroring staleness); staleness
  KeyError fix (f["id"] -> f.get("id") is not None) applied to both the
  staleness guardrail and the consolidation allowed_source_ids comprehension
- config.example.yaml: consolidation section under memory.backend_config
- tests/test_memory_consolidation.py: 40 DI-adapted tests (running, not skipped)
  incl. the staleness KeyError regression

Also includes in-flight phase-2 host-integration work: storage_path semantics
(any absolute/relative value = root dir) and host-default tracing_callback /
should_keep_hidden_message hooks injected into backend_config by the factory.

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(memory): add noop backend template and backends guide

- backends/noop/: complete drop-in template (config.py with zero deer-flow
  imports, noop_manager.py with a 6-step new-backend walkthrough in its
  docstring, commented optional fact-CRUD capabilities).
- backends/README.md: which files to touch when adding/swapping a backend,
  the 5-item backend contract, and common pitfalls.
- manager.py: generalize backend examples in comments (drop mem0-specific
  references).

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(frontend): guard formatTimeAgo against invalid timestamps

Return a neutral placeholder when the input date is invalid (e.g. an empty lastUpdated from a backend with no memories) instead of throwing 'Invalid time value' from date-fns.

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(memory): wire tool-driven memory mode through the MemoryManager ABC

tools.py (memory_search/add/update/delete) now calls get_memory_manager()
instead of the removed host memory module, so tool mode (memory.mode: tool)
works for any backend. DeerMem.search is implemented (case-insensitive
substring match, ranked by confidence) as a stand-in for the planned
semantic retrieval; noop.search returns [] (unchanged). Fact-CRUD tools
use getattr+callable probing -- backends lacking those ops (noop) get a
clear JSON error instead of crashing.

Tests: test_memory_tools rewired to mock the manager (handler tests) +
TestModeGating retained; test_memory_search now covers DeerMem.search;
pluggable stubs test updated (search no longer a stub).

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: resolve lint errors (import sorting, type annotation quotes, E402 in skipped tests)

* docs: restore explanatory comments in config.example.yaml memory section

* fix(security): port html-escape memory facts fix (#4097) to vendored DeerMem prompt.py

* fix(memory): address review + port dropped upstream memory fixes

Review blockers (vendored DeerMem):
- #4044 restore _escape_memory_for_prompt (current_memory blob in
  MEMORY_UPDATE_PROMPT) - prevents </current_memory> breakout
- #4028 html.escape staleness-section cat/content in _build_staleness_section
- #4119 add _escape_summary for injection-path summaries (Work/Personal/
  Current Focus/Recent/Earlier/Background)
- default-model silent no-op: factory injects host default chat model via a
  new host_llm slot (create_chat_model(name=None)); DeerMem prefers host_llm
  over build_llm(model). Zero-config extraction works out of the box again
- MemoryConfigResponse: fix stale docstring (backend-agnostic shape; DeerMem
  knobs live under backend_config, not top-level - restoring flat would
  re-couple the API to DeerMem). Frontend audited: does not read /memory/config
- _host_default_tracing_callback: restore langfuse assistant_id/environment
- search: push category onto the ABC signature; DeerMem filters BEFORE the
  top_k slice (was filtered client-side after slicing -> starved results)
- _do_update_memory_sync: split into wrapper+impl; bind trace_id into the
  request-trace ContextVar on the Timer/executor worker via a new
  trace_context_manager host hook (None trace_id left unbound - no fabrication)
- client.py fact-CRUD now passes user_id (was writing to the global bucket
  while get_memory reads per-user)
- _resolve_manager_class: fail-fast (raise ValueError) on an unresolved
  explicit manager_class instead of silently falling back to DeerMem (memory is
  persistent state - a wrong store is a silent data-integrity footgun)

Upstream memory fixes dropped by the host->vendored rename conflict, re-ported
to backends/deermem/deermem/core/ (+ deer_mem.py):
- #4073 queue busy-timer-spin -> _reprocess_pending flag (core/queue.py)
- #4074 null source.confidence in staleness -> _coerce_source_confidence
  (core/updater.py: _build_staleness_section + _apply_updates stale sort)
- #4075 factsToRemove is optional (drop from _REQUIRED_MEMORY_UPDATE_TOP_LEVEL_KEYS)
- #4076 null confidence in search ranking -> _coerce_source_confidence
  (deer_mem.py DeerMem.search)

host_llm + trace_context_manager are host-injected via backend_config (factory
in manager.py), keeping backends/deermem/ at exactly one `from deerflow` line
(the ABC contract) - portability test preserved.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: resolve lint errors (F541 f-string without placeholders, E501 line too long)

* fix(memory): restore hide_from_ui clarification preservation, expose mode

Two memory-system fixes (F541/E501 lint was already fixed on this branch):

- filter_messages_for_memory: restore default preservation of well-formed
  human_input_response clarification answers (v2 regression). The
  self-containment refactor made the bare function skip ALL hide_from_ui when
  no hook was passed, but upstream preserves well-formed clarification
  responses by default (test_hide_from_ui_human_input_response_is_preserved).
  Inline a host-agnostic _is_human_clarification_response mirror of
  read_human_input_response as the default keep-decision; the host-injected
  should_keep_hidden_message hook still overrides (production path unchanged).
  Portable package stays zero `from deerflow`.

- /memory/config: expose `mode` (middleware|tool) in MemoryConfigResponse +
  the config/status endpoints + client.get_memory_config. mode is a host-
  shared, behavior-determining field missing from the response projection.
  Sync tests (mock .mode; e2e assert mode present).

- Align manager_class field docstring with fail-fast behavior.

Tests: filter/self-contained/portability (35) + memory-config (4) pass;
ruff clean.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(memory): resolve ruff format failures in memory module + tests

`make lint` runs `ruff format --check` in addition to `ruff check`; 8 memory
files had pending format changes -- 7 pre-existing (deer_mem, updater, tools,
test_memory_queue/router/search/tools) + message_processing from the
hide_from_ui fix. Apply `ruff format`: whitespace/wrapping only, no logic
change. 109 memory tests pass; ruff check + format --check both clean.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(memory): address PR review - legacy field migration, fact_id contract, path/docs

Address willem-bd's review on PR head bc8bf0d (risk:high, persistent state):

- config: auto-migrate pre-abstraction top-level memory.* DeerMem fields
  (storage_path, max_facts, debounce_seconds, model_name, token_counting,
  staleness_*, consolidation_*) into backend_config on load + warn, so an
  upgrade does NOT silently revert customized settings (was: silent
  extra='ignore' drop). model_name -> backend_config.model.model. Unknown
  top-level keys warned.
- factory: resolve a relative backend_config.storage_path against runtime_home()
  (base_dir-relative, CWD-independent) to preserve pre-abstraction semantics;
  paths.py stays portable (no runtime_home import).
- tools: memory_add uses the fact_id returned directly by create_fact instead of
  re-deriving it via content-key matching (coupled the tool to the backend's
  content normalization; could misreport a storage cap). create_fact now returns
  (memory_data, fact_id); gateway/client/tool updated. Fix terse
  {"error":"content"} -> {"error":"empty content"}.
- app.py: update stale token_counting=="char" warm-up comment to point at
  manager.warm (DeerMem.warm re-checks char and returns early).
- router: comment explaining reload_memory silent fallback vs fact 501 asymmetry
  (read-only degrade vs write fail-loud).
- CHANGELOG: document breaking changes (/memory/config + client.get_memory_config
  shape flat->backend_config; custom storage_class path moved + __init__ must
  accept config) and the legacy-field auto-migration.
- tests: add regression test pinning the per-user memory path
  ({storage_path}/users/{safe_user_id}/memory.json == host make_safe_user_id)
  across the abstraction; update create_fact mocks for (memory_data, fact_id).

Tests: 273 passed (memory suite); ruff check + format clean.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(memory): address PR review - storage_path, max_facts, tracing, parsing

Six review findings (willem-bd), each verified against upstream:

- storage_path semantics (file -> root dir): migration drops file-style
  (.json) legacy values with a warning; factory raises if storage_path
  resolves to an existing file (avoid silent NotADirectoryError write
  failure). CHANGELOG + config.example.yaml comment updated.
- create_memory_fact enforces max_facts again (via _trim_facts_to_max) and
  returns (memory, None) when the cap evicts the new fact; memory_add tool
  reports "not stored", client raises ValueError, POST /memory/facts -> 409.
- max_facts trim uses _coerce_source_confidence (was raw f.get("confidence",
  0) -> TypeError on non-float imported/legacy confidence, swallowed as
  silent update failure).
- memory-tracing assistant_id restored to "memory_agent" (was "lead-agent"
  copy-paste; matches upstream + DeerMem run_name).
- _is_human_clarification_response cross-checked against
  read_human_input_response (drift guard test).
- empty-string legacy values skipped silently in migration (narrow fix, not
  broad "if not value" which would skip explicit bool False).

8 new regression tests. make lint + 406 memory tests pass.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(memory): address internal review - storage fail-fast, build_llm degrade, config warn, noop template

Addresses 4 findings from the PR #4122 internal supplemental review
(parallel to willem-bd's review, no overlap):

- create_storage fail-fast: a misspelled/unimportable storage_class now
  raises ValueError instead of silently falling back to FileMemoryStorage.
  Memory is persistent state, so a wrong store is a data-integrity footgun;
  mirrors the existing manager_class resolution policy. (storage.py)

- noop template create_fact signature: the commented template used
  keyword-only `content` and returned a bare dict, while DeerMem's actual
  create_fact takes positional `content` and returns tuple[dict, str|None]
  (the memory_add tool passes content positionally; gateway/client/tools all
  tuple-unpack). A backend copied from the template would 500 on fact-CRUD.
  Template fixed; delete_fact/update_fact templates left (callers compatible).
  (noop_manager.py)

- build_llm graceful degrade: wrap init_chat_model in try/except, degrade to
  None + WARNING on failure (mirroring _host_default_llm) so a misconfigured
  explicit model does not crash app startup -- non-LLM memory ops still work
  and an update raises at runtime with the error logged. (llm.py)

- from_backend_config unknown-key warning: log a WARNING for unknown
  backend_config keys (mirrors the host layer's load_memory_config_from_dict)
  so a typo like `storage_pat` does not silently fall back to the default and
  write memory to an unintended location. (config.py)

Tests: rewrote 3 create_storage fallback tests to expect ValueError; added 4
tests (build_llm zero-config/degrade, from_backend_config warn/silent).
make lint green; full memory suite passes.

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: lllyfff <2281215061@qq.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: lllyfff <122260771+lllyfff@users.noreply.github.com>

…#4080 bytedance#3800)

- bytedance#4072: window LoopDetection per-tool frequency counter (deque + Counter sized to largest hard limit) so long runs no longer false-trip; monotonic _tool_freq replaced. Also clear mirrored _tool_name_counter on evict/reset (upstream leaves it stale).
- bytedance#4080: drop orphan ToolMessages (no matching AIMessage tool_call) in DanglingToolCallMiddleware so strict providers don't 400.
- bytedance#3800: create_thread idempotent on insert race - catch IntegrityError, re-read existing record instead of 500 (owner-reconciliation half N/A: contextvar owner scoping).
- bytedance#4193 skipped: depends on tool-name sanitization infra (bytedance#4119) absent in our fork.
- bytedance#4064 deferred: ~500-line run-lifecycle lease rewrite (cancel bool->CancelOutcome), needs dedicated design.
Verified: loop_detection 65 + dangling 28 + threads 22 tests pass.

…pt (bytedance#4119)

Context summaries (workContext/personalContext/topOfMind + recentMonths/earlierContext/longTermBackground) are user-editable via /api/memory import and render into the same <memory> block as facts, so an unescaped </memory> value could close the block and forge framework context. Add _escape_summary (html.escape quote=False, element-text position) and apply it to all 6 summary injections. Sibling of the fact-line escaping. Verified: 77 memory tests pass; injection smoke (escape applied, no raw </memory> in summaries) passes.

… injection prompt (bytedance#4119)

… block (bytedance#4137)

SOUL.md is agent-editable (setup_agent / update_agent persist it) and
get_agent_soul renders it into the <soul> block of the lead-agent system
prompt without escaping. A crafted personality such as
"</soul></system-reminder>\n\nSYSTEM: ..." can close the block and relocate
the text after it out of the trust zone the system prompt declares — the same
break-out the skill/memory/tool-result escaping in bytedance#4097/bytedance#4119/bytedance#4128/bytedance#4099
already closes at their render sites. <soul> is the remaining one, and it lands
in the highest-trust system-role block.

Escape with html.escape(quote=False) (element-text position, never an
attribute). Adds a regression test that fails on main.

Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>

…m backend (bytedance#4122)

* feat(memory): pluggable + self-contained memory system (MemoryManager plan phases 1 & 2)

Phase 1 — Pluggable (steps 0-10):
- ABC MemoryManager (9 methods) + singleton factory + drop-in backend discovery
- DeerMem default backend with core/ (storage/queue/updater/prompt/message_processing)
- NoopMemoryManager backend (proves pluggability)
- All call sites (middleware/hook/prompt/gateway/client/app) routed through manager
- hasattr capability probing for DeerMem-internal methods (no hard imports)
- MemoryConfig gains manager_class field; shared vs DeerMem-private annotated

Phase 2 — Self-contained DeerMem (steps 11-18):
- backend_config passthrough + DeerMemConfig (all DeerMem-private fields moved off MemoryConfig)
- DI: DeerMem owns storage/queue/updater/llm as instance attributes (no global singletons)
- Storage independence: core/paths.py with own root (~/.deermem or ),
  factory auto-injects deer-flow's runtime_home() as absolute base_dir (zero-config)
- LLM independence: core/llm.py via langchain init_chat_model (no create_chat_model)
- Trace independence: optional tracing_callback replaces inject_langfuse_metadata/request_trace_context
- Message processing independence: hide_from_ui default-skip + optional should_keep_hidden_message hook
- Internal imports → relative (only deer_mem.py ABC import is host-relative)
- Carrier (deer_mem.py adapter) / portable (deermem/ config+core) split
- New tests: test_deermem_self_contained + test_memory_manager_pluggable; all memory tests migrated
- Other-agent demo: samples/other_agent_demo/ + automated portability test
- config.example.yaml memory section updated to phase-2 schema

* feat(memory): port consolidation + staleness fix into self-contained DeerMem; phase-2 host hooks

Port upstream bytedance#3996 (memory consolidation) and bytedance#3993 (staleness KeyError fix)
from origin/MemoryManager into the pluggable, self-contained DeerMem structure
(backends/deermem/deermem/), adapted to the DI MemoryUpdater (config injected,
not get_memory_config globals):

- DeerMemConfig: add consolidation_enabled (opt-in, default false) /
  consolidation_min_facts / consolidation_max_groups_per_cycle /
  consolidation_max_sources
- prompt.py: factsToConsolidate JSON field + {consolidation_section} placeholder
  + CONSOLIDATION_PROMPT constant
- updater.py: _coerce_source_confidence / _select_consolidation_candidates /
  _build_consolidation_section module helpers (matching the existing
  _select_stale_candidates style); consolidation normalization in
  _normalize_memory_update_data; consolidation apply in _apply_updates (after
  max_facts trim, with apply-time guardrails mirroring staleness); staleness
  KeyError fix (f["id"] -> f.get("id") is not None) applied to both the
  staleness guardrail and the consolidation allowed_source_ids comprehension
- config.example.yaml: consolidation section under memory.backend_config
- tests/test_memory_consolidation.py: 40 DI-adapted tests (running, not skipped)
  incl. the staleness KeyError regression

Also includes in-flight phase-2 host-integration work: storage_path semantics
(any absolute/relative value = root dir) and host-default tracing_callback /
should_keep_hidden_message hooks injected into backend_config by the factory.

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(memory): add noop backend template and backends guide

- backends/noop/: complete drop-in template (config.py with zero deer-flow
  imports, noop_manager.py with a 6-step new-backend walkthrough in its
  docstring, commented optional fact-CRUD capabilities).
- backends/README.md: which files to touch when adding/swapping a backend,
  the 5-item backend contract, and common pitfalls.
- manager.py: generalize backend examples in comments (drop mem0-specific
  references).

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(frontend): guard formatTimeAgo against invalid timestamps

Return a neutral placeholder when the input date is invalid (e.g. an empty lastUpdated from a backend with no memories) instead of throwing 'Invalid time value' from date-fns.

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(memory): wire tool-driven memory mode through the MemoryManager ABC

tools.py (memory_search/add/update/delete) now calls get_memory_manager()
instead of the removed host memory module, so tool mode (memory.mode: tool)
works for any backend. DeerMem.search is implemented (case-insensitive
substring match, ranked by confidence) as a stand-in for the planned
semantic retrieval; noop.search returns [] (unchanged). Fact-CRUD tools
use getattr+callable probing -- backends lacking those ops (noop) get a
clear JSON error instead of crashing.

Tests: test_memory_tools rewired to mock the manager (handler tests) +
TestModeGating retained; test_memory_search now covers DeerMem.search;
pluggable stubs test updated (search no longer a stub).

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: resolve lint errors (import sorting, type annotation quotes, E402 in skipped tests)

* docs: restore explanatory comments in config.example.yaml memory section

* fix(security): port html-escape memory facts fix (bytedance#4097) to vendored DeerMem prompt.py

* fix(memory): address review + port dropped upstream memory fixes

Review blockers (vendored DeerMem):
- bytedance#4044 restore _escape_memory_for_prompt (current_memory blob in
  MEMORY_UPDATE_PROMPT) - prevents </current_memory> breakout
- bytedance#4028 html.escape staleness-section cat/content in _build_staleness_section
- bytedance#4119 add _escape_summary for injection-path summaries (Work/Personal/
  Current Focus/Recent/Earlier/Background)
- default-model silent no-op: factory injects host default chat model via a
  new host_llm slot (create_chat_model(name=None)); DeerMem prefers host_llm
  over build_llm(model). Zero-config extraction works out of the box again
- MemoryConfigResponse: fix stale docstring (backend-agnostic shape; DeerMem
  knobs live under backend_config, not top-level - restoring flat would
  re-couple the API to DeerMem). Frontend audited: does not read /memory/config
- _host_default_tracing_callback: restore langfuse assistant_id/environment
- search: push category onto the ABC signature; DeerMem filters BEFORE the
  top_k slice (was filtered client-side after slicing -> starved results)
- _do_update_memory_sync: split into wrapper+impl; bind trace_id into the
  request-trace ContextVar on the Timer/executor worker via a new
  trace_context_manager host hook (None trace_id left unbound - no fabrication)
- client.py fact-CRUD now passes user_id (was writing to the global bucket
  while get_memory reads per-user)
- _resolve_manager_class: fail-fast (raise ValueError) on an unresolved
  explicit manager_class instead of silently falling back to DeerMem (memory is
  persistent state - a wrong store is a silent data-integrity footgun)

Upstream memory fixes dropped by the host->vendored rename conflict, re-ported
to backends/deermem/deermem/core/ (+ deer_mem.py):
- bytedance#4073 queue busy-timer-spin -> _reprocess_pending flag (core/queue.py)
- bytedance#4074 null source.confidence in staleness -> _coerce_source_confidence
  (core/updater.py: _build_staleness_section + _apply_updates stale sort)
- bytedance#4075 factsToRemove is optional (drop from _REQUIRED_MEMORY_UPDATE_TOP_LEVEL_KEYS)
- bytedance#4076 null confidence in search ranking -> _coerce_source_confidence
  (deer_mem.py DeerMem.search)

host_llm + trace_context_manager are host-injected via backend_config (factory
in manager.py), keeping backends/deermem/ at exactly one `from deerflow` line
(the ABC contract) - portability test preserved.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: resolve lint errors (F541 f-string without placeholders, E501 line too long)

* fix(memory): restore hide_from_ui clarification preservation, expose mode

Two memory-system fixes (F541/E501 lint was already fixed on this branch):

- filter_messages_for_memory: restore default preservation of well-formed
  human_input_response clarification answers (v2 regression). The
  self-containment refactor made the bare function skip ALL hide_from_ui when
  no hook was passed, but upstream preserves well-formed clarification
  responses by default (test_hide_from_ui_human_input_response_is_preserved).
  Inline a host-agnostic _is_human_clarification_response mirror of
  read_human_input_response as the default keep-decision; the host-injected
  should_keep_hidden_message hook still overrides (production path unchanged).
  Portable package stays zero `from deerflow`.

- /memory/config: expose `mode` (middleware|tool) in MemoryConfigResponse +
  the config/status endpoints + client.get_memory_config. mode is a host-
  shared, behavior-determining field missing from the response projection.
  Sync tests (mock .mode; e2e assert mode present).

- Align manager_class field docstring with fail-fast behavior.

Tests: filter/self-contained/portability (35) + memory-config (4) pass;
ruff clean.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(memory): resolve ruff format failures in memory module + tests

`make lint` runs `ruff format --check` in addition to `ruff check`; 8 memory
files had pending format changes -- 7 pre-existing (deer_mem, updater, tools,
test_memory_queue/router/search/tools) + message_processing from the
hide_from_ui fix. Apply `ruff format`: whitespace/wrapping only, no logic
change. 109 memory tests pass; ruff check + format --check both clean.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(memory): address PR review - legacy field migration, fact_id contract, path/docs

Address willem-bd's review on PR head bc8bf0d (risk:high, persistent state):

- config: auto-migrate pre-abstraction top-level memory.* DeerMem fields
  (storage_path, max_facts, debounce_seconds, model_name, token_counting,
  staleness_*, consolidation_*) into backend_config on load + warn, so an
  upgrade does NOT silently revert customized settings (was: silent
  extra='ignore' drop). model_name -> backend_config.model.model. Unknown
  top-level keys warned.
- factory: resolve a relative backend_config.storage_path against runtime_home()
  (base_dir-relative, CWD-independent) to preserve pre-abstraction semantics;
  paths.py stays portable (no runtime_home import).
- tools: memory_add uses the fact_id returned directly by create_fact instead of
  re-deriving it via content-key matching (coupled the tool to the backend's
  content normalization; could misreport a storage cap). create_fact now returns
  (memory_data, fact_id); gateway/client/tool updated. Fix terse
  {"error":"content"} -> {"error":"empty content"}.
- app.py: update stale token_counting=="char" warm-up comment to point at
  manager.warm (DeerMem.warm re-checks char and returns early).
- router: comment explaining reload_memory silent fallback vs fact 501 asymmetry
  (read-only degrade vs write fail-loud).
- CHANGELOG: document breaking changes (/memory/config + client.get_memory_config
  shape flat->backend_config; custom storage_class path moved + __init__ must
  accept config) and the legacy-field auto-migration.
- tests: add regression test pinning the per-user memory path
  ({storage_path}/users/{safe_user_id}/memory.json == host make_safe_user_id)
  across the abstraction; update create_fact mocks for (memory_data, fact_id).

Tests: 273 passed (memory suite); ruff check + format clean.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(memory): address PR review - storage_path, max_facts, tracing, parsing

Six review findings (willem-bd), each verified against upstream:

- storage_path semantics (file -> root dir): migration drops file-style
  (.json) legacy values with a warning; factory raises if storage_path
  resolves to an existing file (avoid silent NotADirectoryError write
  failure). CHANGELOG + config.example.yaml comment updated.
- create_memory_fact enforces max_facts again (via _trim_facts_to_max) and
  returns (memory, None) when the cap evicts the new fact; memory_add tool
  reports "not stored", client raises ValueError, POST /memory/facts -> 409.
- max_facts trim uses _coerce_source_confidence (was raw f.get("confidence",
  0) -> TypeError on non-float imported/legacy confidence, swallowed as
  silent update failure).
- memory-tracing assistant_id restored to "memory_agent" (was "lead-agent"
  copy-paste; matches upstream + DeerMem run_name).
- _is_human_clarification_response cross-checked against
  read_human_input_response (drift guard test).
- empty-string legacy values skipped silently in migration (narrow fix, not
  broad "if not value" which would skip explicit bool False).

8 new regression tests. make lint + 406 memory tests pass.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(memory): address internal review - storage fail-fast, build_llm degrade, config warn, noop template

Addresses 4 findings from the PR bytedance#4122 internal supplemental review
(parallel to willem-bd's review, no overlap):

- create_storage fail-fast: a misspelled/unimportable storage_class now
  raises ValueError instead of silently falling back to FileMemoryStorage.
  Memory is persistent state, so a wrong store is a data-integrity footgun;
  mirrors the existing manager_class resolution policy. (storage.py)

- noop template create_fact signature: the commented template used
  keyword-only `content` and returned a bare dict, while DeerMem's actual
  create_fact takes positional `content` and returns tuple[dict, str|None]
  (the memory_add tool passes content positionally; gateway/client/tools all
  tuple-unpack). A backend copied from the template would 500 on fact-CRUD.
  Template fixed; delete_fact/update_fact templates left (callers compatible).
  (noop_manager.py)

- build_llm graceful degrade: wrap init_chat_model in try/except, degrade to
  None + WARNING on failure (mirroring _host_default_llm) so a misconfigured
  explicit model does not crash app startup -- non-LLM memory ops still work
  and an update raises at runtime with the error logged. (llm.py)

- from_backend_config unknown-key warning: log a WARNING for unknown
  backend_config keys (mirrors the host layer's load_memory_config_from_dict)
  so a typo like `storage_pat` does not silently fall back to the default and
  write memory to an unintended location. (config.py)

Tests: rewrote 3 create_storage fallback tests to expect ValueError; added 4
tests (build_llm zero-config/degrade, from_backend_config warn/silent).
make lint green; full memory suite passes.

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: lllyfff <2281215061@qq.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: lllyfff <122260771+lllyfff@users.noreply.github.com>

… injection prompt (bytedance#4119)

… block (bytedance#4137)

SOUL.md is agent-editable (setup_agent / update_agent persist it) and
get_agent_soul renders it into the <soul> block of the lead-agent system
prompt without escaping. A crafted personality such as
"</soul></system-reminder>\n\nSYSTEM: ..." can close the block and relocate
the text after it out of the trust zone the system prompt declares — the same
break-out the skill/memory/tool-result escaping in bytedance#4097/bytedance#4119/bytedance#4128/bytedance#4099
already closes at their render sites. <soul> is the remaining one, and it lands
in the highest-trust system-role block.

Escape with html.escape(quote=False) (element-text position, never an
attribute). Adds a regression test that fails on main.

Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>

…m backend (bytedance#4122)

* feat(memory): pluggable + self-contained memory system (MemoryManager plan phases 1 & 2)

Phase 1 — Pluggable (steps 0-10):
- ABC MemoryManager (9 methods) + singleton factory + drop-in backend discovery
- DeerMem default backend with core/ (storage/queue/updater/prompt/message_processing)
- NoopMemoryManager backend (proves pluggability)
- All call sites (middleware/hook/prompt/gateway/client/app) routed through manager
- hasattr capability probing for DeerMem-internal methods (no hard imports)
- MemoryConfig gains manager_class field; shared vs DeerMem-private annotated

Phase 2 — Self-contained DeerMem (steps 11-18):
- backend_config passthrough + DeerMemConfig (all DeerMem-private fields moved off MemoryConfig)
- DI: DeerMem owns storage/queue/updater/llm as instance attributes (no global singletons)
- Storage independence: core/paths.py with own root (~/.deermem or ),
  factory auto-injects deer-flow's runtime_home() as absolute base_dir (zero-config)
- LLM independence: core/llm.py via langchain init_chat_model (no create_chat_model)
- Trace independence: optional tracing_callback replaces inject_langfuse_metadata/request_trace_context
- Message processing independence: hide_from_ui default-skip + optional should_keep_hidden_message hook
- Internal imports → relative (only deer_mem.py ABC import is host-relative)
- Carrier (deer_mem.py adapter) / portable (deermem/ config+core) split
- New tests: test_deermem_self_contained + test_memory_manager_pluggable; all memory tests migrated
- Other-agent demo: samples/other_agent_demo/ + automated portability test
- config.example.yaml memory section updated to phase-2 schema

* feat(memory): port consolidation + staleness fix into self-contained DeerMem; phase-2 host hooks

Port upstream bytedance#3996 (memory consolidation) and bytedance#3993 (staleness KeyError fix)
from origin/MemoryManager into the pluggable, self-contained DeerMem structure
(backends/deermem/deermem/), adapted to the DI MemoryUpdater (config injected,
not get_memory_config globals):

- DeerMemConfig: add consolidation_enabled (opt-in, default false) /
  consolidation_min_facts / consolidation_max_groups_per_cycle /
  consolidation_max_sources
- prompt.py: factsToConsolidate JSON field + {consolidation_section} placeholder
  + CONSOLIDATION_PROMPT constant
- updater.py: _coerce_source_confidence / _select_consolidation_candidates /
  _build_consolidation_section module helpers (matching the existing
  _select_stale_candidates style); consolidation normalization in
  _normalize_memory_update_data; consolidation apply in _apply_updates (after
  max_facts trim, with apply-time guardrails mirroring staleness); staleness
  KeyError fix (f["id"] -> f.get("id") is not None) applied to both the
  staleness guardrail and the consolidation allowed_source_ids comprehension
- config.example.yaml: consolidation section under memory.backend_config
- tests/test_memory_consolidation.py: 40 DI-adapted tests (running, not skipped)
  incl. the staleness KeyError regression

Also includes in-flight phase-2 host-integration work: storage_path semantics
(any absolute/relative value = root dir) and host-default tracing_callback /
should_keep_hidden_message hooks injected into backend_config by the factory.

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(memory): add noop backend template and backends guide

- backends/noop/: complete drop-in template (config.py with zero deer-flow
  imports, noop_manager.py with a 6-step new-backend walkthrough in its
  docstring, commented optional fact-CRUD capabilities).
- backends/README.md: which files to touch when adding/swapping a backend,
  the 5-item backend contract, and common pitfalls.
- manager.py: generalize backend examples in comments (drop mem0-specific
  references).

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(frontend): guard formatTimeAgo against invalid timestamps

Return a neutral placeholder when the input date is invalid (e.g. an empty lastUpdated from a backend with no memories) instead of throwing 'Invalid time value' from date-fns.

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(memory): wire tool-driven memory mode through the MemoryManager ABC

tools.py (memory_search/add/update/delete) now calls get_memory_manager()
instead of the removed host memory module, so tool mode (memory.mode: tool)
works for any backend. DeerMem.search is implemented (case-insensitive
substring match, ranked by confidence) as a stand-in for the planned
semantic retrieval; noop.search returns [] (unchanged). Fact-CRUD tools
use getattr+callable probing -- backends lacking those ops (noop) get a
clear JSON error instead of crashing.

Tests: test_memory_tools rewired to mock the manager (handler tests) +
TestModeGating retained; test_memory_search now covers DeerMem.search;
pluggable stubs test updated (search no longer a stub).

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: resolve lint errors (import sorting, type annotation quotes, E402 in skipped tests)

* docs: restore explanatory comments in config.example.yaml memory section

* fix(security): port html-escape memory facts fix (bytedance#4097) to vendored DeerMem prompt.py

* fix(memory): address review + port dropped upstream memory fixes

Review blockers (vendored DeerMem):
- bytedance#4044 restore _escape_memory_for_prompt (current_memory blob in
  MEMORY_UPDATE_PROMPT) - prevents </current_memory> breakout
- bytedance#4028 html.escape staleness-section cat/content in _build_staleness_section
- bytedance#4119 add _escape_summary for injection-path summaries (Work/Personal/
  Current Focus/Recent/Earlier/Background)
- default-model silent no-op: factory injects host default chat model via a
  new host_llm slot (create_chat_model(name=None)); DeerMem prefers host_llm
  over build_llm(model). Zero-config extraction works out of the box again
- MemoryConfigResponse: fix stale docstring (backend-agnostic shape; DeerMem
  knobs live under backend_config, not top-level - restoring flat would
  re-couple the API to DeerMem). Frontend audited: does not read /memory/config
- _host_default_tracing_callback: restore langfuse assistant_id/environment
- search: push category onto the ABC signature; DeerMem filters BEFORE the
  top_k slice (was filtered client-side after slicing -> starved results)
- _do_update_memory_sync: split into wrapper+impl; bind trace_id into the
  request-trace ContextVar on the Timer/executor worker via a new
  trace_context_manager host hook (None trace_id left unbound - no fabrication)
- client.py fact-CRUD now passes user_id (was writing to the global bucket
  while get_memory reads per-user)
- _resolve_manager_class: fail-fast (raise ValueError) on an unresolved
  explicit manager_class instead of silently falling back to DeerMem (memory is
  persistent state - a wrong store is a silent data-integrity footgun)

Upstream memory fixes dropped by the host->vendored rename conflict, re-ported
to backends/deermem/deermem/core/ (+ deer_mem.py):
- bytedance#4073 queue busy-timer-spin -> _reprocess_pending flag (core/queue.py)
- bytedance#4074 null source.confidence in staleness -> _coerce_source_confidence
  (core/updater.py: _build_staleness_section + _apply_updates stale sort)
- bytedance#4075 factsToRemove is optional (drop from _REQUIRED_MEMORY_UPDATE_TOP_LEVEL_KEYS)
- bytedance#4076 null confidence in search ranking -> _coerce_source_confidence
  (deer_mem.py DeerMem.search)

host_llm + trace_context_manager are host-injected via backend_config (factory
in manager.py), keeping backends/deermem/ at exactly one `from deerflow` line
(the ABC contract) - portability test preserved.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: resolve lint errors (F541 f-string without placeholders, E501 line too long)

* fix(memory): restore hide_from_ui clarification preservation, expose mode

Two memory-system fixes (F541/E501 lint was already fixed on this branch):

- filter_messages_for_memory: restore default preservation of well-formed
  human_input_response clarification answers (v2 regression). The
  self-containment refactor made the bare function skip ALL hide_from_ui when
  no hook was passed, but upstream preserves well-formed clarification
  responses by default (test_hide_from_ui_human_input_response_is_preserved).
  Inline a host-agnostic _is_human_clarification_response mirror of
  read_human_input_response as the default keep-decision; the host-injected
  should_keep_hidden_message hook still overrides (production path unchanged).
  Portable package stays zero `from deerflow`.

- /memory/config: expose `mode` (middleware|tool) in MemoryConfigResponse +
  the config/status endpoints + client.get_memory_config. mode is a host-
  shared, behavior-determining field missing from the response projection.
  Sync tests (mock .mode; e2e assert mode present).

- Align manager_class field docstring with fail-fast behavior.

Tests: filter/self-contained/portability (35) + memory-config (4) pass;
ruff clean.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(memory): resolve ruff format failures in memory module + tests

`make lint` runs `ruff format --check` in addition to `ruff check`; 8 memory
files had pending format changes -- 7 pre-existing (deer_mem, updater, tools,
test_memory_queue/router/search/tools) + message_processing from the
hide_from_ui fix. Apply `ruff format`: whitespace/wrapping only, no logic
change. 109 memory tests pass; ruff check + format --check both clean.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(memory): address PR review - legacy field migration, fact_id contract, path/docs

Address willem-bd's review on PR head bc8bf0d (risk:high, persistent state):

- config: auto-migrate pre-abstraction top-level memory.* DeerMem fields
  (storage_path, max_facts, debounce_seconds, model_name, token_counting,
  staleness_*, consolidation_*) into backend_config on load + warn, so an
  upgrade does NOT silently revert customized settings (was: silent
  extra='ignore' drop). model_name -> backend_config.model.model. Unknown
  top-level keys warned.
- factory: resolve a relative backend_config.storage_path against runtime_home()
  (base_dir-relative, CWD-independent) to preserve pre-abstraction semantics;
  paths.py stays portable (no runtime_home import).
- tools: memory_add uses the fact_id returned directly by create_fact instead of
  re-deriving it via content-key matching (coupled the tool to the backend's
  content normalization; could misreport a storage cap). create_fact now returns
  (memory_data, fact_id); gateway/client/tool updated. Fix terse
  {"error":"content"} -> {"error":"empty content"}.
- app.py: update stale token_counting=="char" warm-up comment to point at
  manager.warm (DeerMem.warm re-checks char and returns early).
- router: comment explaining reload_memory silent fallback vs fact 501 asymmetry
  (read-only degrade vs write fail-loud).
- CHANGELOG: document breaking changes (/memory/config + client.get_memory_config
  shape flat->backend_config; custom storage_class path moved + __init__ must
  accept config) and the legacy-field auto-migration.
- tests: add regression test pinning the per-user memory path
  ({storage_path}/users/{safe_user_id}/memory.json == host make_safe_user_id)
  across the abstraction; update create_fact mocks for (memory_data, fact_id).

Tests: 273 passed (memory suite); ruff check + format clean.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(memory): address PR review - storage_path, max_facts, tracing, parsing

Six review findings (willem-bd), each verified against upstream:

- storage_path semantics (file -> root dir): migration drops file-style
  (.json) legacy values with a warning; factory raises if storage_path
  resolves to an existing file (avoid silent NotADirectoryError write
  failure). CHANGELOG + config.example.yaml comment updated.
- create_memory_fact enforces max_facts again (via _trim_facts_to_max) and
  returns (memory, None) when the cap evicts the new fact; memory_add tool
  reports "not stored", client raises ValueError, POST /memory/facts -> 409.
- max_facts trim uses _coerce_source_confidence (was raw f.get("confidence",
  0) -> TypeError on non-float imported/legacy confidence, swallowed as
  silent update failure).
- memory-tracing assistant_id restored to "memory_agent" (was "lead-agent"
  copy-paste; matches upstream + DeerMem run_name).
- _is_human_clarification_response cross-checked against
  read_human_input_response (drift guard test).
- empty-string legacy values skipped silently in migration (narrow fix, not
  broad "if not value" which would skip explicit bool False).

8 new regression tests. make lint + 406 memory tests pass.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(memory): address internal review - storage fail-fast, build_llm degrade, config warn, noop template

Addresses 4 findings from the PR bytedance#4122 internal supplemental review
(parallel to willem-bd's review, no overlap):

- create_storage fail-fast: a misspelled/unimportable storage_class now
  raises ValueError instead of silently falling back to FileMemoryStorage.
  Memory is persistent state, so a wrong store is a data-integrity footgun;
  mirrors the existing manager_class resolution policy. (storage.py)

- noop template create_fact signature: the commented template used
  keyword-only `content` and returned a bare dict, while DeerMem's actual
  create_fact takes positional `content` and returns tuple[dict, str|None]
  (the memory_add tool passes content positionally; gateway/client/tools all
  tuple-unpack). A backend copied from the template would 500 on fact-CRUD.
  Template fixed; delete_fact/update_fact templates left (callers compatible).
  (noop_manager.py)

- build_llm graceful degrade: wrap init_chat_model in try/except, degrade to
  None + WARNING on failure (mirroring _host_default_llm) so a misconfigured
  explicit model does not crash app startup -- non-LLM memory ops still work
  and an update raises at runtime with the error logged. (llm.py)

- from_backend_config unknown-key warning: log a WARNING for unknown
  backend_config keys (mirrors the host layer's load_memory_config_from_dict)
  so a typo like `storage_pat` does not silently fall back to the default and
  write memory to an unintended location. (config.py)

Tests: rewrote 3 create_storage fallback tests to expect ValueError; added 4
tests (build_llm zero-config/degrade, from_backend_config warn/silent).
make lint green; full memory suite passes.

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: lllyfff <2281215061@qq.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: lllyfff <122260771+lllyfff@users.noreply.github.com>