feat(compression): add audit-safe mode with protected pattern matching - #1899
Merged
chopratejas merged 1 commit intoJul 9, 2026
Merged
Conversation
headroomlabs-ai#1705) Rare, audit-relevant rows (leakage flags, compliance markers, error signatures) can be sampled out by SmartCrusher's purely statistical row selection, or moved behind an opaque <<ccr:...>> retrieval marker the model never asks for. audit_safe=True + protected_patterns scans rows before compression and guarantees matched rows survive verbatim afterward, on both crush_array_json and the _smart_crush_content path apply() actually runs for tool-output compression. A post-splice verification pass fails closed (or ships best-effort with a warning) if protected rows still can't be preserved.
rodboev
requested review from
DevanshiVyas,
JerrettDavis and
chopratejas
as code owners
July 9, 2026 01:45
Contributor
PR governanceThis PR follows the template and is marked ready for human review. |
Merged
chopratejas
pushed a commit
that referenced
this pull request
Jul 9, 2026
🤖 I have created a release *beep* *boop* --- <details><summary>0.31.0</summary> ## [0.31.0](v0.30.0...v0.31.0) (2026-07-09) ### Features * **cache:** provider-agnostic cache-mode delta + cc-agnostic prefix comparison ([#1868](#1868)) ([7c2f0ea](7c2f0ea)) * **ccr:** wire retrieve-tool interception into OpenAI Responses handler ([#1898](#1898)) ([62cd307](62cd307)) * **compression:** add audit-safe mode with protected pattern matching ([#1899](#1899)) ([bb112dd](bb112dd)) * **content-router:** accept any real compression (remove min-savings floor) ([#1771](#1771)) ([6c31db9](6c31db9)) * **content-router:** lossless-first dispatch, cross-turn dedup, and A7 lossy-after-fold ([#1818](#1818)) ([60af15f](60af15f)) * **proxy:** add provider-only HTTP proxy ([#1807](#1807)) ([ebe0a3b](ebe0a3b)) * **proxy:** add turn-hook extension point for buffered model turns ([#1891](#1891)) ([ec950f7](ec950f7)) ### Bug Fixes * **build:** enable Intel macOS pip installs via ort-load-dynamic ([#1538](#1538)) ([32ce99e](32ce99e)) * **cache:** avoid fallback session collisions ([#1827](#1827)) ([0f606b6](0f606b6)) * **ccr:** make expired retrieve misses terminal ([#1781](#1781)) ([9cbdba4](9cbdba4)) * **ccr:** preserve Anthropic re-stream shape ([#1854](#1854)) ([f663894](f663894)) * **ccr:** preserve thinking blocks in buffered stream re-synthesis ([#1897](#1897)) ([ede085c](ede085c)) * **cli/proxy:** preserve explicit HEADROOM_MIN_TOKENS=0 / MAX_ITEMS=0 ([#1886](#1886)) ([3a33af1](3a33af1)) * **code-compressor:** CJK-aware relevance-query symbol matching ([#1747](#1747)) ([b38315c](b38315c)) * **codex:** discover updated Codex state stores ([#1889](#1889)) ([9d42eba](9d42eba)) * **codex:** OpenCode Zen telemetry attribution ([#1648](#1648)) ([f18c6bd](f18c6bd)) * **content-detector:** detect and compress space-separated JSON objects ([#1742](#1742)) ([5194bdc](5194bdc)) * **content-router:** token-measure lossless folds at the acceptance gate ([#1772](#1772)) ([c5493ea](c5493ea)) * **copilot:** normalize subscription routing host ([#1836](#1836)) ([afd9cbd](afd9cbd)) * **copilot:** route mixed-model requests per model ([#1785](#1785)) ([5af5e22](5af5e22)) * **dashboard:** deduplicate repeated savings metrics ([#1804](#1804)) ([88f935a](88f935a)) * **dashboard:** distinguish unavailable RTK from zero stats in Docker ([#1900](#1900)) ([87f6e93](87f6e93)) * **dashboard:** distinguish unavailable RTK from zero stats in Docker ([#1901](#1901)) ([361adcd](361adcd)) * **dashboard:** price proxy savings without litellm ([#1728](#1728)) ([188e382](188e382)) * detect and clear stale ANTHROPIC_BASE_URL from crashed wrap sessions ([#1768](#1768)) ([#1837](#1837)) ([84509a4](84509a4)) * **docker:** persist headroom workspace in compose ([#1839](#1839)) ([5e29c06](5e29c06)) * **docker:** report source build version ([#1862](#1862)) ([3807488](3807488)) * **evals:** default unparseable judge scores below pass threshold ([#1892](#1892)) ([42ebbc6](42ebbc6)) * **install:** pass sc.exe create as raw command line so binPath= quoting survives ([#1654](#1654)) ([#1702](#1702)) ([d6e0710](d6e0710)) * **install:** persist --no-http2 override through install apply ([#1676](#1676)) ([6fb5f3b](6fb5f3b)) * **mcp:** isolate ClaudeRegistrar CLI config env ([#1888](#1888)) ([1c947b1](1c947b1)) * **mcp:** surface dead proxy state ([#1786](#1786)) ([931eed8](931eed8)) * **memory:** resolve Trae cwd metadata from user reminders ([#1737](#1737)) ([#1887](#1887)) ([3e85eb1](3e85eb1)) * **opencode:** use local MCP config ([#1383](#1383)) ([4bd3ddf](4bd3ddf)) * **proxy/openai:** thread savings-profile kwargs into chat completions ([#1606](#1606)) ([7ff842d](7ff842d)) * **proxy/openai:** translate max_tokens -> max_completion_tokens on chat path ([#1774](#1774)) ([285808b](285808b)) * **proxy:** bound Codex WS compression fallback latency ([#1802](#1802)) ([d24a3f8](d24a3f8)) * **proxy:** bound HF tokenizer load and offload token counting off event loop ([#1738](#1738)) ([46d5d68](46d5d68)) * **proxy:** cancel retry backoff on shutdown ([#1834](#1834)) ([da2d8dc](da2d8dc)) * **proxy:** compress Anthropic user text blocks when enabled ([#1875](#1875)) ([e36439a](e36439a)) * **proxy:** freeze must forward cached (compressed) prefix byte-identical — stop token-mode cache busting ([#1850](#1850)) ([248ae0f](248ae0f)) * **proxy:** fsync savings dir after atomic rename ([#1764](#1764)) ([7de2c1e](7de2c1e)) * **proxy:** keep cache_control bounded + stable so the freeze overlay stops busting ([#1852](#1852)) ([4820134](4820134)) * **proxy:** persist lifetime cache-read savings across restarts ([#1665](#1665)) ([908997e](908997e)) * **proxy:** preserve streaming passthrough beta headers ([#1783](#1783)) ([0f553a8](0f553a8)) * **proxy:** release _active_streams session lock on setup-phase errors ([#1864](#1864)) ([2ccd831](2ccd831)) * **proxy:** retry HTTP/2 stream resets instead of 502ing ([#1645](#1645)) ([2ce19c2](2ce19c2)) * **proxy:** retry passthrough on transient upstream connection close ([#1513](#1513)) ([5d14080](5d14080)) * **proxy:** route Foundry Anthropic messages ([#1878](#1878)) ([739f654](739f654)) * **proxy:** serve /favicon.ico locally instead of tunneling upstream ([#1787](#1787)) ([#1847](#1847)) ([3076e32](3076e32)) * **proxy:** stop rtk stat failures from corrupting session baseline ([#1693](#1693)) ([681b9a8](681b9a8)) * **proxy:** strip 1m model suffix before upstream forwarding ([#1840](#1840)) ([e22d745](e22d745)) * **proxy:** subtract cache write premiums from net savings ([#1800](#1800)) ([53a465b](53a465b)) * **router:** honor MCP aliases in excluded tools ([#1822](#1822)) ([#1863](#1863)) ([140d6e4](140d6e4)) * **rtk:** link managed rtk onto PATH instead of mutating the hook ([#1698](#1698)) ([140cb05](140cb05)) * **streaming:** preserve server_tool_use sse blocks ([#1826](#1826)) ([4ac5493](4ac5493)) * **toin:** publish skip compression recommendations ([#1782](#1782)) ([be51008](be51008)) * **transforms:** normalize diff compressor context ([#1801](#1801)) ([838c523](838c523)) * **transforms:** pass through ragged tables instead of misaligning columns ([#1713](#1713)) ([c7665ca](c7665ca)) * use rtk native Cursor hook instead of injecting .cursorrules ([#756](#756)) ([#1846](#1846)) ([1573f1f](1573f1f)) * **wrap:** replace stale-proxy detection with Vite-style port fallback ([#1406](#1406)) ([b4205c6](b4205c6)) ### Performance Improvements * **proxy:** cap compression workers to CPU count ([#1803](#1803)) ([0a3851b](0a3851b)) * **savings:** batch tracker persistence off the request hot path ([#1817](#1817)) ([451b9f0](451b9f0)) ### Dependencies * bump the cargo-minor-patch group across 1 directory with 7 updates ([#1909](#1909)) ([45601d9](45601d9)) * bump the npm-minor-patch group across 4 directories with 18 updates ([#1907](#1907)) ([8872bbc](8872bbc)) </details> --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
JerrettDavis
pushed a commit
to peterlodri-sec/headroom
that referenced
this pull request
Jul 14, 2026
headroomlabs-ai#1899) ## Description `SmartCrusher.crush_array_json` (`headroom/transforms/smart_crusher.py`) selects rows to keep using statistical signals such as variance, structural anomaly, and position. It has no concept of "this row is audit/compliance-relevant and must stay visible in the prompt." A rare row, such as a leakage flag, compliance marker, or non-standard failure line, can be sampled out like any routine row, or moved behind an opaque `<<ccr:HASH ...>>` retrieval marker the model has no reason to ask for. In audit, SRE, and quant-falsification workloads, rare rows are frequently the most important evidence, so silent disappearance is a real safety issue rather than only a lossy-compression tradeoff. This adds an opt-in `audit_safe` mode to `SmartCrusher`: - `SmartCrusherConfig(audit_safe=True, protected_patterns=[...], fail_closed_on_protected_loss=True)` - Rows are scanned for pattern matches, string or regex, against each row's canonical JSON text before compression runs. - After compression, any protected row missing from the output is spliced back in verbatim, whether it was dropped by the statistical selector or left only behind a CCR marker. - A verification pass re-counts protected-row survivors after splicing. If the count is still short, the crusher fails closed and returns the original, uncompressed content instead of shipping a result with fewer protected matches than the input had. Setting `fail_closed_on_protected_loss=False` ships the best-effort spliced result with a logged warning instead. Protection applies on both `crush_array_json`, the dict-shaped API used by direct callers and the CCR retrieval flow, and `_smart_crush_content`, the tuple-shaped API `apply()` actually calls for every compressed tool/tool_result message. It is live on the real tool-output compression path. Scope: this covers JSON-array-shaped content routed through `SmartCrusher`, the common case for tool outputs such as API results, log lines, and DB rows returned as JSON. Raw CSV/plain-text content compressed by other transforms, including Kompress and log/tabular compressors, is out of scope for this PR; `protected_patterns` only has row structure to match against when the content is or renders to a JSON array. Closes headroomlabs-ai#1705 ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/transforms/smart_crusher.py`: added `audit_safe`, `protected_patterns`, and `fail_closed_on_protected_loss` fields to `SmartCrusherConfig`; these stay Python-side and do not reach the Rust config because this is post-processing around existing Rust-backed compression. - Added `_compile_protected_patterns`, `_canon`, `_row_matches_protected`, `_scan_protected_rows`, and `_splice_missing_protected` as the shared scan/match/splice primitives. - Added `_apply_audit_safe_protection` for dict-shaped `crush_array_json` results and `_apply_audit_safe_protection_to_content` for tuple-shaped `_smart_crush_content` / `apply()` results. Both splice missing protected rows back in, then verify and fail closed or warn on residual loss. - Wired both `crush_array_json` and `_smart_crush_content` to scan for protected rows before compression and apply protection after. - `CHANGELOG.md`: added an `Unreleased / Features` entry. - Default `audit_safe=False`, so existing callers keep current behavior. A regression test compares a configured-but-disabled crusher's output byte-for-byte against an unconfigured one. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_transforms/test_smart_crusher_audit_safe.py`) - [x] Linting passes (`uv run ruff check .`) - [x] Type checking passes (`uv run mypy headroom/transforms/smart_crusher.py`) - [x] New tests added for new functionality when applicable - [x] Manual testing performed ### Test Output ```text $ uv run pytest tests/ -k "(smart_crusher or crush or audit) and not test_optimizer_not_called_in_audit_mode" --no-header -q ... tests\test_transforms\test_smart_crusher_audit_safe.py ........... [ 65%] ... 169 passed, 10 skipped, 8176 deselected, 1 warning in 21.84s $ uv run ruff check . All checks passed! $ uv run mypy headroom/transforms/smart_crusher.py Success: no issues found in 1 source file ``` `-k` excludes `test_optimizer_not_called_in_audit_mode` (`tests/test_cache/test_client_integration.py`), a pre-existing, unrelated Windows temp-path failure in SQLite storage init that reproduces identically on a clean `origin/main` checkout with none of this PR's changes applied; it matched the `-k audit` filter by name coincidence only. ## Real Behavior Proof - Environment: Windows, Python 3.12 via uv-managed venv, `headroom._core` built locally via `maturin` / cargo 1.95.0, no LLM provider needed because this is pure transform-layer behavior. - Exact command / steps: Ran `uv run pytest tests/ -k "(smart_crusher or crush or audit) and not test_optimizer_not_called_in_audit_mode" --no-header -q`, `uv run ruff check .`, and `uv run mypy headroom/transforms/smart_crusher.py`; also exercised the audit-safe tests that build a 62-row JSON array with two `AUDIT_FLAG` rows, run it through `SmartCrusher(SmartCrusherConfig(audit_safe=True, protected_patterns=["AUDIT_FLAG"]), with_compaction=False)` via both `crush_array_json` and `Transform.apply()` over a synthetic tool message, parse the compressed output back to JSON, and drive the splice/verify/fail-closed helper paths with engineered row-drop and forced-mismatch scenarios. - Observed result: Protected rows are present in the compressed output in every tested scenario; the fail-closed branch returns the original content byte-for-byte with `strategy_info == "audit_safe:fail_closed"` when verification detects residual loss; `audit_safe=False` produces output byte-identical to a crusher with no audit-safe configuration. - Not tested: Raw CSV/plain-text tool output compressed via non-SmartCrusher transforms, including Kompress and log/tabular compressors, is out of scope. Top-level `headroom.compress()` / `CompressConfig` wiring for `audit_safe` and `protected_patterns` is a natural follow-up and is not included here. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes No user-facing docs were updated because I did not find an existing `SmartCrusherConfig` field reference doc to extend. The top-level `compress()` / `CompressConfig` wiring mentioned in "Not tested" is a reasonable immediate follow-up if this mechanism is the right shape.
JerrettDavis
pushed a commit
to peterlodri-sec/headroom
that referenced
this pull request
Jul 14, 2026
🤖 I have created a release *beep* *boop* --- <details><summary>0.31.0</summary> ## [0.31.0](headroomlabs-ai/headroom@v0.30.0...v0.31.0) (2026-07-09) ### Features * **cache:** provider-agnostic cache-mode delta + cc-agnostic prefix comparison ([headroomlabs-ai#1868](headroomlabs-ai#1868)) ([7c2f0ea](headroomlabs-ai@7c2f0ea)) * **ccr:** wire retrieve-tool interception into OpenAI Responses handler ([headroomlabs-ai#1898](headroomlabs-ai#1898)) ([62cd307](headroomlabs-ai@62cd307)) * **compression:** add audit-safe mode with protected pattern matching ([headroomlabs-ai#1899](headroomlabs-ai#1899)) ([bb112dd](headroomlabs-ai@bb112dd)) * **content-router:** accept any real compression (remove min-savings floor) ([headroomlabs-ai#1771](headroomlabs-ai#1771)) ([6c31db9](headroomlabs-ai@6c31db9)) * **content-router:** lossless-first dispatch, cross-turn dedup, and A7 lossy-after-fold ([headroomlabs-ai#1818](headroomlabs-ai#1818)) ([60af15f](headroomlabs-ai@60af15f)) * **proxy:** add provider-only HTTP proxy ([headroomlabs-ai#1807](headroomlabs-ai#1807)) ([ebe0a3b](headroomlabs-ai@ebe0a3b)) * **proxy:** add turn-hook extension point for buffered model turns ([headroomlabs-ai#1891](headroomlabs-ai#1891)) ([ec950f7](headroomlabs-ai@ec950f7)) ### Bug Fixes * **build:** enable Intel macOS pip installs via ort-load-dynamic ([headroomlabs-ai#1538](headroomlabs-ai#1538)) ([32ce99e](headroomlabs-ai@32ce99e)) * **cache:** avoid fallback session collisions ([headroomlabs-ai#1827](headroomlabs-ai#1827)) ([0f606b6](headroomlabs-ai@0f606b6)) * **ccr:** make expired retrieve misses terminal ([headroomlabs-ai#1781](headroomlabs-ai#1781)) ([9cbdba4](headroomlabs-ai@9cbdba4)) * **ccr:** preserve Anthropic re-stream shape ([headroomlabs-ai#1854](headroomlabs-ai#1854)) ([f663894](headroomlabs-ai@f663894)) * **ccr:** preserve thinking blocks in buffered stream re-synthesis ([headroomlabs-ai#1897](headroomlabs-ai#1897)) ([ede085c](headroomlabs-ai@ede085c)) * **cli/proxy:** preserve explicit HEADROOM_MIN_TOKENS=0 / MAX_ITEMS=0 ([headroomlabs-ai#1886](headroomlabs-ai#1886)) ([3a33af1](headroomlabs-ai@3a33af1)) * **code-compressor:** CJK-aware relevance-query symbol matching ([headroomlabs-ai#1747](headroomlabs-ai#1747)) ([b38315c](headroomlabs-ai@b38315c)) * **codex:** discover updated Codex state stores ([headroomlabs-ai#1889](headroomlabs-ai#1889)) ([9d42eba](headroomlabs-ai@9d42eba)) * **codex:** OpenCode Zen telemetry attribution ([headroomlabs-ai#1648](headroomlabs-ai#1648)) ([f18c6bd](headroomlabs-ai@f18c6bd)) * **content-detector:** detect and compress space-separated JSON objects ([headroomlabs-ai#1742](headroomlabs-ai#1742)) ([5194bdc](headroomlabs-ai@5194bdc)) * **content-router:** token-measure lossless folds at the acceptance gate ([headroomlabs-ai#1772](headroomlabs-ai#1772)) ([c5493ea](headroomlabs-ai@c5493ea)) * **copilot:** normalize subscription routing host ([headroomlabs-ai#1836](headroomlabs-ai#1836)) ([afd9cbd](headroomlabs-ai@afd9cbd)) * **copilot:** route mixed-model requests per model ([headroomlabs-ai#1785](headroomlabs-ai#1785)) ([5af5e22](headroomlabs-ai@5af5e22)) * **dashboard:** deduplicate repeated savings metrics ([headroomlabs-ai#1804](headroomlabs-ai#1804)) ([88f935a](headroomlabs-ai@88f935a)) * **dashboard:** distinguish unavailable RTK from zero stats in Docker ([headroomlabs-ai#1900](headroomlabs-ai#1900)) ([87f6e93](headroomlabs-ai@87f6e93)) * **dashboard:** distinguish unavailable RTK from zero stats in Docker ([headroomlabs-ai#1901](headroomlabs-ai#1901)) ([361adcd](headroomlabs-ai@361adcd)) * **dashboard:** price proxy savings without litellm ([headroomlabs-ai#1728](headroomlabs-ai#1728)) ([188e382](headroomlabs-ai@188e382)) * detect and clear stale ANTHROPIC_BASE_URL from crashed wrap sessions ([headroomlabs-ai#1768](headroomlabs-ai#1768)) ([headroomlabs-ai#1837](headroomlabs-ai#1837)) ([84509a4](headroomlabs-ai@84509a4)) * **docker:** persist headroom workspace in compose ([headroomlabs-ai#1839](headroomlabs-ai#1839)) ([5e29c06](headroomlabs-ai@5e29c06)) * **docker:** report source build version ([headroomlabs-ai#1862](headroomlabs-ai#1862)) ([3807488](headroomlabs-ai@3807488)) * **evals:** default unparseable judge scores below pass threshold ([headroomlabs-ai#1892](headroomlabs-ai#1892)) ([42ebbc6](headroomlabs-ai@42ebbc6)) * **install:** pass sc.exe create as raw command line so binPath= quoting survives ([headroomlabs-ai#1654](headroomlabs-ai#1654)) ([headroomlabs-ai#1702](headroomlabs-ai#1702)) ([d6e0710](headroomlabs-ai@d6e0710)) * **install:** persist --no-http2 override through install apply ([headroomlabs-ai#1676](headroomlabs-ai#1676)) ([6fb5f3b](headroomlabs-ai@6fb5f3b)) * **mcp:** isolate ClaudeRegistrar CLI config env ([headroomlabs-ai#1888](headroomlabs-ai#1888)) ([1c947b1](headroomlabs-ai@1c947b1)) * **mcp:** surface dead proxy state ([headroomlabs-ai#1786](headroomlabs-ai#1786)) ([931eed8](headroomlabs-ai@931eed8)) * **memory:** resolve Trae cwd metadata from user reminders ([headroomlabs-ai#1737](headroomlabs-ai#1737)) ([headroomlabs-ai#1887](headroomlabs-ai#1887)) ([3e85eb1](headroomlabs-ai@3e85eb1)) * **opencode:** use local MCP config ([headroomlabs-ai#1383](headroomlabs-ai#1383)) ([4bd3ddf](headroomlabs-ai@4bd3ddf)) * **proxy/openai:** thread savings-profile kwargs into chat completions ([headroomlabs-ai#1606](headroomlabs-ai#1606)) ([7ff842d](headroomlabs-ai@7ff842d)) * **proxy/openai:** translate max_tokens -> max_completion_tokens on chat path ([headroomlabs-ai#1774](headroomlabs-ai#1774)) ([285808b](headroomlabs-ai@285808b)) * **proxy:** bound Codex WS compression fallback latency ([headroomlabs-ai#1802](headroomlabs-ai#1802)) ([d24a3f8](headroomlabs-ai@d24a3f8)) * **proxy:** bound HF tokenizer load and offload token counting off event loop ([headroomlabs-ai#1738](headroomlabs-ai#1738)) ([46d5d68](headroomlabs-ai@46d5d68)) * **proxy:** cancel retry backoff on shutdown ([headroomlabs-ai#1834](headroomlabs-ai#1834)) ([da2d8dc](headroomlabs-ai@da2d8dc)) * **proxy:** compress Anthropic user text blocks when enabled ([headroomlabs-ai#1875](headroomlabs-ai#1875)) ([e36439a](headroomlabs-ai@e36439a)) * **proxy:** freeze must forward cached (compressed) prefix byte-identical — stop token-mode cache busting ([headroomlabs-ai#1850](headroomlabs-ai#1850)) ([248ae0f](headroomlabs-ai@248ae0f)) * **proxy:** fsync savings dir after atomic rename ([headroomlabs-ai#1764](headroomlabs-ai#1764)) ([7de2c1e](headroomlabs-ai@7de2c1e)) * **proxy:** keep cache_control bounded + stable so the freeze overlay stops busting ([headroomlabs-ai#1852](headroomlabs-ai#1852)) ([4820134](headroomlabs-ai@4820134)) * **proxy:** persist lifetime cache-read savings across restarts ([headroomlabs-ai#1665](headroomlabs-ai#1665)) ([908997e](headroomlabs-ai@908997e)) * **proxy:** preserve streaming passthrough beta headers ([headroomlabs-ai#1783](headroomlabs-ai#1783)) ([0f553a8](headroomlabs-ai@0f553a8)) * **proxy:** release _active_streams session lock on setup-phase errors ([headroomlabs-ai#1864](headroomlabs-ai#1864)) ([2ccd831](headroomlabs-ai@2ccd831)) * **proxy:** retry HTTP/2 stream resets instead of 502ing ([headroomlabs-ai#1645](headroomlabs-ai#1645)) ([2ce19c2](headroomlabs-ai@2ce19c2)) * **proxy:** retry passthrough on transient upstream connection close ([headroomlabs-ai#1513](headroomlabs-ai#1513)) ([5d14080](headroomlabs-ai@5d14080)) * **proxy:** route Foundry Anthropic messages ([headroomlabs-ai#1878](headroomlabs-ai#1878)) ([739f654](headroomlabs-ai@739f654)) * **proxy:** serve /favicon.ico locally instead of tunneling upstream ([headroomlabs-ai#1787](headroomlabs-ai#1787)) ([headroomlabs-ai#1847](headroomlabs-ai#1847)) ([3076e32](headroomlabs-ai@3076e32)) * **proxy:** stop rtk stat failures from corrupting session baseline ([headroomlabs-ai#1693](headroomlabs-ai#1693)) ([681b9a8](headroomlabs-ai@681b9a8)) * **proxy:** strip 1m model suffix before upstream forwarding ([headroomlabs-ai#1840](headroomlabs-ai#1840)) ([e22d745](headroomlabs-ai@e22d745)) * **proxy:** subtract cache write premiums from net savings ([headroomlabs-ai#1800](headroomlabs-ai#1800)) ([53a465b](headroomlabs-ai@53a465b)) * **router:** honor MCP aliases in excluded tools ([headroomlabs-ai#1822](headroomlabs-ai#1822)) ([headroomlabs-ai#1863](headroomlabs-ai#1863)) ([140d6e4](headroomlabs-ai@140d6e4)) * **rtk:** link managed rtk onto PATH instead of mutating the hook ([headroomlabs-ai#1698](headroomlabs-ai#1698)) ([140cb05](headroomlabs-ai@140cb05)) * **streaming:** preserve server_tool_use sse blocks ([headroomlabs-ai#1826](headroomlabs-ai#1826)) ([4ac5493](headroomlabs-ai@4ac5493)) * **toin:** publish skip compression recommendations ([headroomlabs-ai#1782](headroomlabs-ai#1782)) ([be51008](headroomlabs-ai@be51008)) * **transforms:** normalize diff compressor context ([headroomlabs-ai#1801](headroomlabs-ai#1801)) ([838c523](headroomlabs-ai@838c523)) * **transforms:** pass through ragged tables instead of misaligning columns ([headroomlabs-ai#1713](headroomlabs-ai#1713)) ([c7665ca](headroomlabs-ai@c7665ca)) * use rtk native Cursor hook instead of injecting .cursorrules ([headroomlabs-ai#756](headroomlabs-ai#756)) ([headroomlabs-ai#1846](headroomlabs-ai#1846)) ([1573f1f](headroomlabs-ai@1573f1f)) * **wrap:** replace stale-proxy detection with Vite-style port fallback ([headroomlabs-ai#1406](headroomlabs-ai#1406)) ([b4205c6](headroomlabs-ai@b4205c6)) ### Performance Improvements * **proxy:** cap compression workers to CPU count ([headroomlabs-ai#1803](headroomlabs-ai#1803)) ([0a3851b](headroomlabs-ai@0a3851b)) * **savings:** batch tracker persistence off the request hot path ([headroomlabs-ai#1817](headroomlabs-ai#1817)) ([451b9f0](headroomlabs-ai@451b9f0)) ### Dependencies * bump the cargo-minor-patch group across 1 directory with 7 updates ([headroomlabs-ai#1909](headroomlabs-ai#1909)) ([45601d9](headroomlabs-ai@45601d9)) * bump the npm-minor-patch group across 4 directories with 18 updates ([headroomlabs-ai#1907](headroomlabs-ai#1907)) ([8872bbc](headroomlabs-ai@8872bbc)) </details> --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
SmartCrusher.crush_array_json(headroom/transforms/smart_crusher.py) selects rows to keep using statistical signals such as variance, structural anomaly, and position. It has no concept of "this row is audit/compliance-relevant and must stay visible in the prompt." A rare row, such as a leakage flag, compliance marker, or non-standard failure line, can be sampled out like any routine row, or moved behind an opaque<<ccr:HASH ...>>retrieval marker the model has no reason to ask for. In audit, SRE, and quant-falsification workloads, rare rows are frequently the most important evidence, so silent disappearance is a real safety issue rather than only a lossy-compression tradeoff.This adds an opt-in
audit_safemode toSmartCrusher:SmartCrusherConfig(audit_safe=True, protected_patterns=[...], fail_closed_on_protected_loss=True)fail_closed_on_protected_loss=Falseships the best-effort spliced result with a logged warning instead.Protection applies on both
crush_array_json, the dict-shaped API used by direct callers and the CCR retrieval flow, and_smart_crush_content, the tuple-shaped APIapply()actually calls for every compressed tool/tool_result message. It is live on the real tool-output compression path.Scope: this covers JSON-array-shaped content routed through
SmartCrusher, the common case for tool outputs such as API results, log lines, and DB rows returned as JSON. Raw CSV/plain-text content compressed by other transforms, including Kompress and log/tabular compressors, is out of scope for this PR;protected_patternsonly has row structure to match against when the content is or renders to a JSON array.Closes #1705
Type of Change
Changes Made
headroom/transforms/smart_crusher.py: addedaudit_safe,protected_patterns, andfail_closed_on_protected_lossfields toSmartCrusherConfig; these stay Python-side and do not reach the Rust config because this is post-processing around existing Rust-backed compression._compile_protected_patterns,_canon,_row_matches_protected,_scan_protected_rows, and_splice_missing_protectedas the shared scan/match/splice primitives._apply_audit_safe_protectionfor dict-shapedcrush_array_jsonresults and_apply_audit_safe_protection_to_contentfor tuple-shaped_smart_crush_content/apply()results. Both splice missing protected rows back in, then verify and fail closed or warn on residual loss.crush_array_jsonand_smart_crush_contentto scan for protected rows before compression and apply protection after.CHANGELOG.md: added anUnreleased / Featuresentry.audit_safe=False, so existing callers keep current behavior. A regression test compares a configured-but-disabled crusher's output byte-for-byte against an unconfigured one.Testing
uv run pytest tests/test_transforms/test_smart_crusher_audit_safe.py)uv run ruff check .)uv run mypy headroom/transforms/smart_crusher.py)Test Output
-kexcludestest_optimizer_not_called_in_audit_mode(tests/test_cache/test_client_integration.py), a pre-existing, unrelated Windows temp-path failure in SQLite storage init that reproduces identically on a cleanorigin/maincheckout with none of this PR's changes applied; it matched the-k auditfilter by name coincidence only.Real Behavior Proof
headroom._corebuilt locally viamaturin/ cargo 1.95.0, no LLM provider needed because this is pure transform-layer behavior.uv run pytest tests/ -k "(smart_crusher or crush or audit) and not test_optimizer_not_called_in_audit_mode" --no-header -q,uv run ruff check ., anduv run mypy headroom/transforms/smart_crusher.py; also exercised the audit-safe tests that build a 62-row JSON array with twoAUDIT_FLAGrows, run it throughSmartCrusher(SmartCrusherConfig(audit_safe=True, protected_patterns=["AUDIT_FLAG"]), with_compaction=False)via bothcrush_array_jsonandTransform.apply()over a synthetic tool message, parse the compressed output back to JSON, and drive the splice/verify/fail-closed helper paths with engineered row-drop and forced-mismatch scenarios.strategy_info == "audit_safe:fail_closed"when verification detects residual loss;audit_safe=Falseproduces output byte-identical to a crusher with no audit-safe configuration.headroom.compress()/CompressConfigwiring foraudit_safeandprotected_patternsis a natural follow-up and is not included here.Review Readiness
Checklist
Additional Notes
No user-facing docs were updated because I did not find an existing
SmartCrusherConfigfield reference doc to extend. The top-levelcompress()/CompressConfigwiring mentioned in "Not tested" is a reasonable immediate follow-up if this mechanism is the right shape.