Repository navigation
feat(types): privacy pass over a manifest before registry promotion - #7819
Conversation
An `AgentManifest` written on one operator's install is not a publishable artefact. It carries an absolute workspace path under their home directory, the environment variable holding their provider credentials, a private or self-hosted base URL, a command and environment allowlist that encodes their local security policy, an arbitrary metadata bag, and whatever free text they pasted into a system prompt or a context injection. The registry validator requires only `name`, `description` and `module`, so none of it is caught on the way in, and once published it is in git history. `librefang_types::manifest_privacy` adds the two halves separately, per #7771: - `sanitize_for_publication` returns a publishable copy with the instance-specific fields removed or reset and the portable half intact. - `scan_for_publication` reports what it found with a bounded preview per finding, including inside the fields the sanitiser keeps — an operator can paste anything into a system prompt, and no structural rule separates portable configuration from an internal hostname in free text. Keeping them apart is the point: the operator confirms, rather than having their file quietly rewritten on their behalf. The classification is exhaustive by construction rather than a field list that rots. The sanitiser destructures and rebuilds `AgentManifest` with no rest pattern and no struct-update syntax, so a new field fails to compile here until someone names it on both sides; the same holds for the three types whose interiors are reduced. `CLASSIFICATION` is the readable half of the same decision, asserted against the serialized struct in both directions, and the detector walks the serialized publishable manifest so a new portable field is scanned from the moment it exists. `GET /api/templates/{name}` now carries the read-only result as `promotion_preview`. The endpoint that would actually publish an agent type does not exist on `main` and awaits #6931.
clippy::explicit_counter_loop was failing -D warnings.
CLAUDE.md's prose-wrapping rule requires breaking only at sentence boundaries, not at a column limit.
… line CLAUDE.md's prose-wrapping rule requires breaking only at sentence boundaries, not at a column limit; the new module's doc comments and its integration tests were hard-wrapped at ~65-80 columns.
houko
left a comment
There was a problem hiding this comment.
Automated review against CLAUDE.md. Pushed 3 mechanical fixes to this branch:
- fix(types): a clippy
explicit_counter_loopfailure intruncate_preview(was breakingcargo clippy -- -D warnings) - docs(api) / docs(types): rewrapped hard-wrapped doc comments in the new
manifest_privacymodule and its API wiring to one sentence per line, per the repo's prose-wrapping convention
All 34 manifest_privacy unit tests and all 13 profiles_templates_routes_integration tests still pass after the fixes, and cargo clippy -p librefang-types --lib / -p librefang-api --lib are clean.
Left two inline comments on things I didn't fix unilaterally — a possible IPv6-literal detection gap in looks_like_private_endpoint/split_host, and a minor double-sanitization in promotion_preview — both are judgment calls on scope rather than safe one-line patches.
Generated by Claude Code
|
|
||
| /// Split the host out of a URL-ish token, reporting whether a scheme was present. | ||
| /// Returns `None` for tokens that cannot carry a host at all. | ||
| fn split_host(token: &str) -> Option<(&str, bool)> { |
There was a problem hiding this comment.
split_host's port-stripping can misparse a bare (unbracketed or non-schemed) IPv6 literal, letting a loopback/private address slip past looks_like_private_endpoint:
- For input
"::1"(no scheme, no brackets — plausible as a raw manifest string value),authority.rfind(':')finds the second colon and strips"1"as a bogus port, leavinghost = ":". The later comparison against"::1"then fails to match, so the loopback literal isn't flagged. - For
"[::1]:8080"(bracketed host with an explicit port), theauthority.starts_with('[')branch returns the whole"[::1]:8080"unstripped, so aftertrim_start_matches('[').trim_end_matches(']')you get"::1]:8080", which also fails the"::1"equality check. It happens to still get caught today only via the "single-label host with scheme" fallback (since it still lacks a.), but a bare"::1"or"[::1]:8080"with no scheme goes undetected.
Since this module exists specifically to catch host-specific network literals before publication, a pasted IPv6 loopback/private address in a retained field (e.g. a system prompt or tag) would go unreported. Worth a look — scope/fix (special-casing bracketed vs. unbracketed IPv6 host+port splitting, plus test coverage) is a judgment call I'd leave to you rather than patch blind.
Generated by Claude Code
| let findings = scan_for_publication(manifest); | ||
| let requires_review = findings.iter().any(|finding| !finding.removed_by_sanitizer); | ||
|
|
||
| let publishable = sanitize_for_publication(manifest); |
There was a problem hiding this comment.
Minor: sanitize_for_publication runs twice per request here — once inside scan_for_publication (via collect_retained_findings, line 255) to build the JSON walk, and again directly on line 258 for manifest_toml. Harmless given manifest size and this endpoint's traffic, but if you want to dedupe it, that's an API change to manifest_privacy (e.g. a combined sanitize_and_scan helper, or having scan_for_publication accept an already-sanitised manifest) rather than a local one-liner — flagging for your call rather than changing the module's public surface unilaterally.
Generated by Claude Code
…vacy' into feat/7771-registry-promotion-privacy
Deploying librefang-docs with
|
| Latest commit: |
4cc92f7
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://786c2449.librefang-docs.pages.dev |
| Branch Preview URL: | https://feat-7771-registry-promotion.librefang-docs.pages.dev |
…7819, librefang#7817, librefang#7820) Brings in four upstream commits and reconciles them with the fork's own work in the same files. Every conflict keeps both sides' intent; nothing from either side was dropped. Conflicts resolved: - crates/librefang-api/dashboard/src/components/ui/SliderInput.tsx — both sides fixed the same dimming bug from opposite directions. Upstream's bounds normalization (`lowerBound` / `upperBound` / `clamp` / `boundedValue` / `emitValue`) now feeds the fork's index-addressed `steps` mode, so the stepped context-window slider inherits inverted-bound and non-finite-value handling it never had. The fork's position-accurate tick legend replaces upstream's `flex justify-between`, which spaces labels evenly regardless of value; upstream's `${tick}-${index}` key is adopted so a repeated tick value still renders both labels. The fork's toggle geometry (w-9 track, p-0.5 inset, w-4 knob) is kept over upstream's w-8/h-[18px] because it contains the knob by construction. - crates/librefang-api/dashboard/src/components/ui/SliderInput.test.tsx — added by both. Union of the two suites: the fork's toggle-geometry, tick-position and fixed-steps cases, plus upstream's inverted-bounds, clamping, non-finite-input and inherited-state cases. 25 tests pass. - crates/librefang-api/src/routes/agent_templates.rs — upstream added `promotion_preview` to `get_agent_template`; the fork had renamed that handler to `get_agent_type` behind the canonical `/api/agent-types` routes with `/templates` kept as a deprecated alias. `promotion_preview` is wired into the canonical handler, so both path spellings serve it. The response keeps the fork's full serialized manifest rather than upstream's hand-picked subset — a superset of what upstream's tests read. - crates/librefang-kernel/src/kernel/mod.rs — both re-exports kept (`PendingSkillMcpDeclarations`, `SemanticMemoryAccess`, `SkillReloadOutcome`). - crates/librefang-kernel/src/kernel/tools_and_skills.rs — the fork's unconditional `workflow_create` injection and upstream's semantic-memory capability gate are independent blocks; both run, injection first. - openapi.json — regenerated from source rather than hand-merged. - xtask/baselines/openapi.sha256 — regenerated from the regenerated spec. Integration fixes the merge itself required: - crates/librefang-types/src/manifest_privacy.rs — upstream's `reduce_model` exhaustively destructures `ModelConfig`, which the fork has extended with `mode` and `router_override`. Both survive promotion: they select how a type picks a model, which is what the type does, and neither carries a host path, credential binding or private endpoint. Stripping them would republish a flexible-routing type as a fixed one — a behaviour change, not a redaction. - crates/librefang-api/src/custom_gateway_catalog.rs — upstream's new `ModelCatalogEntry::limits_known` reached the fork's gateway discovery path through `..Default::default()`, which is `true`. That path legitimately produces `context_window: 0` when a gateway publishes no capacity, and `limits_known: true` there records the gateway's silence as a discovered ceiling. The flag is now derived alongside the numbers, the way `pricing_known` already was: true only when `/model/info` reported a value or a prior entry carried a real one forward. Two tests pin both directions. Verification: - cargo check -p librefang-types -p librefang-memory -p librefang-kernel -p librefang-api --lib — clean. - cargo check -p librefang-cli --bins — clean. - cargo test -p librefang-types --lib manifest_privacy — 34 passed. - cargo test -p librefang-kernel --lib tools_and_skills — 13 passed. - cargo test -p librefang-kernel --lib semantic_memory — 6 passed. - cargo test -p librefang-api --test profiles_templates_routes_integration — 15 passed, 2 failed. Both failures are pre-existing on this branch and unrelated to the merge: `templates_list_carries_provider_and_model_from_the_manifest` and `templates_list_skips_a_malformed_manifest_instead_of_failing` seed `workspaces/agents/`, which this branch's `list_agent_types` deliberately stopped listing. Fixed in the following commit. - dashboard: vitest SliderInput 25 passed, tsc --noEmit clean. Locale keys: upstream touched no `.ftl` file in these four commits; key counts are unchanged.
Closes #7771.
What this is, and what it deliberately is not
The issue asks for the privacy pass that has to run before an operator-authored agent type can be contributed to
librefang/librefang-registry.I checked what the promotion path actually is on
origin/mainfirst, because the issue's premise matters:crates/librefang-api/src/routes/agent_templates.rsregisters onlyGET /templates,GET /templates/{name}andGET /templates/{name}/toml; there is no promote/publish path for an agent type anywhere incrates/— not in the API, not inlibrefang-cli(AgentCommandshasnew/spawn/list/chat/kill/delete/reset-uuid/merge-history/set, no export or publish), and not inlibrefang-extensions. The only registry-contribution machinery that exists islibrefang-skills/src/registry_pr.rs, and it is skill-shaped, not manifest-shaped.So the endpoint half awaits PR #6931, which adds local
POST/PUT/DELETEfor agent types. This PR is the primitives plus their tests, wired at the one call site that exists today rather than landed as dead code.Substance
New module
crates/librefang-types/src/manifest_privacy.rs, with the two operations kept separate exactly as the issue asks:sanitize_for_publication(&AgentManifest) -> AgentManifest— a publishable copy. Borrows the input and never writes a file.scan_for_publication(&AgentManifest) -> Vec<Finding>— what it found, with a bounded single-line preview per finding capped at 96 characters.The detector is not a diff against the sanitiser. Each
Findingcarriesremoved_by_sanitizer:truemeans confirming publication is enough,falsemeans the value sits inside a field promotion has to keep and the operator must edit it by hand. That second case is the one that matters — nothing structural distinguishes portable configuration from an internal hostname sitting in a system prompt.Classification
CLASSIFICATIONin the module is the readable table; the doc comment carries the per-field reasoning. Summary:workspace,workspaces(bothpathandmountdescribe host layout, and the aliases are often the operator's own vocabulary).model.api_key_env,model.base_url, and the same pair on everyfallback_modelsentry.exec_policy,capabilities.shell,capabilities.network,capabilities.ofp_connect,tool_exec_backend,context_engine(plugin names, local Python hook paths, a sidecar endpoint).context_injection,tools(per-toolparamsis an untyped operator bag),metadata,model.extra_paramsand itsfallback_modelscounterpart,channel_overrides,author.triggers(target_agentandworkflow_idname records on this install),is_hand,enabled.name,version,description,module,schedule,session_mode,model.provider/model.model/system_prompt/ sampling knobs,skills,capabilities.toolsand the memory / spawn / message grants,mcp_servers,channels,tags,routing,resources,priority,thinking,compaction,profile, tool allow/blocklists,allowed_plugins,response_format, and the remaining behavioural knobs.Two of those deserve a note because the issue's framing and the code disagree slightly, and the code is right:
mcp_serversandchannelsare names, not bindings — MCP endpoints and headers, and channel tokens and chat ids, live inKernelConfig, never in a manifest. Stripping them would remove the reproducibility the type exists for while protecting nothing, so they travel. Converselyauthoris stripped even though the issue does not list it: an operator'sauthoris routinely a local username or an e-mail address, and attribution for a contribution belongs to the pull request carrying it.Why the classification cannot silently rot
Three mechanisms, and the first is the one the issue asks for:
sanitize_for_publicationdestructuresAgentManifestwith no..rest pattern and rebuilds it with no..Default::default(). Adding a field toAgentManifestis therefore anE0027("pattern does not mention field") on the destructure and anE0063("missing field in initializer") on the construction, in this module, until someone names it on both sides and so decides what happens to it. The same holds forModelConfig,FallbackModelandManifestCapabilities, the three types whose interiors are reduced rather than kept or dropped whole. This is the mechanism the issue's "prefer an exhaustive match" asks for — a struct destructure without..gives the identical guarantee for a struct that a match without a wildcard arm gives for an enum.classification_covers_every_serialized_manifest_fieldassertsCLASSIFICATIONand the serialized struct agree in both directions, so the table cannot fall behind the struct and a rename cannot leave a stale entry. The fixture populates every field carryingskip_serializing_ifso all 58 keys are emitted.Value-level scanners
The credential and PII shapes reuse
librefang_types::taint's existing denylist throughTaintSink::net_fetch()(the sink that blocksSecretandPii) rather than growing a second copy. Two adjustments were needed:OpaqueTokenis skipped for values with no unbroken alphanumeric run of 20 or more characters. Without that gate it fires on every long model id —accounts/fireworks/models/llama-v3p1-405b-instructandclaude-sonnet-4-20250514both top out at eight — and a detector that cries wolf onmodel.modelteaches the operator to click through the findings that matter. There is a test pinning that specifically.Absolute-host-path and private-endpoint detection are new here. Both are documented as best-effort with their gaps stated: the POSIX path rule requires a known filesystem root, which means
/api/v1/messagesin a prompt is not reported and neither is a path under a non-standard root like/nvme0/agents/; the private-endpoint rule covers loopback, RFC 1918 and link-local literals (four octets required, so10.0.0is not read as an address), the.local/.internal/.lan/.corpfamily, and a single-label host behind an explicit scheme.Wiring
GET /api/templates/{name}gains an additive, read-onlypromotion_previewobject:findings,requires_review(any finding the sanitiser cannot handle on its own), andmanifest_toml— the scrubbed manifest, so an operator can attach it to a registry pull request by hand today.manifest_tomlisnullif the publishable copy cannot be rendered as TOML, with aWARN; that is not a reason to withhold the findings, which are the part that protects the operator.Nothing existing changes: the raw
manifest_tomlat the top level is still the operator's file verbatim, and there is a test asserting that. The dashboard consumes only/templates/{name}/toml, so nothing in the SPA is affected and no locale strings are added.Verification
cargo test -p librefang-types --lib manifest_privacy— 34 passed. Includes one test per stripped category asserting the value does not survive;no_planted_sentinel_survives_promotion, which greps the serialized publishable manifest for all 19 planted sentinels at once;portable_fields_survive_promotionandportable_collections_survive_promotionfor the benign half;classification_covers_every_serialized_manifest_field;sanitizing_does_not_mutate_the_caller_s_manifest;publishable_manifest_round_trips_through_toml;detector_does_not_mistake_a_model_id_for_a_credential;detector_output_is_stable_across_runs;previews_are_bounded_and_single_line; and shape-rule tables with both positive and negative cases.cargo test -p librefang-api --test profiles_templates_routes_integration— 13 passed, 4 of them new#[tokio::test]s against the real router: the findings shape, the scrubbed-TOML round trip (host specifics absent, portable half present, raw file unmodified), a secret inside a retained system prompt settingrequires_review, and silence on an already-portable template.cargo check -p librefang-api --lib— clean.Not run:
cargo clippy. Free space on the build host dropped from 45 GiB to 2.4 GiB during these runs (the shared target directory is contended by other concurrent sessions), so I stopped building rather than risk the machine and am relying on CI for the clippy gate.One environment note for anyone reproducing locally:
librefang-apicannot compile from a fresh worktree withoutcrates/librefang-api/static/react/existing, becauseinclude_dir!panics on the missing directory. I created a gitignored placeholder; it is not part of this diff.Deferred
Only the half that cannot exist yet: there is no promotion endpoint or CLI command to gate, so nothing calls
sanitize_for_publicationon the way to a registry. Once #6931 lands itsPOST/PUT/DELETE, the promote path isscan_for_publication→ show the operator the findings → confirm →sanitize_for_publication→ theregistry_pr.rs-shaped GitHub flow. I have not started that here because it would be guessing at #6931's shape.