Telemetry insights — 2026-09-24

Executive summary

1. Privacy gap: local file paths with OS account names reach telemetry and the aggregate CSVs. llm.model is sent without sanitization. The extract's redaction only knows emails, UUIDs and ak_ hashes. In this window, 46 provider_stack_daily.csv rows (20 distinct values, 67,010 completed runs) have an llm_model containing a home-directory segment (/home/…, /Users/…, /root/…). That includes the current release, 1.6.0. Open PR SDK-775 feat: Make pipeline telemetry diagnosable #5159 adds embedding.model through the same regexes, so it would widen the gap.
2. Two forks report under upstream version labels, and earlier reports read them as upstream releases. "1.0.3" and "1.1.0" emit SearchType values that have never existed in cognee: 31.3% and 26.3% of their search events. In W1 these two cohorts are 9.94% of fleet pipeline starts, 13.14% of fleet errors, and 40.9% of the +30% WoW error increase. This changes how Telemetry insights: 2026-09-19 — fleet-wide pipeline error step change around Sept 10 #5140 F5, Telemetry insights: 2026-09-22 — demo Postgres graph backend carries 19% of fleet completed runs #5167 F3 and part of Telemetry insights: 2026-09-23 — source/Docker builds now 67% of fleet pipeline errors #5182 should be read.
3. Telemetry insights: 2026-09-23 — source/Docker builds now 67% of fleet pipeline errors #5182 persists on the first complete day since it was filed. On 09-23, -local builds failed at 29.04% and the rest at 11.73%; 1.5.4-local failed at 47.87% vs 12.33% for other 1.5.4.
4. Telemetry insights: 2026-09-20 — v1.6.0 GA day: 67% of pipeline runs emit no terminal event #5147's 1.6.0 silent gap dropped back to 7.30% on 09-23 (15.93% on 09-22). It still swings with deployment mix, as Telemetry insights: 2026-09-23 — source/Docker builds now 67% of fleet pipeline errors #5182 described.
5. The other trend breaks (POST /v1/add −28.1% WoW, neo4j completions −38.4% WoW, azure_openai +381%) trace back to one to four deployments each. They are listed under Watching, not filed.

Data window and method

• Window: telemetry_aggregates/*.csv, 2026-07-16 → 2026-09-24 (WINDOW.txt: window_days=70). 09-24 is partial: 41,126 fleet pipeline starts vs 362,045 on 09-23. It is left out of every rate. 09-23 is the only day that became complete since Telemetry insights: 2026-09-23 — source/Docker builds now 67% of fleet pipeline errors #5182. Weeks: W1 = 09-17…09-23, W2 = 09-10…09-16, W3 = 09-03…09-09.
• Tooling: earlier reports had to hand-sum. This run computed every aggregate with jq over the CSVs (python, awk and bc are still refused). The fleet 09-22 totals reproduce Telemetry insights: 2026-09-23 — source/Docker builds now 67% of fleet pipeline errors #5182's hand sums exactly (357,673 started / 225,761 completed / 81,035 errored). 1.6.0's 09-23 cohort split adds back to its pipeline_outcomes_daily.csv total (30,272 + 50,414 = 80,686).
• Fleet pipeline, last complete days (pipeline_outcomes_daily.csv):

• Surface split: origin still takes only sdk (3,837 rows) and unknown (32,081 rows). There is no cli or cloud (already reported in Telemetry insights: 2026-09-19 — fleet-wide pipeline error step change around Sept 10 #5140 F3c). Analysis 1 therefore uses event families.

Finding 1 — llm.model sends local filesystem paths, including OS account names, and the extract's redaction misses them (severity: high, privacy)

Computed

provider_stack_daily.csv, whole window. Counts only: no matching value is reproduced here, on purpose.

• Versions affected (home-directory slice): 1.2.2, 1.4.0, 1.4.1, 1.4.2, 1.5.3 and 1.6.0. Two of the 46 rows fall in the last seven days.
• llm_provider on those rows: openai 25, custom 18, llama_cpp 3. These are local GGUF or model-directory setups served through OpenAI-compatible or llama.cpp endpoints. That is exactly what the keyless and local-model path in 1.6.0 encourages.
• Each row covers at most 2 identities. So this is a handful of deployments, each with its account name written into a CSV that gets posted to the analysis runner. The row inspected while finding this had an OS account name right after /home/.

Cause (code reading, not a hypothesis)

• Client: run_tasks_with_telemetry (cognee/modules/pipelines/operations/run_tasks_with_telemetry.py:19-33) merges get_current_settings() into every Pipeline Run Started/Completed/Errored event. That dict carries llm.model = llm_config.llm_model verbatim (cognee/modules/settings/get_current_settings.py:43-46). send_telemetry only hashes keys named in TELEMETRY_SANITIZED_PROPERTIES = ["url", "session_id", "session_ids", "datasets"] (cognee/shared/utils.py:134, applied at :406). The DB url fields are hashed; model is not. So the raw path, untruncated, is stored in the warehouse.
• Extract: CELL_PATTERNS (.github/scripts/telemetry_aggregate_extract.py:38-42) are email, UUID and ak_ hash. The same patterns drive the SQL bucketing (_provider_dimension, :46-56, applied at :151-155) and the post-write privacy guard (:196-210). A path matches none of them, so it goes through both the redaction and the guard that is supposed to fail the job.
• Open PR SDK-775 feat: Make pipeline telemetry diagnosable #5159 extends the gap. It adds "embedding": {"provider", "model"} to get_current_settings() and an embedding_model column to the extract, redacted "like the LLM ones". Local embedding models are just as often given as filesystem paths.

Proposed fix

1. Client (the root cause). Sanitize model identifiers in cognee/shared/utils.py before sending. If llm.model (and embedding.model once SDK-775 feat: Make pipeline telemetry diagnosable #5159 lands) looks like a filesystem path (leading / or ~, a drive letter, a backslash, or a provider//abs/path form), replace it with its basename or a fixed local-path token. The provider prefix is still useful. Add a unit test next to cognee/tests/unit/shared/test_telemetry_dataset_names.py.
2. Extract (defence in depth). Add a path pattern to CELL_PATTERNS, for example (^|[/\\])(home|users|root)[/\\] plus ^[a-z]:[/\\]. That lets the SQL redaction and the post-write guard both catch paths. Extend test_redacts_identifiers_in_provider_dimensions in cognee/tests/unit/test_telemetry_aggregate_extract.py. Apply it to SDK-775 feat: Make pipeline telemetry diagnosable #5159's embedding_model column before that PR merges.
3. Warehouse. Consider scrubbing already-stored properties.llm.model values that match the pattern. The extract fix only hides them from the CSVs.
4. Confirming observable. In the next extract after (2), rows whose llm_model matches the home-directory pattern should be 0 (from 46), and the redacted bucket should grow by the matching runs. After (1) ships, new-version rows should show basenames or local-path instead of redacted.

Duplicate check

• gh issue list --label telemetry-insights --state all → Telemetry insights: 2026-09-19 — fleet-wide pipeline error step change around Sept 10 #5140, Telemetry insights: 2026-09-20 — v1.6.0 GA day: 67% of pipeline runs emit no terminal event #5147, Telemetry insights: 2026-09-22 — demo Postgres graph backend carries 19% of fleet completed runs #5167, Telemetry insights: 2026-09-23 — source/Docker builds now 67% of fleet pipeline errors #5182. None mentions paths, usernames or model-string privacy. New.
• gh pr list --state all --search "telemetry model path redact" → nothing relevant. "telemetry sanitize llm model" → fix: Sanitize identifiers in telemetry aggregates (COG-6288) #4941 (CLOSED, unmerged, "Sanitize identifiers in telemetry aggregates") and SDK-775 feat: Make pipeline telemetry diagnosable #5159 (OPEN, adds embedding.model through the same patterns, no path handling). The merged fix(ci): redact identifiers in telemetry provider aggregates #5034 (b66f515df) introduced the current email/UUID/ak_ redaction and covers no paths. The dataset-name fingerprinting commits of 09-17 (0109e1c96, 24ee40f26) do not touch model. Not addressed by a merged PR; the leak persists on 1.6.0, released after all of them.

Finding 2 — "1.0.3" and "1.1.0" are forks, and they distort fleet error and search metrics (severity: medium, analysis integrity)

Computed

Search types never in upstream. search_type_daily.csv, whole window, checked against cognee/modules/search/types/SearchType.py. git log --all -S/-G finds none of these names in the file's history:

(Also DISPUTES 114 on 1.5.4 on 09-17 only, which is negligible. The CamelCase values on 0.1.3/0.2.0 are real legacy upstream names.)

Weight in W1 fleet pipeline metrics (daily_event_volumes.csv, split by the self_hosted flag, which means -local):

• They drive much of the error WoW. Pipeline Run Errored rose +30.0% WoW (386,345 → 502,424, +116,079), which clears the analysis-1 threshold. Per version (pipeline_outcomes_daily.csv): 1.0.3 +29,919, 1.1.0 +17,514. Together that is 40.9% of the increase. Most of the rest is already tracked: 1.5.2 +39,015 (Telemetry insights: 2026-09-22 — demo Postgres graph backend carries 19% of fleet completed runs #5167 F2), 1.5.4 +26,180 and 1.4.1 +17,810 (Telemetry insights: 2026-09-23 — source/Docker builds now 67% of fleet pipeline errors #5182), and 1.6.0 +25,760 (new in W1).
• 1.0.3's search counters stopped matching up on 09-18. sdk_exec_outcomes_daily.csv 1.0.3 search: completed/started was ≤1.0 every day through 09-17. From 09-18 → 09-23 it was 1.13, 1.74, 1.83, 1.34, 1.74, 1.88 (for example 09-23: 5,025 started / 9,424 completed). W1 overall: 77,709 / 101,416 = 130.5%. That is nearly all of the fleet's cognee.search EXECUTION COMPLETED +23.2% WoW. Upstream 1.0.3 code cannot have changed on 09-18, so this is the fork's own instrumentation.

Hypothesis

Two downstream forks keep upstream's pyproject.toml version string and add their own retrievers. Upstream 1.0.3 was superseded on 2026-05-02 (798d84280, bump to 1.0.4). The "1.0.3" fork is a git/Docker build (-local). The "1.1.0" fork is installed as a package (non--local). Earlier reports treated both labels as upstream releases:

• Telemetry insights: 2026-09-19 — fleet-wide pipeline error step change around Sept 10 #5140 F5 ("1.1.0 surged +344%… a downstream product that vendored cognee 1.1.0") and Telemetry insights: 2026-09-22 — demo Postgres graph backend carries 19% of fleet completed runs #5167 F3 ("1.1.0's errors are a constant") recommended upgrade and support outreach on the premise that this is upstream 1.1.0. The fork's own search types show at least part of it is not. Upgrading upstream would not reach it.
• Telemetry insights: 2026-09-19 — fleet-wide pipeline error step change around Sept 10 #5140/Telemetry insights: 2026-09-20 — v1.6.0 GA day: 67% of pipeline runs emit no terminal event #5147 called HYBRID/HYBRID_RERANK "legacy". They were never upstream.
• Telemetry insights: 2026-09-23 — source/Docker builds now 67% of fleet pipeline errors #5182's -local cohort includes the 1.0.3 fork (23.02% error rate). That does not overturn Telemetry insights: 2026-09-23 — source/Docker builds now 67% of fleet pipeline errors #5182: the effect there was driven by 1.5.4/1.5.2/1.6.0. But fleet medians and cohort rates in these reports include fork traffic.

The aggregates cannot say whether every deployment under these labels is the fork (1.0.3-local has 19–32 identities a day). They only show that fork traffic is a large share of it.

Proposed fix

1. Extract (.github/scripts/telemetry_aggregate_extract.py). Add a per-(version, self_hosted) conformance flag: whether any search_type outside the upstream SearchType enum was seen in the window. Put it in version_lifecycle.csv so the analysis can exclude fork cohorts from fleet medians and WoW decompositions. This needs only the already-exported search_type dimension; no new data is collected.
2. Client. Telemetry insights: 2026-09-23 — source/Docker builds now 67% of fleet pipeline errors #5182 proposed stamping a build identifier next to SDK-775 feat: Make pipeline telemetry diagnosable #5159's install_kind. Include a stable fingerprint of the SearchType member set (or the distribution's direct_url.json VCS URL hash for git installs). That separates upstream from forks even when a fork never calls a custom search type.
3. Confirming observable. With the flag in place, the W1 fleet error rate excluding the two cohorts should be (502,424 − 66,022) / (2,375,848 − 236,046) = 20.39%, against 21.15% including them. The 1.1.0 constant-error pattern (Telemetry insights: 2026-09-22 — demo Postgres graph backend carries 19% of fleet completed runs #5167 F3) should sit entirely inside the flagged cohort.

Duplicate check

• telemetry-insights issues: Telemetry insights: 2026-09-19 — fleet-wide pipeline error step change around Sept 10 #5140 F3b reports counter-integrity violations (search completed > started) as an upstream instrumentation scope bug. It does not name forks as the source. Telemetry insights: 2026-09-19 — fleet-wide pipeline error step change around Sept 10 #5140 F5 and Telemetry insights: 2026-09-22 — demo Postgres graph backend carries 19% of fleet completed runs #5167 F3 analyse 1.1.0 as an upstream release. No issue identifies either label as a fork. New, and it corrects prior findings.
• gh pr list --state all --search "fork version telemetry search type" → nothing relevant. Not addressed.

Verification of prior findings (not re-filed)

• Telemetry insights: 2026-09-23 — source/Docker builds now 67% of fleet pipeline errors #5182 (-local builds carry most errors): persists. On 09-23, -local was 48,921 / 168,464 = 29.04% and the rest 22,604 / 192,784 = 11.73%. 1.5.4-local was 17,312 / 36,167 = 47.87% (56.86% on 09-22) and other 1.5.4 was 5,259 / 42,635 = 12.33%. 1.6.0-local was 7,946 / 50,414 = 15.76% and non--local 1.6.0 was 1,271 / 30,272 = 4.20%. SDK-775 feat: Make pipeline telemetry diagnosable #5159 is still open.
• Telemetry insights: 2026-09-20 — v1.6.0 GA day: 67% of pipeline runs emit no terminal event #5147 (1.6.0 silent gap): 09-23 gap = 5,887 / 80,686 = 7.30%, down from 15.93% on 09-22 and up from 3.06% on 09-21. It still moves with deployment mix, not steadily. Keep open until SDK-775 feat: Make pipeline telemetry diagnosable #5159 merges.
• Telemetry insights: 2026-09-22 — demo Postgres graph backend carries 19% of fleet completed runs #5167 F2 (single-deployment runaways): persists. On 09-23, 1.5.2 was 14,276 / 16,341 = 87.36%, errors from 1 deployment. 1.0.9 was 2,550 / 11,186 = 22.80% with a 32.56% silent gap. 0.5.4 fits the same class: 66.57% in W1 (10,031 / 15,069) and 67–71% on each of 09-21…23, from 2 deployments reporting errors.
• Telemetry insights: 2026-09-22 — demo Postgres graph backend carries 19% of fleet completed runs #5167 F3 (1.1.0 constant error count): holds at 5,220 on 09-23 (09-21 5,054, 09-22 5,310). See Finding 2 for the reinterpretation.
• Telemetry insights: 2026-09-19 — fleet-wide pipeline error step change around Sept 10 #5140 F1 (fleet error step): persists at 19.86% on 09-23.

Watching (near-threshold, single-deployment, or one day)

• 1.6.0 non--local error rate is creeping up: 0.79% → 2.59% → 4.20% (09-21 → 09-23), with 25 → 32 → 27 deployments reporting errors. Still far below the fleet. Re-check next run.
• 1.5.0: errors 326 → 443 → 1,814 (09-21 → 23). 25.13% on 09-23, one day.
• 1.0.5 silent gap 39.94% in W1 (30,681 started). Persistent since Telemetry insights: 2026-09-20 — v1.6.0 GA day: 67% of pipeline runs emit no terminal event #5147's watch note (40.8% on 09-19). It is a pre-CLO-365 fix: mark cancelled pipeline runs as errored, not stuck #4680 build, so the expected cause is runs that were cancelled but never marked errored. Not a regression candidate.
• SDK cognify: W1 had 3,401 started / 3 completed / 3,216 errored (94.6%), from 1–3 deployments, version=unknown. The Telemetry insights: 2026-09-19 — fleet-wide pipeline error step change around Sept 10 #5140 F3b class; still under 1,000 events a day.
• 1.6.0 search mix is CHUNKS-heavy: in W1, CHUNKS was 4,342 of 6,107 events (71.1%) and HYBRID_COMPLETION (the recall() default with an LLM key) was 407 (6.7%). This fits keyless setups, where recall() falls back to CHUNKS, or explicit CHUNKS calls; the aggregates cannot tell which. Note that 1.6.0 SDK search starts (29,422 in W1) far exceed its search_type rows. Worth keeping an eye on as keyless adoption grows.
• Explained trend breaks (no product signal): POST /v1/add −28.1% WoW (347,895 → 250,280). 1.4.2 −73,402, where the daily non--local add volume came from 1–3 deployments and stopped after 09-14, and 1.0.1 −22,278. Neo4j completed runs −38.4% WoW (344,909 → 212,523), led by one single-identity 1.0.8 stack (91,343 → 22,086). azure_openai +381% (34,297 → 165,080), almost all on 1.4.2 with at most 4 identities per row; this is also why 1.4.2's pipeline volume roughly doubled at a 1.98% W1 error rate. vertex_ai +1,557% is one 1.5.4 deployment (48,049). Search-mix shifts are version-local: 1.5.0 CHUNKS_LEXICAL −10,267, 1.5.3 GRAPH_COMPLETION +11,573, and the 1.0.3 fork's RAG_COMPLETION −5,394. None is a routing change in a current release.

Data window and privacy note

This analysis used only the pre-aggregated, anonymized CSVs in telemetry_aggregates/ (2026-07-16 → 2026-09-24, 09-24 partial), plus this repository's git history and public issue/PR metadata via gh. No warehouse or external data source was queried. Finding 1 describes identifier-bearing values found in those CSVs. This report deliberately reproduces none of them; only counts are given. No attempt was made to get finer-grained data.