traces

Extract, redact, and export AI coding-agent sessions into privacy-safe JSONL.

traces turns local history from Claude Code, Codex, Cursor, Grok Build, Gemini, OpenCode, Continue, Trae, and Windsurf into distillation-ready datasets. The default path is privacy-first: rule-based redaction always runs; optional layers add a privacy model, gitleaks, and LLM residue cleanup. Publish is refused if a leftover-pattern scan fails.

PyPI: coding-agent-traces (import package and CLI remain traces)
Repo: Infatoshi/traces
License: MIT — extractor/redaction provenance in ATTRIBUTION.md

Raw agent logs are not training data. Formats differ by product, tool calls are often incomplete or uncorrelated, and sessions are full of secrets, paths, and identity. This tool:

1. Normalizes multi-harness session stores into one conversation schema
2. Redacts secrets, paths, emails, and identifiers to stable placeholders
3. Exports DistillKit-compatible chat formats plus causal agent-step formats
4. Verifies before you publish

Related projects (different jobs):

Python ≥ 3.10. Prefer uv:

# From git (recommended while the package is young)
uv pip install "git+https://github.com/Infatoshi/traces"

# Editable checkout
git clone https://github.com/Infatoshi/traces
cd traces
uv sync --extra dev --extra publish

Optional extras:

Optional non-Python tools:

• gitleaks binary for --gitleaks / --gitleaks-fix (brew install gitleaks)
• Any OpenAI-compatible endpoint for --llm-residue http://localhost:8000

# Extract → redact → export (messages + sharegpt by default)
uv run traces run --out ./output --user "$USER" --home "$HOME"

# Prefer native causal formats + legacy DistillKit variants
uv run traces run --out ./output --user "$USER" --home "$HOME" --variant all

uv run traces verify --in ./output/redacted
uv run traces stats  --in ./output/export
uv run traces sample --in ./output/export -n 5 --out ./preview

Output layout:

output/
├── raw_extracted/<source>.jsonl
├── redacted/<source>.jsonl
├── REDACTION_REPORT.json
├── export/
│   ├── messages.jsonl          # DistillKit chat
│   ├── sharegpt.jsonl          # DistillKit ShareGPT pairs
│   ├── execution.jsonl         # native tool-correlated tasks
│   └── assistant_steps.jsonl   # staircase index → execution_id
└── MANIFEST.json

Identity is always passed as flags (--user, --home). Nothing about a particular machine or username is baked into library defaults.

Native export preserves correlated tool calls and real tool results in task-bounded windows split at user turns. For assistant-step index i, messages[:i] is context and messages[i] is the target. Tool outputs are never training targets.

Structural completeness is not task success: tools_complete only means call/result IDs match. Prefer explicit test exit codes or benchmark joins for quality labels.

Exports include a privacy-safe metadata envelope (schema 1.0): opaque trace/project IDs, source, model, day-bucketed lifecycle fields, token totals, and tool/diff counts. Filtering uses fields each adapter normalizes — never regex guesses over conversation text. Set TRACES_METADATA_KEY to keep opaque IDs stable across separate runs.

TRACES_METADATA_KEY='<stable local key>' \
uv run traces export --in ./output/redacted --out ./output/filtered \
  --include-source 'claude_code' --model '*sonnet*' \
  --since 2026-01-01T00:00:00Z --require-tools \
  --require-complete-tools --require-final-assistant --dedup \
  --variant all

Available filters: --include-source, --drop-sources, --project, --model, --status, --reasoning-effort, --session-kind, --since, --until, --min-turns, --max-turns, --min-assistant-chars, --require-tools, --require-diffs, --require-complete-tools, --require-final-assistant, --dedup. Shell globs work for source, project, and model.

No live cursor_cli extractor (~/.cursor/chats) yet. Source formats change often — adapters are the main maintenance surface (see CONTRIBUTING.md).

Always-on: deterministic rule layers (secrets, paths, identifiers, URLs, …).

Optional layers (upstream order): privacy-filter → gitleaks-fix → LLM residue → gitleaks final scan.

uv run traces verify --in ./output/redacted   # exit 1 on leftovers
# Conservative subset if verify fails:
uv run traces clean --in ./output/redacted --out ./output/clean_redacted

uv run --extra publish traces publish \
  --repo YOUR_HF_USER/your-dataset \
  --in ./output/export --variant all --private

publish rescans the export directory and refuses upload if patterns remain. Never publish without a clean verify. Your own datasets are your responsibility; this tool does not make residual risk zero.

For extracting on several machines and combining redacted JSONL, see docs/multi-host.md.

uv sync --extra dev --extra publish
uv run pytest -q
uv run ruff check . --fix

Agent/ops command reference: AGENTS.md.

Redaction is best-effort. Long coding sessions can still encode proprietary code, private strategy, and identity via context that rules do not catch. Review samples (traces sample) before any public upload. Prefer private Hub datasets unless you have curated, reviewed subsets.

MIT. Copyright notices and upstream licenses: LICENSE, NOTICE.md, ATTRIBUTION.md.