Add Copilot OTel, Kimi/Kimi Code, and Qwen local usage data sources #12

Context

toki currently has a strong local-first story for Claude Code and Codex CLI. For the contest positioning, the next useful step is to show that the same daemon/indexer model can generalize to other AI coding agents without sending prompts to a central service.

This issue tracks the first expansion batch that looks realistic before the contest deadline:

1. GitHub Copilot CLI OTel JSONL
2. Kimi / Kimi Code wire JSONL
3. Qwen Code chat JSONL

These are intentionally preferred over transcript-only sources because they expose actual token usage fields. The goal is accurate usage accounting, not text-length estimates.

Related market references:

• ccusage supports/experiments with Copilot, Kimi, and Qwen data sources.
• Tokscale and TokenTracker are already broadening from Claude/Codex into multi-agent usage tracking.
• Copilot's OTel support is especially useful for telling the "personal local-first observability" story because it connects directly to OpenTelemetry / GenAI observability.

Source 1: GitHub Copilot CLI OTel JSONL

File discovery

Support both:

~/.copilot/otel/*.jsonl
$COPILOT_OTEL_FILE_EXPORTER_PATH

Recommended user setup for file export:

export COPILOT_OTEL_ENABLED=true
export COPILOT_OTEL_EXPORTER_TYPE=file
mkdir -p "$HOME/.copilot/otel"
export COPILOT_OTEL_FILE_EXPORTER_PATH="$HOME/.copilot/otel/copilot-otel-$(date +%Y%m%d-%H%M%S).jsonl"

If file export was not enabled before a Copilot session, old activity cannot be reconstructed from local files. Surface this clearly in docs/status output.

Expected record shape

Copilot emits OpenTelemetry JSONL records. Usage-bearing records are usually span/log records with an attributes object. Candidate attributes to handle:

{
  "type": "span",
  "name": "chat gpt-5.4-mini",
  "attributes": {
    "gen_ai.operation.name": "chat",
    "gen_ai.response.model": "gpt-5.4-mini",
    "gen_ai.conversation.id": "session-id",
    "gen_ai.usage.input_tokens": 1234,
    "gen_ai.usage.output_tokens": 567,
    "gen_ai.usage.cache_read.input_tokens": 890,
    "gen_ai.usage.reasoning.output_tokens": 123
  }
}

Parser should be lenient because OTel file exporters may encode timestamps and span context in slightly different shapes.

Mapping

• source: copilot
• model: prefer gen_ai.response.model, fallback to model parsed from span/log name if needed
• session_id: prefer gen_ai.conversation.id, fallback to trace id/span context, fallback to file path scoped id
• timestamp: prefer span start/end time or observed timestamp, fallback to file mtime
• input_tokens: gen_ai.usage.input_tokens
• output_tokens: gen_ai.usage.output_tokens
• cache_read_input_tokens: gen_ai.usage.cache_read.input_tokens
• reasoning_output_tokens: gen_ai.usage.reasoning.output_tokens
• cache_creation_input_tokens: only if a stable attribute exists

Dedupe / correctness

• Prefer one usage row per LLM chat span/log.
• Deduplicate by stable span id / response id / trace id + model + timestamp when available.
• Avoid double-counting tool spans and cumulative metric rows if both appear in the same OTel file.
• Include fixture tests for duplicate spans and mixed trace/log/metric rows.

Acceptance criteria

• toki report includes Copilot rows when OTel JSONL exists.
• toki query can filter by source="copilot".
• Missing OTel setup results in a useful status/doc message, not an error.
• No prompt/body text is stored in toki's DB.

Source 2: Kimi / Kimi Code wire JSONL

File discovery

Kimi legacy/current CLI:

~/.kimi/sessions/{GROUP_ID}/{SESSION_UUID}/wire.jsonl
$KIMI_DATA_DIR/sessions/{GROUP_ID}/{SESSION_UUID}/wire.jsonl

Kimi Code newer layout observed in the ecosystem:

~/.kimi-code/sessions/{WORKDIR}/{SESSION_UUID}/agents/{AGENT}/wire.jsonl

Support env override as either one path or comma-separated paths:

KIMI_DATA_DIR=/path/to/.kimi,/path/to/.kimi-code

Expected record shapes

Legacy Kimi wire log uses StatusUpdate with token_usage:

{
  "timestamp": 1770983426.420942,
  "message": {
    "type": "StatusUpdate",
    "payload": {
      "message_id": "chatcmpl-xxx",
      "token_usage": {
        "input_other": 1562,
        "output": 2463,
        "input_cache_read": 0,
        "input_cache_creation": 0
      }
    }
  }
}

Newer Kimi Code reports have been observed as usage.record events:

{
  "type": "usage.record",
  "model": "kimi-code/kimi-for-coding",
  "usage": {
    "inputOther": 1163,
    "output": 352,
    "inputCacheRead": 22272,
    "inputCacheCreation": 0
  },
  "usageScope": "turn",
  "time": 1780410897480
}

Mapping

For legacy Kimi:

• source: kimi
• model: read from nearby config when possible; fallback to kimi-for-coding
• session_id: derive from path {GROUP_ID}/{SESSION_UUID}
• message_id: message.payload.message_id
• timestamp: numeric timestamp seconds -> ms/ISO
• input_tokens: token_usage.input_other
• output_tokens: token_usage.output
• cache_read_input_tokens: token_usage.input_cache_read
• cache_creation_input_tokens: token_usage.input_cache_creation

For Kimi Code:

• source: kimi-code or normalized under kimi with source variant metadata
• model: model, fallback kimi-for-coding
• timestamp: time ms
• input_tokens: usage.inputOther
• output_tokens: usage.output
• cache_read_input_tokens: usage.inputCacheRead
• cache_creation_input_tokens: usage.inputCacheCreation

Dedupe / correctness

• Deduplicate by path + message id when present.
• For usage.record, use a scoped key from path + timestamp + model + usage tuple if no explicit id is available.
• Treat malformed JSONL lines as skipped lines, not fatal file errors.
• Kimi pricing may be difficult because kimi-for-coding can be a dynamic subscription model. Token totals should still be counted even if cost is unknown or zero.

Acceptance criteria

• Legacy ~/.kimi wire logs and new ~/.kimi-code wire logs both have fixtures.
• Reports show input/output/cache split correctly.
• Cost calculation does not block token reporting when pricing is missing.
• Docs mark this source as experimental because the wire schema may still change.

Source 3: Qwen Code chat JSONL

File discovery

~/.qwen/projects/{PROJECT_PATH}/chats/{CHAT_ID}.jsonl
$QWEN_DATA_DIR/projects/{PROJECT_PATH}/chats/{CHAT_ID}.jsonl

Support comma-separated env roots:

QWEN_DATA_DIR="$HOME/.qwen,/backup/qwen"

Expected record shape

Qwen chat JSONL assistant records carry Gemini-style usageMetadata:

{
  "type": "assistant",
  "model": "qwen3-coder-plus",
  "timestamp": "2026-06-30T12:00:00.000Z",
  "sessionId": "chat-session-id",
  "usageMetadata": {
    "promptTokenCount": 1234,
    "candidatesTokenCount": 567,
    "thoughtsTokenCount": 123,
    "cachedContentTokenCount": 890,
    "totalTokenCount": 2814
  }
}

Mapping

• source: qwen
• project: derive from path segment under projects/
• session_id: prefer sessionId, fallback to chat file name
• model: model, fallback unknown
• timestamp: timestamp, fallback file mtime
• input_tokens: usageMetadata.promptTokenCount
• output_tokens: usageMetadata.candidatesTokenCount
• reasoning_output_tokens: usageMetadata.thoughtsTokenCount
• cache_read_input_tokens: usageMetadata.cachedContentTokenCount
• total_tokens: use totalTokenCount only as fallback/extra consistency check, not as a replacement for split fields

Dedupe / correctness

• Only count assistant records with usageMetadata.
• Dedupe by session id + timestamp + model + usage tuple if no message id is available.
• Skip user/tool records that do not carry usage.
• Add tests for cache/reasoning tokens and total-token fallback.

Acceptance criteria

• toki report --source qwen or equivalent filtering works.
• Project attribution works from the encoded path.
• Assistant-only filtering prevents double counting.
• Fixture includes malformed/non-usage lines.

Implementation shape

Suggested internal structure:

• Add a generic UsageSource enum or string-backed source identifier if the current schema assumes only Claude/Codex.
• Keep one parser module per provider:
  • copilot_otel
  • kimi_wire
  • qwen_chat
• Reuse existing JSONL scanning and cursor/incremental logic where possible.
• Maintain a provider-specific cursor per file: inode/mtime/size + last parsed byte or line.
• For append-only JSONL, do incremental reads and do not rescan full files on every daemon tick.
• Add redacted fixtures under tests so parser behavior is locked before wiring into daemon indexing.
• Add docs/status output that explains which paths were scanned and why a source was skipped.

Non-goals for this batch

• Antigravity exact token accounting. Current public/local formats are inconsistent across tools, and transcript-only parsing would require estimates. If added, it should be a separate experimental issue.
• Hermes Agent. Useful, but it is SQLite (~/.hermes/state.db) rather than JSONL, so it should be a separate adapter issue.
• Cursor API scraping. This moves away from the local read-only JSONL story and needs separate auth/security review.

References

• GitHub Copilot CLI OpenTelemetry docs: https://docs.github.com/en/copilot/reference/copilot-cli-reference/cli-command-reference#opentelemetry-monitoring
• OpenTelemetry GenAI semantic conventions: https://opentelemetry.io/docs/specs/semconv/gen-ai/
• ccusage source support policy: https://ccusage.com/guide/source-support-qa/
• ccusage Copilot source: https://ccusage.com/guide/copilot/
• ccusage Kimi source: https://ccusage.com/guide/kimi/
• ccusage Qwen source: https://ccusage.com/guide/qwen/
• Kimi Code CLI: https://github.com/MoonshotAI/kimi-code
• Qwen Code: https://github.com/QwenLM/qwen-code