cdviz-collector

keywords: cdevents, sdlc, cicd

A service & CLI to collect SDLC/CI/CD events and dispatch as CDEvents.

Documentation | Quick Start | Installation

Features

• Multiple destinations (HTTP webhooks, HTTP SSE, File system, Kafka, PostgreSQL, ClickHouse)
• Multiple event sources (HTTP webhooks, HTTP SSE, File system, S3, Kafka)
• Multi-format parsing: JSON, JSONL, CSV, XML, YAML, TAP, and raw text/text_line
• Event transformation via VRL (Vector Remap Language)
• Three operation modes: server, one-shot sending, batch transformation

For comprehensive feature documentation, see cdviz.dev/docs/cdviz-collector.

Installation

cdviz-collector is distributed as:

• Pre-built binaries for Linux and macOS (GitHub Releases)
• Docker image at ghcr.io/cdviz-dev/cdviz-collector
• Helm chart for Kubernetes deployments
• Cargo package on crates.io

See the Installation Guide for detailed instructions.

Getting Started

See the Quick Start Guide for a 5-minute walkthrough.

Architecture

cdviz-collector uses a pipeline architecture: events flow from sources through an in-memory queue to multiple sinks.

---
config:
  theme: 'base'
  look: 'handDrawn'
  themeVariables:
    darkMode: true
    mainBkg: '#00000000'
    background: '#00000000'
    primaryColor: '#00000000'
    primaryTextColor: '#f08c00'
    secondaryTextColor: '#f08c00'
    tertiaryTextColor: '#f08c00'
    primaryBorderColor: '#f08c00'
    secondaryBorderColor: '#f08c00'
    tertiaryBorderColor: '#f08c00'
    noteTextColor: '#f08c00'
    noteBorderColor: '#f08c00'
    lineColor: '#f08c00'
    lineWidth: 2
---
flowchart LR
  classDef future stroke-dasharray: 5 5

  q>in memory queue of cdevents]

  subgraph sources
    src_cli(cli/stdin)
    src_http(HTTP webhook / SSE)
    src_fs_content(FS folder with cdevents)
    src_fs_activity(FS folder activity)
    src_s3_content(S3 with cdevents)
    src_s3_activity(S3 activity)
    src_kafka(Kafka)
    src_nats(NATS)
    src_ecr(AWS ECR):::future
    src_misc(...):::future
  end
  src_cli --> q
  src_http --> q
  src_fs_content --> q
  src_fs_activity --> q
  src_s3_content --> q
  src_s3_activity --> q
  src_kafka --> q
  src_nats --> q
  src_ecr --> q
  src_misc --> q

  subgraph sinks
    sink_stdout(stdout)
    sink_db(DB)
    sink_clickhouse(ClickHouse)
    sink_http(HTTP)
    sink_kafka(Kafka)
    sink_nats(NATS)
  end
  q --> sink_stdout
  q --> sink_http
  q --> sink_db
  q --> sink_clickhouse
  q --> sink_kafka
  q --> sink_nats

For detailed architecture information, see CLAUDE.md.

Configuration

Configure cdviz-collector via TOML files with environment variable overrides:

• Example config: examples/assets/cdviz-collector.toml
• Base config: src/assets/cdviz-collector.base.toml
• Environment variables: CDVIZ_COLLECTOR__SECTION__KEY__VALUE

See the Configuration Guide and Use Cases & Examples for detailed information.

Usage

cdviz-collector provides three commands:

connect - Run as a Service

Launch collector as a long-running server to connect sources to sinks.

cdviz-collector connect --config cdviz-collector.toml

See the connect documentation for detailed options.

send - Send Events Directly

Send JSON data directly to a sink for testing and scripting.

cdviz-collector send --url https://api.example.com/webhook --data '{"test": "value"}'

See the send documentation for detailed options.

transform - Batch File Transformation

Transform local JSON files using configured transformers.

cdviz-collector transform --input ./input --output ./output --transformer-refs github_events

See the transform documentation for detailed options.

For all available options, use --help with any command or see the usage documentation.

GitHub Action

Send CDEvents from GitHub Actions workflows using cdviz-dev/send-cdevents.

Development

This project uses mise for task management:

mise install          # Setup environment
mise run build        # Build project
mise run test         # Run tests
mise run lint         # Run linting
mise run ci           # Full CI pipeline

See CONTRIBUTING.md for detailed development guidelines.

For AI assistance context, see CLAUDE.md.

License

This software is distributed under Apache Software License 2.0 ASL-2.0. You can subscribe to commercial support at http://cdviz.dev.

By using this software, you agree to comply with the terms of one of the above licenses.

For exceptions, see LICENSING.md.

Scripts, Transformer's template and Licensing

• Built-in scripts (included in this repository) are licensed under Apache Software License 2.0 ASL-2.0.
• User-provided scripts (loaded at runtime) are not subject to ASL-2.0 and can be under any license.

Contributing

We welcome contributions! Please see our Contributing Guide for more details.

By contributing to this project, you agree to the Contributor License Agreement (CLA).

Note: This project has been developed with assistance from Claude Code. All AI-generated code has been carefully reviewed, tested, and validated to ensure quality, security, and adherence to Rust best practices.

Downloads