Kernel-wide Insight, Instant Observability, AutoTracing, Continuous Profiling

中文文档 · Quick Start · Documentation ·

HUATUO is a cloud-native operating system observability project open-sourced by DIDI and incubated under the CCF. It delivers kernel-level observability for general-purpose cloud-native computing, AI computing, and bare-metal infrastructure services. By integrating Linux kernel dynamic tracing technologies like kprobe, tracepoint, ftrace, and eBPF, HUATUO provides kernel-wide insights: finer-grained metrics, automatic context capture from kernel runtime, and intelligent tracing. Deployed at scale in Didi’s production environment, HUATUO plays a key role in troubleshooting system failures, enhancing the high availability and performance of cloud-native operating systems.

For more information, visit: https://huatuo.tech

• Kernel-Wide Insight: Leverages BPF to maintain performance overhead below 1%, delivering full-stack, low-level observability insights into Linux kernel subsystems like MM, CPU scheduling, networking, and block I/O.
• Instant Observability: An event-driven runtime context capture mechanism that instruments kernel slow paths. It automatically triggers on events such as page faults, scheduling delays, generating detailed data for immediate analysis.
• AutoTracing: Employs automated snapshot retention to resolve performance jitters typical in cloud‑native and AI infrastructure environments, tackling issues such as CPU idle drops, CPU sys spikes, I/O surges, and Loadavg spikes.
• Continuous Profiling: A comprehensive and continuous performance profiling of the operating system and applications, covering CPU, Memory, I/O, and Locks. This drives business innovation and plays a key role in Chaos, HA and Stability Engineering.
• Distributed Tracing: Network-centric distributed tracing for service requests, which maps system call hierarchies and node relationships. It provides end-to-end visibility of microservice interactions in large-scale distributed systems.
• Ecosystem Integration: Integration with mainstream open-source observability stacks like Prometheus, Grafana, Pyroscope, and Elasticsearch. It automatically associates K8s container labels/annotations. Achieved through zero-instrumentation, kernel-level programming with eBPF, ensuring broad compatibility across hardware platforms and Linux distributions.

• Quick Run

  To launch the HUATUO service with Docker:

    $ docker run --privileged --cgroupns=host --network=host -v /sys:/sys -v /run:/run huatuo/huatuo-bamai:latest
  

  To pull metrics from another terminal:

    $ curl -s localhost:19704/metrics
  

• Quick Setup

  To launch the full stack (Elasticsearch, Prometheus, Grafana, and huatuo) using Docker Compose:

    $ docker compose --project-directory ./build/docker up
  

  Once running, access the monitoring dashboard at http://localhost:3000.

  

• NOTE

  Do not deploy images with the latest tag to production environments, as this is a development and testing image. Use a formal release image or binary.

The project supports kernel version 4.18 and later. The following kernel and OS distribution are primarily tested.

For more information, visit https://docs.huatuo.tech

• WeChat Group and Official Account:

This project is open source under the Apache License 2.0. The BPF code is licensed under the GPL license.