This marks a new chapter for us, and we couldn’t be more energized about what’s ahead. Cursor is fundamentally rethinking the future of software engineering, and we’re looking forward to bringing our experience to that mission.

Our goal with Rotel has always been to enable OpenTelemetry adoption everywhere. Rotel’s focus on resource efficiency and performance has opened the door for developers to implement OpenTelemetry in environments and workloads where cost, resource constraints, or strict runtime requirements made OpenTelemetry impractical. Whether it’s at the edge in serverless environments, on embedded platforms, or at petabyte scale gateways, our goal has always been to push the frontiers of observability.

Today, AI is reshaping how software is built and run in production, and what it means to observe those systems is changing with it. As humans delegate the production of software to agents, observability will only continue to grow in importance. Rotel’s composable architecture is well suited to adapt to the form factors required for observability in an agentic world. We believe working on the frontier of AI provides us the best opportunity to continue Rotel’s mission.

For our users, the most important thing to know is this: Rotel isn’t going anywhere. The codebase will remain available, free and open source under the Apache 2.0 license, and will continue to live on GitHub. Rotel will continue to be built in the open, you’ll have full access as always, and contributions from the community will continue to be accepted. All Rotel repositories, builds, and distributions can now be found at the open source GitHub organization: github.com/rotel-dev.

To our investors, HeavyBit, Uncork, Sunflower, Essence, and our many angels, we are deeply grateful for your encouragement and belief in us over the years. We’ve truly enjoyed the relationships we have made and look forward to continuing them for years to come.

Finally for the community, we’re incredibly grateful for your support, feedback, and confidence in Rotel. It’s meant a lot to us, and it’s played a real role in getting us to this point. Thank you for being part of the journey.

• Efficiency at scale matters: Small reductions in resource usage can have massive implications on cost and efficiency
• Benchmarking OTel + ClickHouse: We construct a pipeline to benchmark streaming trace data into ClickHouse
• Rotel scales 4x: We explore how we scaled from 137K trace spans/sec per-core with the OTel Collector to 462K trace spans/sec per-core with Rotel, detailing several key performance improvements
• Tools and resources: Find the tools we used to drive our benchmark evaluation

Introduction​

Operating an observability platform at petabyte scale demands constant, deliberate attention to resource efficiency. Even small improvements in per-core performance or memory consumption can translate into significant reductions of infrastructure costs.

This post grew out of a talk we gave at a ClickHouse meetup in Denver. We presented our work on Rotel, an OpenTelemetry data plane designed for high performance in large scale systems.

ClickHouse is increasingly chosen for large-scale OpenTelemetry workloads because of its high compression and cost efficiency, while the OTel Collector often emerges as the most expensive part of the pipeline.

Recently, ClickHouse published an article on the importance of efficiency at scale and their experience operating OTel on their internal platform LogHouse.

One detail of the post stood out:

At roughly 2.5k logs per sec per core, for average log lines this would translate into just 10MB per sec on an 8-core box, which is far below what modern hardware can sustain. ClickHouse can process over 12.5k OTel events per second per core, more than five times the ingestion rate, leaving ingestion as the limiting factor rather than storage. While we can not reproduce ClickHouse’s own internal platform, we felt it was worth benchmarking modern OpenTelemetry pipelines. So for the talk we wanted to explore a central question: How do different OpenTelemetry data planes perform when sending tracing data to ClickHouse?

This post walks through our benchmark comparing the OpenTelemetry Collector and Rotel using a synthetic tracing pipeline that streams trace spans through Kafka and writes them to ClickHouse. After benchmarking the OTel collector at 1.1M spans/sec, we walk through how we scaled Rotel to handle up to 3.7M spans/sec on the same hardware by implementing JSON binary serialization, perf analysis of Tokio task management, and leveraging improved LZ4 compression.

Our benchmark framework and tools are presented at the end.

Benchmark framework​

Before we walk through the results, let’s cover a bit of what and how we intend to evaluate the OpenTelemetry collector and Rotel. You can jump directly to the test results if you’d like instead, here.

Tracing pipeline​

We focused our benchmark on writing trace spans to ClickHouse, since trace data grows quickly in large systems. We decided to model our benchmark after a highly reliable streaming pipeline, using Kafka as the log streaming tier. The goal was to represent a system where a larger number of edge collectors publish data to a stream consumed by a smaller number of collectors that write batch data efficiently to ClickHouse. In the pipeline both Rotel and the OTel Collector support the same Kafka Protobuf encoding, so they are interchangeable

Evaluation approach​

Our goal was to determine the maximum throughput a single collector could sustain on fixed hardware, focusing on efficiency rather than scaling up or out. We wanted to see how far one node could be pushed before degrading, while staying within our evaluation budget. We centered the test on the gateway collector because it exports directly to ClickHouse, which performs best with large batch inserts. Maximizing batching efficiency favors running fewer, larger gateway collectors, so this was the component we optimized and measured.

Detecting Saturation

We identified two signals that would indicate we had hit the maximum capacity a collector could handle:

1. Memory growth - Collectors internally buffer payloads in the face of back pressure from downstream components, which shows up as a rapid increase in memory usage.
2. Kafka consumer lag - When a collector can’t keep up with incoming messages, its Kafka consumer lag grows - the time gap between the last read message and current wall time.

Test Setup

For each test we ran the pipeline at a point just below saturation for a total of 15 minutes, recording two metrics:

• Trace spans/sec, as recorded by the load generator
• MB/sec ingest into ClickHouse, as measured by AWS Cloudwatch

Aggregate bandwidth helps provide a normalized throughput metric, given trace spans can have large variations in size between environments. In our pipeline configuration the single edge collector would emit optimized batches to Kafka, reducing overall message count while increasing individual message size.

For each test we capture the average CPU for Rotel and ClickHouse.

OpenTelemetry ClickHouse schema​

For our benchmark we used a single ClickHouse data schema that is compatible with both the OpenTelemetry collector and Rotel. This schema is recommended for use with ClickHouse and the ClickStack observability solution, for OTel metrics, logs, and traces.

The OTel data model relies heavily on key/value attributes to represent the unique properties of infrastructure and application environments that are important for analysis. In the original ClickHouse schema these fields were stored using the Map column type, but a recent feature flag of the OTel collector updates this schema to use the new JSON column type. The JSON column type places a heavier CPU load on ClickHouse, yet allows more flexible and expressive queries. For our evaluation we chose to use the new JSON column type. You can find the full ClickHouse trace schema we used here.

A key part of our setup was the ClickHouse Null table engine, which let us remove disk I/O from the benchmark. The Null engine accepts writes but discards them immediately, so we could measure ingestion throughput and validate schema correctness without storage latency. After identifying the pipeline’s peak throughput, we can focus on scaling ClickHouse to handle real disk writes.

Load generation​

We were unable to push the necessary data volume required through the telemetrygen CLI for the benchmark. Instead, we relied on an internal load generator we had built previously for testing OpenTelemetry and other telemetry pipelines. You can find the project in the otel-loadgen Github repo. It includes additional features for verifying end-to-end data delivery, which we’ll cover in a future post.

We generated approximately 100 spans per trace with varied attributes and metadata. As with any synthetic load test, the dataset won’t identically mirror real production traffic.

Evaluation hardware​

All benchmarks were performed on AWS EC2 instances. Each tier of the evaluation pipeline was run on a separate instance and all instances were created in the same availability zone.

For Kafka and ClickHouse we mounted the data volumes on the instance storage NVMe drives for maximum disk throughput. We used the Amazon Linux 2023 distribution and Docker Compose to run the components.

The goal of these benchmarks were to find what the maximum throughput we could push through a single gateway collector box. We decided on the m8i.2xlarge, a box with 8 cores and 32GB of memory. We had to expand other boxes in the pipeline as we grew, but the gateway collector we kept at an m8i.2xlarge.

Testing OpenTelemetry Collector​

We started our benchmark with the OpenTelemetry Collector, running in the benchmark as both the edge collector and gateway collector.

Set up​

You can find the docker-compose config here.

Results​

Running a single instance of the collector we initially hit a cliff around 700k spans/sec (40MB/s). Beyond that point, the collector’s memory footprint began to grow steadily, even though CPU utilization was only around 50%.

The OpenTelemetry Kafka receiver processes messages in a single goroutine, which likely limited how much traffic it could handle. We experimented with a few Kafka configuration options, including message size parameters, but none noticeably improved throughput. Instead, we scaled horizontally by running a second collector instance on the same box (scale: 2 in our Docker Compose setup).

With two collector processes running, each consuming half the Kafka partitions, the highest throughput we could maintain before saturation was 1.1M trace spans/sec (69MB/s). Past that point the sending queue began to fill and memory usage increased rapidly. When the sending queue was entirely full the Kafka receiver would continue to read messages, but drop them immediately. This meant that the consumer lag didn’t grow, but we were also losing data!

CPU on the gateway collector peaked at a little over 83% during the test and appeared to be the limiting factor. ClickHouse CPU sat around 23%.

Testing Rotel​

Set up​

You can find the docker-compose config here.

Results​

Rotel has a single receiver loop that pulls messages from Kafka, similar to the OTel collector. This led us to believe we had a serial processing bottleneck, which was initially confirmed when we were able to scale to the full instance CPU running two instances of the Rotel process.

With dual Rotel processes, we were able to push up to 1.45M trace spans/sec (76 MB/sec), about a 1.3x improvement on total throughput from the OTel collector. Past 1.45M we noticed Kafka consumer lag slowly increase, implying that we were unable to consume from Kafka fast enough.

At 1.45M trace spans/sec, CPU on the gateway collector instance became a bottleneck, with ClickHouse CPU rising to about 60%.

We continued to search for optimization opportunities, which turned our attention to how we were sending JSON column types.

Improving RowBinary JSON​

In Rotel we use a modified version of the official Rust ClickHouse crate, clickhouse-rs. The clickhouse-rs crate uses RowBinary, a row-oriented, binary serialization format transmitted over HTTP, to write and read data from ClickHouse. In contrast, the OTel Collector’s Go driver and ClickHouse’s internal server communication use the column-oriented native protocol.

When working with JSON column types, the clickhouse-rs crate recommends serializing values to a JSON string before sending them over the wire. ClickHouse does not store JSON columns as raw strings, so this stringification step is only for transport, but it still adds overhead. The client must serialize the JSON, and the server must parse it again. This requires scanning JSON keys and string values to escape characters such as quotes and backslashes, which becomes expensive on large strings at high throughput.

With help from the ClickHouse Slack community, we discovered that JSON columns can be encoded in the native RowBinary format instead. A JSON column is encoded as a sequence of key and value pairs. You write the string key, followed by a type tag for the value, followed by the raw value itself. This avoids JSON serialization costs and lets us stream structured data directly to ClickHouse.

For example, take the simple JSON object:

{
  "a": 42,
  "b": "dog",
  "c": 98
}

The RowBinary encoding for that looks like:

JSON RowBinary first encodes the number of key/value pairs as a variable-length integer (varint), so 03 pairs, followed by the encoding for each key/value pair. For a given key/value pair the encoding uses a varint for the key length, 01 here, followed by the string bytes, finally followed by the value encoding. If the type of the value is known in the JSON type declaration, like a is in the example above, then it encodes that type directly, 2a 00 00 00 00 00 00 00 for the value 42. If the type is not declared, it uses the Dynamic Type encoding. For example, the key c uses the encoding 0a to indicate an Int64, followed by the value 98 (62 00 00 00 00 00 00 00). Finally, the key b is followed by 15 to indicate that the value is of string type, followed by the length of the string 03, then the string “dog”.

This is much more efficient, reducing any serialization/deserialization time for both client and server. While this JSON encoding is not supported in the clickhouse-rs crate yet, we plan to help contribute support soon.

Rerunning the test​

After updating Rotel to use this improved encoding approach, we reran our tests to measure the impact. The results showed that we were not able to exceed the previous peak of 1.45M spans per second, but we did observe roughly a 10% reduction in CPU usage on the ClickHouse server and a small reduction in CPU on the gateway collector. This reduction appeared consistently across multiple runs, which suggests that the lower server-side deserialization cost provided a real benefit.

The synthetic load used in this evaluation does not contain large string values, and the number of attributes per span may differ from production workloads. Although we did not see a significant improvement on the client side, we believe the faster JSON serialization and deserialization path will be more noticeable for spans that contain larger attribute sets.

Doubling throughput with one weird trick (finding and resolving allocator lock contention)​

At this point in our testing we required two instances of Rotel on the gateway collector to saturate the host’s 8 vCPUs and achieve 1.45m events/sec into ClickHouse. The Rotel Kafka receiver ran in a single Tokio task, that when simplified, looked something like this.

There are two problems with this approach:

1. Each step must be done sequentially, i.e. no parallelism.
2. Unmarshaling is an expensive CPU-bound computation that has the potential to block Tokio executor threads.

Tokio is an asynchronous runtime for the Rust programming language that relies on cooperative scheduling. That means that tasks are expected to voluntarily yield control back to the runtime at .await points or other yield points. There are numerous articles on the internet that explain the importance of this contract and the potentially disastrous implications of ignoring it. However, suffice it to say, a tokio task "should never be too far away from an .await point" and a good rule of thumb is to never spend more than 10 to 100 microseconds between each .await.

Throughout Rotel's exporters we use a separate thread pool for the CPU intensive work of marshaling and compressing outgoing request payloads. With our Kafka receiver, payloads are decompressed in a background thread of the rust-rdkafka library before they reach the recv() call. In our initial Kafka receiver implementation we had kept the work of unmarshaling the incoming payloads in the Tokio async task. After determining this unmarshaling was CPU intensive, we updated the Kafka receiver to use the same thread pool as the exporters for running blocking tasks.

After refactoring, the receiver's main processing loop looked something like:

loop {
  select! {
    message = recv() => {
      unmarshaling_futures.push(spawn_blocking(unmarshal(message)))
    },

    unmarshaled_res = unmarshaling_futures.next() => {
      send_to_pipeline(unmarshaled_res)
    }
  }
}

We then reran the tests with a single Rotel process on the gateway collector. We restarted our load generator at the previous maximum of 1.45M trace spans/s and were able to process the same throughput as before. However, we were shocked to find that the CPU load had dropped 40%!

Previously it took two Rotel instances to saturate the vCPUs, which suggested that the Kafka receiver was hitting a serial bottleneck. Adding parallelism by moving marshaling work into separate threads should have removed that limitation. We expected that a single Rotel instance, with the bottleneck resolved, would reach the same throughput as two instances and show similar CPU usage.

With CPU load now significantly lower than before, we continued to increase the throughput. We were able to increase throughput by 2x from 1.45M to 3.6M trace spans/s!

At 3.6M trace spans/s we again reached CPU saturation around 93%.

With the performance gains confirmed, our next step was to understand what drove the improvement in CPU efficiency. To do this, we turned to detailed profiling, comparing the old and new builds using Linux Perf and flame graphs to visualize where CPU time was being spent.

Profiling Rotel’s Kafka receiver with flamegraphs​

We reran the tests against the old and new version of Rotel’s Kafka receivers and this time captured some flamegraphs. On first inspection, nothing immediately jumped right out to us. Can you spot it? In both versions we can clearly see that preparing and exporting traces to ClickHouse is dominating runtime and we’re also spending a fair amount of time unmarshaling messages at the receiver (the unmarshaling occurs in the prost::message::Message::decode routine). The workload generates a lot of short lived objects, so we spend a lot of time allocating and freeing memory.

Old:

New:

Profiling Kafka receiver changes with Linux Perf​

Running Linux perf stat showed some striking differences between the builds.

perf stat -c cycles,instructions,cache-misses,cache-references,context-switches,cpu-migrations

Old:

295612663445 cycles
264853636815 instructions # 0.90 insn per cycle
615670230 cache-misses # 32.963 % of all cache refs
1867733351 cache-references
1224819 context-switches
1230 cpu-migrations
50.296446757 seconds time elapsed

New:

150590256805 cycles
287007890213 instructions # 1.91 insn per cycle
598469068 cache-misses # 51.429 % of all cache refs
1163669589 cache-references
37675 context-switches
43 cpu-migrations
43.716966122 seconds time elapsed

The new build is averaging 1.9 instructions per cycle with only 862 context switches/sec - not amazing for instruction-level parallelism (ILP), but not terrible. The old version was averaging .9 instructions per cycle and an astounding 24,350 context switches/sec, that’s a 32.5x decrease in context switches in the new build! Essentially we were getting no ILP and we were constantly parking and unparking threads. Further, the new version averages 1 cpu migration/sec, showing excellent cache affinity while the old version averaged 24.5 migrations/sec - that’s a 24.5x decrease in the new build indicating the scheduler couldn’t keep threads on the same cores in the old build.

The new version shows much better parallelization characteristics, allowing us to ramp our throughput even further than before. However, all we did was just move some work to different threads.

We could assume the poor performance in the old version was likely due to blocking Tokio executor threads, thereby creating additional overhead in polling, spinning, and attempted work stealing. However digging into the results of a perf report led to our actual more nuanced answer.

Digging into profiles with Linux Perf Report​

After running the tests again with a perf record we had a much more detailed view of the old new and old builds. The new version looks pretty healthy. We’re spending most of our time in compressing data for exporting, transforming OTLP into rows for ClickHouse, and allocating and deallocating memory.

However the old version has quite a different story. It spent 15% of the time freeing memory and only 9.75% of its time compressing and preparing data for ClickHouse, as opposed to ~20% in the new version. We were also spending a lot of time in _raw_spin_unlock_irqrestore, finish_task_switch.isra.0, __lll_lock_wait_private and __lll_lock_wake_private.

If we view the report with children, we can see the time spent waiting on locks was happening when we were attempting to free memory.

So what do these functions do?

_raw_spin_unlock_irqrestore is a Linux kernel function that re-enables interrupts and releases a spinlock, restoring the interrupt context to what it was before a corresponding _raw_spin_lock_irqsave call. More importantly _raw_spin_unlock_irqrestore is called when a task is about to be preempted, allowing the scheduler to perform a context switch. finish_task_switch.isra.0 on the other hand is the compiler optimized version of finish_task_switch which performs cleanup and post context-switch actions.These functions correspond to the massive increase in context switches we observed in the older version.

__lll_lock_wait_private and __lll_lock_wake_private are internal, low-level functions within glibc and are related to the implementation of mutexes and other synchronization primitives. What’s interesting here is that we see the locking showing up when attempting to free memory.

In retrospect, looking back at our old version’s flamegraph, the problem is obvious. Ideally we’d have used a tool like differential flame graphs to compare the two flame graphs side by side making it easier to spot the difference (my kingdom for more, easy to use, flame graph tooling). Fortunately we quickly uncovered the root cause with perf stat and perf record. The contention is in the ClickHouse exporter marshaling functions (TransformPayload) rather than the Kafka receiver.

Glibc multi-threaded allocation​

So now it’s clear why our changes caused CPU utilization to drop while throughput improved. The old version was doing a lot of work, just not the type of work we wanted to be doing! Essentially the old version was spin locking attempting to free memory.

To understand what was going on requires a brief understanding of how the glibc allocator works. Memory is separated into regions called “arenas”, each of these arenas has a mutex lock protecting both allocation and deallocation. Where possible threads will attempt to create separate arenas to avoid lock contention, with the number of allocated arenas growing as a thread pool increases. However, if deallocation occurs on a different thread to the one on which allocation was performed, then the de-allocating thread will have to lock the owning arena - causing other threads to wait and thus contention. In our older version, we allocated our memory for processing trace data during the Kafka receiver’s unmarshaling routine on a Tokio executor I/O task. This work was scheduled on the reduced set of Tokio async executor threads, for which there were only eight, one per core, on the gateway collector node. Later in the pipeline that memory was deallocated as part of the ClickHouse exporter request marashaling, which executed as a blocking task on a much larger thread pool - 10s to 100s of threads.

Here’s an illustration of the data flow and locking pattern running on a two-core box with just two Tokio async executor threads:

In the new version, the allocations occurring during the CPU bound Kafka receiver unmarshaling routine executes on the same, much larger, thread pool as the deallocation was occurring. As the pipeline volume grows, increasing the amount of unmarshaling/marshaling, the blocking thread pool grows. This increases the number of arenas, reducing the chance of lock contention.

The pipeline now appears more like this:

Confirming arena lock contention with jemalloc​

A helpful early reviewer of this post posed the question “I’m curious if this phenomenon reproduces on jemalloc?”. Jemalloc is a general purpose malloc implementation that is specifically designed to reduce lock contention for threaded programs on multi-processor systems. We had tested Rotel with jemalloc in the past and found no significant performance wins. However, our ClickHouse exporter workload combined with the recent Kafka receiver, clearly puts pressure on allocation so we were keen to try running both the old and new versions with jemalloc.

After modifying the previous version with an older task scheduling model to use jemalloc, we saw a reduction in CPU utilization from 93% to 40%. This reduction matched what we saw moving the unmarshaling to the shared thread pool, confirming our findings.

Although jemalloc lowered CPU usage in the previous version, it introduced higher Kafka lag at full throughput. Combined with the fact that jemalloc is no longer actively maintained, we do not plan to adopt it as the default allocator. We may add feature flags so users can opt in to custom allocators such as jemalloc or mimalloc when needed.

Fast LZ4 compression for an additional push​

Rotel uses the recommended LZ4 compression setting for ClickHouse payloads to reduce over-the-wire data transfer. It leverages the same lz4_flex crate that clickhouse-rs uses, but we had imported it directly. When we pulled over the compression support that relied on the lz4-flex crate, we failed to notice the feature settings in the Cargo.toml import.

The lz4-flex crate has both unsafe and safe implementations, with the unsafe versions providing slightly improved performance (read about Rust unsafe here). With lz4-flex, you must opt-in to the unsafe, faster, variant of the library.

The clickhouse-rs crate opts into the unsafe variant of lz4-flex, but we had missed that. Enabling that provided a slight additional boost, taking our gateway collector throughput from 3.6M to 3.7M trace spans/sec and 209 MB/sec.

CPU also decreased just slightly on the gateway collector.

Complete end-to-end evaluation​

After working through multiple optimizations of Rotel and evaluating it against the Null table engine of ClickHouse, we were able to scale the single instance throughput of Rotel from an initial 1.1M trace spans/sec to a total of 3.7M trace spans/sec or 462.5 K trace spans/sec per-core. This is nearly over 4x the original 1.1 M/sec throughput we saw when testing the OTel collector.

We now turned our attention to the final piece which was getting the data permanently into ClickHouse and committed fully to disk. Scaling ClickHouse often comes down to optimizing the schema for both the incoming writes as well as your read queries. In our case we are using the default OTel schema, so our focus was primarily on selecting a box capable of sustaining the write load we had achieved.

To support the massive write load for this test we upgraded our ClickHouse instance. We ended up settling on the following AWS instance. To further remove the possibility of hitting disk utilization limits, we built a RAID0 mount across the four instance store disks.

While testing full ClickHouse writes to disk, we disabled Rotel's use of async inserts and significantly increased the batch size to improve write performance. We updated the Rotel configuration with --clickhouse-exporter-async-inserts=false and --batch-max-size=102400, which allowed us to reach the gateway collector’s previous throughput limit of 3.7 million trace spans per second.

ClickHouse CPU was about 50% with up to 210MB/s of compressed traffic.

Visual inspection​

We were able to see upwards of 3B trace spans in ClickHouse now!

Wrap up​

Efficiency at extreme scale​

At the petabyte scale that ClickHouse runs their internal LogHouse platform, efficiency becomes a practical necessity. They improved their pipeline throughput 20x, while retaining just 10% of the capacity footprint they required before. If they were to continue on their previous trajectory, it would have been prohibitively expensive to operate. Other organizations like Netflix and OpenAI, have also reached similar conclusions: that once you reach this level of volume, efficiency is critically important to the business. This is the context in which we have sought to improve the efficiency of OpenTelemetry collection and have developed Rotel.

Near 4x efficiency improvement​

This work pushed us to optimize Rotel into a high-throughput pipeline for streaming OpenTelemetry tracing data into ClickHouse. In our tests, Rotel reached nearly four times the throughput of the OpenTelemetry Collector on the same hardware, which can translate into meaningful resource savings at large scale. Rotel has first-class support for OpenTelemetry traces, metrics, and logs, and while this post focused on tracing, we plan to evaluate log and metric workloads as part of future benchmarking.

We are also interested in learning which capabilities matter most when operating at this level of volume. If you have ideas or want to share your own scaling challenges, you can join the community on Discord or contribute on GitHub.

Further work​

Here are some topics we would like to explore more after working on this post:

Deep dive on Kafka reliability​

We only briefly touched on it in this post, but Rotel supports reliable delivery guarantees via end-to-end message acknowledgement, to ensure at-least-once delivery when streaming data from Kafka rather than relying on auto-commit which can easily result in data-loss . This support required a number of changes to the pipeline and rigorous testing to ensure Rotel does not drop data while also reducing duplicate deliveries. We would like to deep-dive into what reliable delivery means, how we built it, and the verification tools we built to test it. Look for a follow up post on this.

Explore ClickHouse native protocol​

Rotel’s ClickHouse integration is built on the clickhouse-rs Rust crate, which uses the RowBinary protocol over HTTP. The OpenTelemetry Collector uses the Go ClickHouse driver, which communicates using the native ClickHouse protocol. The native protocol is the same one used for internal ClickHouse to ClickHouse transfers, and benchmarks have shown it to be more than 20 percent faster than RowBinary. ClickHouse has also added support for Apache Arrow Flight, which uses the Arrow in-memory format for efficient transport. We intend to evaluate moving from the row-oriented RowBinary format to one of the column-oriented formats, since they may provide another increase in throughput for Rotel. We plan to share our testing results in a future update.

Further exploring blocking tasks on tokio​

Blocking tasks like deserializing payloads, can have large impacts on Tokio performance. It wasn’t until we worked through this benchmark that we discovered how much, so we want to find other potential impacts of this. We already know that Rotel’s OTLP receiver performs expensive Protobuf deserialization inline during the async task processing of the connection. This is built into the tonic crate, so we would like to explore how best to split this out. From some early analysis with the perf tools we think there may be big wins here.

Reducing memory allocation impacts​

While Rust does not have a garbage collector, at high volumes memory allocation and deallocation still have a significant performance impact. In Rotel we are regularly cycling through memory allocations for short-lived objects as they move through the pipeline. Adopting a memory free list approach that allows us to skip the allocator for common reused buffers should cut performance overheads significantly. Of course, implementing it can be tricky and runs the risk of ballooning your memory usage if you’re not careful. This is another area where we may need to dive into the tonic crate to make necessary changes.

We’d like to thank Sujay Jayakar, Ben Sigelman, Rick Branson, Vlad Seliverstov, Rory Crispin and Achille Roussel for reviewing early drafts of this post.

Addendum​

Considered projects​

While compiling these benchmarks we wanted to include other data planes that offered OpenTelemetry support, but found them incompatible with the benchmark. We chose distributed tracing for this evaluation because it is a strong driver for OTel adoption and data volumes can grow quickly at scale. However, logging and metrics are more classic monitoring telemetry, so many tools still have limited tracing support. While we didn’t have the capacity to include them in this post, we hope to explore logging and metrics benchmarks in the future.

Vector​

Vector is a lightweight tool built for constructing high-performance telemetry pipelines. It has extensive support for many sources and destination sinks, allowing it to integrate with many services and projects. Development is now driven by DataDog and Vector supports their observability pipelines product offering.

OpenTelemetry support in Vector is relatively new and not compatible with many destinations. The Vector data model did not originally have native support for traces, so OTel traces in particular have limited support. Given this, we were unable to evaluate Vector due to the lack of tracing support in both the Kafka and ClickHouse sinks.

• Send traces from opentelemetry source to kafka sink
• Storing trace spans in ClickHouse

Fluent Bit​

Fluent Bit is the alternative to Fluentd, written in C, that is focused on performance. Fluent Bit has inputs and outputs for OpenTelemetry data, supporting logs, metrics and traces. Fluent Bit can output and input data to Kafka, allowing you to build a reliable streaming pipeline. However, during our evaluation we discovered that metrics and traces are not yet supported when connecting an OpenTelemetry input to the Kafka output nor via the HTTP output used for ClickHouse.

Easy OTel ClickHouse migrations​

The ClickHouse docs recommend that you disable auto schema creation in the exporter and populate the schemas ahead of time. It is not clear how best to deploy these schema migrations since they are bundled in the OTel exporter.

For Rotel we decided to split DDL management out to a separate CLI utility to make it easy to deploy these migrations. The clickhouse-ddl tool will create schemas compatible with Rotel and the OTel collector.

We've shipped it as a docker container, so you can easily create the necessary tables for ingesting OpenTelemetry data. Here’s an example creating the tables necessary for storing trace spans:

docker run streamfold/rotel-clickhouse-ddl create \
   --endpoint https://abcd1234.us-east-1.aws.clickhouse.cloud:8443 \
   --traces --enable-json

You can also create the migrations with the Null table engine as we did during this post for benchmarking:

docker run streamfold/rotel-clickhouse-ddl create \
   --endpoint https://abcd1234.us-east-1.aws.clickhouse.cloud:8443 \
   --traces --enable-json \
   --engine Null

References​

• Benchmark framework: https://github.com/rotel-dev/rotel-clickhouse-benchmark
• Rotel: https://rotel.dev
• OTel loadgen: https://github.com/rotel-dev/otel-loadgen

Rotel is a new open-source alternative for collecting OpenTelemetry data and is engineered to be an efficient, high-performance solution for receiving, processing, and exporting OpenTelemetry data. Rotel runs as a standalone process and consumes telemetry from external processes or other collection agents. It consumes 75% less memory and 50% less CPU in benchmarks, so it is particularly well-suited for environments where resource optimization is paramount.

Today we’re excited to announce Rotel and we welcome the community to build with us.

Systems approach to telemetry collection​

The OpenTelemetry Collector reference implementation is written in Go and has been successfully deployed at scale, handling large volumes of telemetry data. It’s often deployed to collect telemetry from the application and to aggregate, transform, and forward data from multiple collectors to external vendors. In many architectures, telemetry must be collected close to the application itself by running collection tools in sidecars on Kubernetes, in lightweight serverless functions, or even at the edge. Without a systems-level approach to building these tools, we risk the unpredictability of garbage collection impacting performance and resource usage. Rust offers a unique memory model with strict ownership and borrowing rules, which permit direct control over memory allocation and deallocation. This results in a process that can maintain strict limits on its resource usage at all times, something that a garbage collected runtime just can’t do with the predictability these environments demand.

Drawing from our experience developing infrastructure close to the systems-level, we wanted to tackle the pain points we’ve seen firsthand, things like slow cold starts in AWS Lambda or excessive memory usage in Kubernetes. Our hypothesis was that these challenges could be mitigated by selecting a language that provides more control and predictability over every slice of memory and CPU. We believe certain workloads demand this level of efficiency, and Rotel is our first step toward addressing that need.

Unlocking New OpenTelemetry Possibilities​

We felt confident that Rotel could significantly ease resource utilization and reduce the cost of high-volume telemetry processing, but then earlier this year, another question began to form: What additional benefits would emerge from building a collector for OpenTelemetry from the ground up, in a low-level language?

Could we make it easier for developers to own, configure and ship OpenTelemetry collection by simply installing a module in a project? Could we make filtering and transformation of OpenTelemetry as easy as pulling up your favorite IDE and writing a snippet of code?

Beyond the practical implications, these questions also felt like an exciting technical challenge – a chance to apply our hard-won experience to a problem we deeply cared about and that's why we're excited to introduce you to Rotel!

Key Capabilities​

• Comprehensive OTLP Support: Rotel acts as an OTLP receiver and exporter, supporting gRPC, HTTP/Protobuf, and HTTP/JSON.
• Flexible Exporter Options: Beyond standard OTLP export, Rotel includes exporters for ClickHouse, Datadog, Kafka, AWS EMF, and AWS X-Ray.
• Resource Efficiency: Up to 75% less memory utilization and 50% less cpu usage on OpenTelemetry collector benchmarks.
• Serverless Extension: Easily deploy Rotel on AWS Lambda as an extension with very fast cold-start times.
• Robust Data Delivery: Built-in batching, retry mechanisms, and configurable timeouts ensure reliable data delivery
• Runtime Integrations: Easily bundle Rotel with popular runtimes. We provide simple to use integrations for Python (rotel-dev/pyrotel) and Node.js (rotel-dev/rotel-nodejs).

Getting Started​

The fastest way to experience Rotel is running directly with Docker:

docker run -ti -p 4317-4318:4317-4318 streamfold/rotel \
    --debug-log traces --exporter blackhole

Then use the OpenTelemetry telemetry generator to emit some fake telemetry:

go install github.com/open-telemetry/opentelemetry-collector-contrib/cmd/telemetrygen@latest
telemetrygen traces --otlp-insecure --duration 5s

From the Rotel container you’ll then see log messages indicating it has received trace spans.

See the full Getting Started guide for more information.

Advanced Capabilities​

Python Processor SDK​

One of Rotel's most innovative features is its Python processor SDK. This SDK enables developers to write custom Python code to modify OpenTelemetry data in-flight before export. Rust’s zero-overhead FFI allows Rotel to provide a high-performance Python extension for processing OpenTelemetry.

The SDK includes LSP support for code completion and syntax highlighting, along with prebuilt processors for common use cases such as attribute modification and data redaction. You can install the SDK with: pip install rotel-sdk.

Here's an example of adding a custom attribute to spans:

from rotel_sdk.open_telemetry.common.v1 import KeyValue, AnyValue
from rotel_sdk.open_telemetry.trace.v1 import ResourceSpans

def process_spans(resource_spans: ResourceSpans):
    for scope_spans in resource_spans.scope_spans:
        for span in scope_spans.spans:
            # Add custom attribute to all spans
            span.attributes.append(KeyValue("processed.by",
                AnyValue("my_processor")))

Python and Node.js packages​

You can also install Rotel directly from Python and Node.js packages. These distributions bundle a build of the Rotel agent directly inside the package, so you can install Rotel alongside your application using the same dependency management tools. Developers can deploy their application and OpenTelemetry collection in a single artifact, particularly convenient for single container PaaS environments.

These packages also provide configuration as code, allowing you to configure Rotel in the same language as your application. There’s no requirement to compose large YAML configs or write custom DSL syntax to get started.

Check out the Rotel Python or Node.js packages.

AWS Lambda Extension​

We’ve also developed a native AWS Lambda Extension, available on GitHub, to deploy OpenTelemetry in serverless applications. Existing OpenTelemetry Lambda builds, such as ADOT, are often large and can add up to a second of cold-start initialization time, directly impacting Lambda function response and increasing serverless costs. The Rotel Lambda Extension compresses down to a size of 8MB and starts in under 70ms. This, combined with an adaptive flushing technique that minimizes post-function delays, drastically reduces cold-start impacts.

Beyond reduced cold-start times, the Rotel Lambda Extension integrates with the TelemetryAPI to convert all function logs to OpenTelemetry format and export them to your configured exporter destinations. This allows you to reduce your CloudWatch costs by replacing it with your preferred logging vendor.

ClickHouse Support​

ClickHouse has been a reliable choice for large-scale telemetry data storage thanks to its efficiency and ability to handle very large data volumes. With the introduction of ClickStack, ClickHouse’s open-source observability solution, the project now offers a fully integrated solution for OpenTelemetry.

Rotel’s ClickHouse exporter provides complete support for ClickHouse, allowing you to export metrics, logs, and traces. It also supports the new JSON column type (available in recent ClickHouse releases), which replaces the older map type and simplifies working with key/value attributes common in OpenTelemetry data. This makes it easier to write flexible queries, for example filtering traces by attributes or aggregating metrics by custom tags. For teams building or scaling observability pipelines, combining Rotel with ClickStack offers a straightforward, high-performance observability solution.

Performance​

Rotel has been built from the ground up using Rust, which allows us to achieve system-level performance reliably, without the concern of undefined behavior. Rust’s strict memory model enables high performance without the overhead associated with traditional garbage-collected memory models. It’s a popular choice for systems requiring high performance and low resource usage.

To get a sense of Rotel’s performance, we adapted the OpenTelemetry Collector Contrib benchmarks to run against Rotel. The results: Rotel requires 75% less memory and uses 50% less CPU. These benchmarks run nightly against the latest versions of both the OpenTelemetry Collector and Rotel.

Up next​

Rotel is open-sourced under the Apache 2.0 license and is under active development. While it is currently in alpha and we are actively gathering feedback, many users have already successfully deployed Rotel to production environments and are engaged members of our community.

Rotel’s early roadmap came from our own experience and the feedback of users. While our mission is to build the highest performing, most resource efficient data plane for processing OpenTelemetry, user feedback plays a critical role in helping us decide what to build next. In addition to continuing to support our users and improve upon the features we’ve built, these are some general categories we want to focus on next.

Extending Receiver and Exporter Integrations​

Rotel started with full OpenTelemetry Protocol (OTLP) compatibility and has grown to support a handful of exporter integrations. We want to expand support for additional use cases, both on the receiver and exporter sides. Object storage is crucial for building large-scale telemetry systems, so we plan to add an exporter for it along with support for popular columnar storage formats. Work is already in progress on an Apache Parquet writer.

Rotel only receives OTLP data at the moment, but we want to expand this to include file log tailing and remote scraping of endpoints like Prometheus. We have recently added a Kafka exporter and Kafka receiver, allowing users to build end-to-end stream processing of OpenTelemetry data with Rotel.

Expanding Processor and OTTL support​

We have shown that we can integrate native language processor support with Python via our SDK with low overhead. We plan to expand on these efforts to improve the integration of Python into the processing path and develop a rich processor framework users can build upon. We’re also exploring pure Rust processors for high-volume and compute intensive workloads like: filtering, transformation, and tail sampling.

We also intend to investigate OTTL support. While anything written in a domain specific language like OTTL can be expressed in Python, supporting OTTL will help alleviate switching costs for users who have invested significant time building OTTL processor expressions. We would love to know which processors are most important to the community and which may be particularly well suited for a systems-level approach.

Durable Pipelines​

You can hope for the best, but sometimes you have to plan for the worst, and telemetry pipelines are no exception. Rotel should provide the option to build durable pipelines with options like guaranteed at-least-once delivery, for receivers and exporters that support it. Rotel should allow telemetry to be spooled to disk or S3 when export destinations are offline for extended durations. We want to support durability with little to no overhead and in a way that is simple to configure. Check out the RFC to join in the discussion on adding delivery acknowledgement to Rotel.

OpenTelemetry Arrow​

The OTEL Arrow project is an ongoing effort to augment OTLP with the Apache Arrow columnar format for inter-process communication. Apache Arrow is particularly well suited for the repetitiveness of telemetry data and could significantly improve the efficiency of telemetry pipelines. Many of the leading Arrow libraries are implemented in Rust so Rotel is well suited to integrate with these efforts.

Get Involved​

Ready to explore Rotel? Visit rotel.dev and github.com/rotel-dev/rotel for detailed documentation and examples.

Join our Discord server to discuss the project, share feedback, or ask questions. Whether you're evaluating observability solutions or are simply interested in the technical approach, we welcome your participation.