@rakyll this is interesting, but we do things a bit different there. We re-use the connection between workers in case of OTLP for example (grpc does a better job when connection is shared). We apply the same logic for http as well because the http client deals with open/close/keep connections alive, etc.

Could you please direct me to the documentation that explains how to configure and optimize the settings for the number of connections, worker threads, and queue size (if applicable) for an OpenTelemetry collector, specifically also for HTTP receivers?

In our load testing environment, which is built on Azure Container Apps, we've identified an optimal point at 50 requests per second per node. This configuration utilizes neither significant CPU nor RAM resources. However, when we attempt to handle a higher volume of requests beyond this threshold, we observe an increase in error rates.

It appears that the OpenTelemetry collector's connection handling might be limited by default settings. We aim to make better use of the available CPU and RAM resources, potentially by choosing larger virtual machines, so we can efficiently manage our expected workloads without the need to scale to hundreds of nodes.

Furthermore, when dealing with an even larger number of clients that generate more connections while maintaining the same level of throughput (which we currently simulate with a low number of nodes), we face additional pressure to increase the number of requests per collector pod.

Could you kindly provide guidance or point us towards the relevant documentation to adjust and fine-tune these settings for optimal performance in our OpenTelemetry collector setup for HTTP and gRPC receivers
@andrewhsu