Tags: zhiqinghuang/zipkin
Tags
Add a Kafka 0.10+ compatible collector and attempt to integrate into … …zipkin-server (openzipkin#1586) * Added module to house a kafka 0.10 compatible collector. Made an initial pass at converting the 0.8 collector over to the new Kafka Consumer API. Still need to work on tests, which don't even compile at this point, and shake out issues withthe implementation. * Got tests working for Kafka 0.10 Collector. Added some logging and resource cleanup to the collector. There are still some outstanding todos and logging levels in the tests are probably a bit verbose. * Added auto-configure module for Kafka 0.10 collector. Still need to wire it into zipkin-server. * Wired Kafka 0.10 collector auto-configuration into zipkin-server. * Removed todo comment about adding max message size config point. Determined that this is not necessary. The new consumer will always fetch at least one message, even if it is larger than the configured maximum size per partition or per fetch. See in max.partition.fetch.bytes and fetch.max.bytes https://kafka.apache.org/documentation/#newconsumerconfigs. These settings can still be modified via overrides if there is a need to tune these soft limits on fetch sizes. * Added some documentation on configuration properties. Cleaned up some unused imports and variables. * Switched environment variables from KAFKA10_* to KAFKA_*. * Doc updates for changing environment variables from KAFKA10_* to KAFKA_*. * Merged zipkin.collector.kafka10 configuration properties into zipkin.collector.kafka properties. * Updated test missed when merging zipkin.collector.kafka10 properties into zipkin.collector.kafka properties. * Replaced zipkin.collector.kafka.consumer-threads property with pre-existing zipkin.collector.kafka.streams property. Updated docs to better describe configuration opations for Kafka 0.10 collector. Added a todo for unit testing the collector with more than one consumer thread. * Pom version bumps to get caught up with changes from openzipkin/zipkin/master. * Added packaging of module jar to Kafka 0.10 autoconfigure module. Removed kafak10 collector dependency from zipkin-server. * Added new test cases for kafka10 collector autoconfigure. The first ensures Kafka consumer configuration overrides are correctly included in properties used to construct new consumers. The second ensures the kafka10 collector will not be loaded if the bootstrap servers property is set to an empty value. * Cleaned up log binding and configuration for zipkin-collector-kafka10 unit tests. Deleted unused log4j config file and made kafka consumer logging less verbose when running tests. * Verified that the kafka10 collector external module can be successfully loaded and run with zipkin-server. Moved documentation on running the kafka10 collector from the zipkin-server readme to the zipkin-autoconfigure-collector-kafka10 readme, and updated with explanation of how to use the external module. * kafka10 collector documentation fixes and tweaks. * Fixed spelling error. * Added test to verify kafka10 collector functions correctly with multiple consumer threads. * Attempting to fix failed test on CirleCI. The test checks to make that two different threads were used to consume span messages from Kafka. It previously made an assumption that the names of threads in the thread pool backing the executor service are unique. The theory is that is not always true and the check now looks for unique instances of Thread. * Cleaned up and expanded unit tests for configuration and connectivity failures with the kafka10 collector. * Second attempt to fix test for multi-threaded Kafka collector that fails intermittently. Added check to wait for all consumer threads to receive topic partitions assignments before producing messages. Without this check, whether the messages were consumed all by one thread or distributed across both depended on timing. If the first thread's Kafka consumer started and was assigned both partitions before the second thread's Kafka consumer was started and assigned a partition, the first thread could consume both messages. * Updated handling of thread interrupts in the kafak10 collector to be detected and handled in a more consistent way. Added some logging to capture the reason workers exit. * Changes based on code review feedback. * Version bumps to 1.25.1 to catch up with upstream/master.
PreviousNext