Repository navigation
Conversation
…ix rollback NPE in flatMessage mode 1. KafkaCanalConnector / KafkaOffsetCanalConnector add an overloaded constructor accepting custom kafka consumer Properties, so clients can pass security.protocol / sasl.mechanism / sasl.jaas.config (or SSL settings) to connect to secured kafka clusters. Server side already supports this via kafka.* passthrough in canal.properties, but canal.client had no entry. Existing constructors are kept and delegate to the new one. 2. KafkaCanalConnector.rollback() called kafkaConsumer.commitSync() on the kafkaConsumer2 (flatMessage) branch, which throws NPE in flatMessage mode.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
背景 / Motivation
服务端(
CanalKafkaProducer)和 client-adapter(CanalKafkaConsumer)都支持通过kafka.*前缀把任意配置透传给 Kafka 客户端,canal.properties里也有 SASL 的示例:但
canal.client模块的KafkaCanalConnector只有一个构造方法,consumer 配置在构造方法里写死,没有任何注入自定义配置的入口,导致 client 无法直接连接启用了 SASL / SSL 认证的 Kafka 集群(参考 #3066,只能自己改源码)。Server side and client-adapter already pass
kafka.*properties through to the Kafka client, butKafkaCanalConnectorincanal.clienthardcodes all consumer properties and offers no way to injectsecurity.protocol/sasl.mechanism/sasl.jaas.config, so it cannot connect to a SASL/SSL secured cluster without patching the source.改动 / Changes
KafkaCanalConnector/KafkaOffsetCanalConnector新增重载构造方法,多接一个Properties customProperties,在默认配置之后putAll,可用于传 SASL / SSL 或其他任意 consumer 参数。原构造方法保留并委托给新方法,完全向后兼容。KafkaOffsetCanalConnector仅在customProperties未指定auto.offset.reset时才写入默认的earliest。修复
KafkaCanalConnector.rollback()在 flatMessage 模式下的 NPE:kafkaConsumer2分支里seek的是kafkaConsumer2,但commitSync()调的是kafkaConsumer(flatMessage 模式下为null)。Fix NPE in
rollback(): thekafkaConsumer2branch seeks onkafkaConsumer2but committed onkafkaConsumer, which isnullin flatMessage mode.验证 / Verification
mvn -pl client -am compile(JDK 8) 通过