Skip to content

fix(connector): complete the 16 incomplete connector implementations - #5394

Merged
qqeasonchen merged 3 commits into
apache:developfrom
qqeasonchen:fix/connector-impl-complete
Sep 11, 2026
Merged

qqeasonchen merged 3 commits into
apache:developfrom
qqeasonchen:fix/connector-impl-complete

Conversation

@qqeasonchen

@qqeasonchen qqeasonchen commented Sep 11, 2026 •

Copy link
Copy Markdown
Contributor

What changes?

Completes the 16 incomplete connector implementations under eventmesh-connector-plugin/ on the new architecture. All follow the new-architecture SourceConnector/SinkConnector contract (init/resume/poll/commit + init/put/commit); external-system logic is ported from the master-branch (openconnect) implementations where available.

# Plugin Side Implementation
1 canal source canal TCP client (getWithoutAck → CloudEvent; batch ack on commit)
2 canal sink all-or-nothing JDBC batch (rollback + throw on failure → redelivery)
3 chatgpt source HTTP prompt endpoint (JDK HttpServer) + OpenAI REST completion, no SDK
4 dingtalk source outgoing-callback webhook receiver w/ HMAC-SHA256 signature verification
5 lark source event-subscription callback receiver w/ url_verification challenge
6 slack source Events API receiver w/ v0 HMAC signature verification + challenge
7 wechat source Official Account callback (GET SHA-1 echostr verification + POST XML)
8 wecom source WeCom callback receiver (URL challenge + JSON events)
9 mcp source JSON-RPC 2.0 notification receiver
10 mcp sink JSON-RPC 2.0 forward over HTTP (2xx = ack, throw otherwise)
11 openfunction source function output push receiver
12 openfunction sink function HTTP trigger invoke (Knative-serving style, Ce-* headers)
13 pravega source EventStreamReader via reader-group (native offsets; reconnect on rebalance)
14 pravega sink EventStreamWriter (auto scope/stream creation; write + join)
15 prometheus sink Pushgateway text-exposition push (batched single request)
16 spring sink injectable EventForwarder bridge (fail-fast until the Spring context wires it)

Design notes

  • Push-style sources (IM webhooks, chatgpt, openfunction, mcp): a small JDK HttpServer receives platform callbacks into a buffer; poll() drains it — matching the ConnectorRuntime §8 poll/commit loop without threads inside the connector.
  • At-least-once semantics preserved everywhere: sinks throw on failure (no ACK → EventMesh redelivers); canal source acks the canal batch only after EventMesh accepted the publish.
  • No heavyweight SDKs: OpenAI/IM integrations use plain HttpURLConnection; only canal (canal.client/protocol 1.1.7) and pravega (pravega-client 0.11.0) get real clients (both versions match master). JSON plugins add jackson-databind 2.18.0 (already the repo-managed version).

Verification

  • compileJava green on all 12 touched modules (JDK 21)
  • checkstyleMain 0 violations on all 12 modules (canonical ASF header, ImportOrder groups, VariableDeclarationUsageDistance)
  • No existing tests touched (modified modules have only src/test/resources configs, no test classes)
  • All blobs LF-only (no CRLF autocrlf drift)

Follow-up (out of scope): unit tests per new implementation, resume() offset wiring for push-style sources.

Fixes #5395

All follow the new-architecture SourceConnector/SinkConnector contract
(init/resume/poll/commit + init/put/commit); external-system logic ported
from the master-branch (openconnect) implementations.

- canal source: canal TCP client (getWithoutAck -> CloudEvent; ack on commit)
- canal sink: all-or-nothing JDBC batch (rollback + throw on failure)
- chatgpt source: HTTP prompt endpoint + OpenAI REST completion
- dingtalk/lark/slack/wechat/wecom sources: platform webhook receivers
  (HMAC/SHA1/token signature verification, challenge handshakes)
- mcp source/sink: JSON-RPC 2.0 notification receive + forward
- openfunction source/sink: function output receive + HTTP trigger invoke
- pravega source/sink: EventStreamReader / EventStreamWriter
- prometheus sink: Pushgateway text-format push
- spring sink: injectable EventForwarder bridge (fail-fast until wired)

Dependencies added: canal.client 1.1.7, pravega-client 0.11.0,
jackson-databind 2.18.0 (JSON plugins), mysql-connector-j compileOnly.
Overview (model, running, catalog, matrix) + one page per plugin
(classes, source/sink behavior, config key tables from the actual
props.getProperty defaults, run examples). Replaces the master-era
website 03-connect docs (9 plugins) with full 23-plugin coverage
matching the new-architecture contract.
@qqeasonchen

Copy link
Copy Markdown
Contributor Author

Update: folded the connector docs into this PR so the implementation fix and its operator-facing docs land together (single review/merge unit).

  • Adds docs/feature/connectors/ — overview (model, running, catalog, 23-plugin matrix) + one guide per plugin (classes, source/sink config tables from the real getProperty defaults, runnable examples) + _category_.json + a Connectors bullet in docs/index.md.
  • Docs follow the v1.12.0 website 03-connect structure but cover all 23 plugins under the new-architecture contract (the old site docs covered 9).
  • Supersedes docs(connector): add per-plugin guides for all 23 connectors #5396 (closed in favor of this single PR).

@qqeasonchen
qqeasonchen merged commit 62d51db into apache:develop Sep 11, 2026
1 check passed
qqeasonchen added a commit to qqeasonchen/eventmesh-site that referenced this pull request Sep 11, 2026
…rsion

Sync from apache/eventmesh develop docs/feature/connectors (the docs
folded into PR apache/eventmesh#5394). Website adaptations:
- README.md renamed to connectors.md (versioned_docs convention)
- sidebar_position frontmatter on every page; category at 6.5
- internal links rebased one level deeper (feature/connectors/)
- documentation-map.md gains the Connectors bullet
qqeasonchen added a commit that referenced this pull request Sep 17, 2026
Bring the single source of truth up to date with what actually landed:

- SSE / WS push: unified ACK-tracked redelivery + DLQ now shared with
  long-polling via ReliableDispatcher; real-broker e2e suite in #5389
- Connector Runtime: all 23 plugins fully implemented since #5394 (the
  "rest are templates" note is stale); unit tests still cover only
  file/kafka/pulsar/rocketmq
- A2A: task reaper + Meta-backed agent cards landed in #5346, quota
  classification in #5373; Beta promotion gated on Testcontainers E2E
  (#5340) instead of "pending" items
- add the memory storage backend row (default, zero-dependency dev/CI;
  #5400) and mention it in the features list, both languages
qqeasonchen added a commit that referenced this pull request Sep 17, 2026
…in, A2A to 10108, 10205 gRPC reserved

8080/8081 collide with the most common dev-service ports (Tomcat, Jenkins,
Spring Boot apps, proxies). Move back onto the project's own 1.x default
port block, which keeps long-time users on familiar numbers:/n
- traffic HTTP 8080 -> 10105 (all app protocols multiplex on it, incl.
  the legacy /eventmesh/* bridge)
- admin HTTP    8081 -> 10106
- WebSocket     8082 -> 10107 (opt-in, unchanged default -1)
- A2A Gateway   10105 -> 10108 (frees 10105 for the CORE runtime; the
  experimental gateway takes the adjacent 10108)
- 10205 documented as RESERVED for a future gRPC protocol port (the 1.x
  default): the uni runtime does not ship a gRPC server yet, and the
  port must not be casually taken so a later gRPC addition lands on the
  historical number

Touched: EventMeshApplication defaults, bin/start.sh, docker/Dockerfile(+.connector),
K8s manifests + README, SDK CloudEventsClient default URL + javadocs, A2AClient /
ExampleConstants / A2AGatewayDemo, e2e test literals, docs (quickstart,
configuration, deployment, a2a) and eventmesh.properties comments.

Configuration keys are unchanged - only defaults move; existing
deployments pinning the old ports keep working.

Also point image-smoke-test.yml at the new ports throughout (host = container,
10105/10106 - no offset mapping left).

README / README.zh-CN quick-start port mentions updated too.

Feature docs swept too (client-java, pubsub, streaming, introduction, index,
admin-api, http-api, observability, connectors overview) - connector-local
webhook ports are untouched.

Fix a port-sweep typo (10100 -> 10105) in the documentation map.

README / README.zh-CN capability-status refreshed against the current tree:/n- SSE/WS push: ACK-tracked redelivery + DLQ shared with long-polling
  (ReliableDispatcher); e2e-real-broker suite in #5389
- Connector Runtime: all 23 plugins fully implemented since #5394 (no
  template stubs left); 4 of 23 still carry the only unit tests
- A2A Gateway: reaper + Meta-backed agent cards landed #5346, quota #5373;
  Beta promotion gated on Testcontainers E2E (#5340)
- new Memory storage row (Beta, zero-dependency dev/CI default, #5400)
qqeasonchen added a commit to qqeasonchen/eventmesh that referenced this pull request Sep 21, 2026
…bility table (en/zh)

- Connector Runtime: 23 plugins complete since apache#5394, hardening in apache#5328
- A2A: the Testcontainers E2E promotion gate (apache#5340) closed 2026-09
- Recommendation wording Evaluate -> Usable; stale gate wording updated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

connector: 16 incomplete connector implementations under eventmesh-connector-plugin (stubs and empty bodies)

1 participant