Repository navigation
ESQL: Add Connector SPI and gRPC/Arrow Flight module - #142667
Merged
Merged
Conversation
Extend the external data source framework with a request/response Connector SPI for services that return data over a network protocol. Implement the first connector for Apache Arrow Flight over gRPC. - ConnectorFactory/Connector/ResultCursor: new SPI interfaces for connection-oriented external sources - AsyncConnectorSourceOperatorFactory: async operator that drains a connector on a background thread with backpressure - OperatorFactoryRegistry: dispatches ConnectorFactory via the ExternalSourceFactory hierarchy - FlightConnectorFactory/FlightConnector: Arrow Flight implementation with schema resolution and streaming result cursor - FlightTypeMapping: Arrow-to-ESQL type conversion - GrpcDataSourcePlugin: plugin wiring via ServiceLoader - EmployeeFlightServer: in-memory test fixture serving employee data - FlightFormatSpecIT: integration tests via csv-spec against a live Flight server - ExternalSourceDrainUtils: shared drain loop with timeout - entitlement-policy.yaml: grants manage_threads and outbound_network Developed using AI-assisted tooling
Collaborator
|
Pinging @elastic/es-analytical-engine (Team:Analytics) |
Collaborator
|
Hi @costin, I've created a changelog YAML for you. |
- Add internal-test-artifact plugin so QA can resolve test sources - Add lintian codeless-jar override for grpc-context metadata jar - remove unused licenses
costin
force-pushed
the
esql/connector-spi
branch
from
February 20, 2026 15:18
fc0e227 to
754e5cd
Compare
FileSourceFactory was only registered under "file" but format readers return their format name (e.g. "parquet", "csv") as the sourceType. Register the fallback factory under each format name so the operator factory registry can find it at execution time.
IcebergTableCatalog.canHandle() claims all S3 paths, intercepting plain CSV/Parquet files before FileSourceFactory gets a chance. Change resolveSingleSource() to try-and-continue: if a factory claims canHandle() but fails in resolveMetadata(), fall through to the next factory instead of propagating the error immediately. - ExternalSourceResolver: catch-and-continue in resolution loop
Member
Author
|
@elasticmachine run elasticsearch-ci/part-1 |
bpintea
approved these changes
Feb 21, 2026
PeteGillinElastic
pushed a commit
to PeteGillinElastic/elasticsearch
that referenced
this pull request
Feb 23, 2026
Extend the external data source framework with a request/response Connector SPI for services that return data over a network protocol. Implement the first connector for Apache Arrow Flight over gRPC. ConnectorFactory/Connector/ResultCursor: new SPI interfaces for connection-oriented external sources AsyncConnectorSourceOperatorFactory: async operator that drains a connector on a background thread with backpressure OperatorFactoryRegistry: dispatches ConnectorFactory via the ExternalSourceFactory hierarchy FlightConnectorFactory/FlightConnector: Arrow Flight implementation with schema resolution and streaming result cursor FlightTypeMapping: Arrow-to-ESQL type conversion GrpcDataSourcePlugin: plugin wiring via ServiceLoader EmployeeFlightServer: in-memory test fixture serving employee data FlightFormatSpecIT: integration tests via csv-spec against a live Flight server ExternalSourceDrainUtils: shared drain loop with timeout entitlement-policy.yaml: grants manage_threads and outbound_network Developed using AI-assisted tooling
elasticsearchmachine
pushed a commit
that referenced
this pull request
Jul 16, 2026
… test (#154075) ## Summary - Excludes `grpc-context` from the `esql-datasource-grpc` plugin's runtime classpath so it is not bundled into the deb package - Removes the now-unnecessary lintian override for `codeless-jar` on that path - Unmutes `DebMetadataTests#test05CheckLintian` ## Background Since gRPC Java 1.57.0, `grpc-context` is intentionally an empty JAR — all classes were moved to `grpc-api` to fix a JPMS split-package issue ([grpc/grpc-java#3522](grpc/grpc-java#3522)). The `grpc-context` artifact now exists solely as a backward-compatible Maven coordinate that pulls in `grpc-api`. `grpc-api:1.78.0` (directly declared in `esql-datasource-grpc`) transitively pulls `grpc-context:1.78.0` into the plugin bundle. Lintian flags this empty jar with a `codeless-jar` warning, failing the packaging test. A lintian override was added in #142667 but proved ineffective on CI Ubuntu 2204/2404 (lintian reported `0 hints overridden; 9 unused overrides`). The correct fix is to not ship the jar at all. This mirrors the approach already used in `esql-datasource-gcs`, which excludes `grpc-context` for the same reason. Closes #142819
elasticsearchmachine
pushed a commit
that referenced
this pull request
Jul 16, 2026
… test (#154075) (#154109) ## Summary - Excludes `grpc-context` from the `esql-datasource-grpc` plugin's runtime classpath so it is not bundled into the deb package - Removes the now-unnecessary lintian override for `codeless-jar` on that path - Unmutes `DebMetadataTests#test05CheckLintian` ## Background Since gRPC Java 1.57.0, `grpc-context` is intentionally an empty JAR — all classes were moved to `grpc-api` to fix a JPMS split-package issue ([grpc/grpc-java#3522](grpc/grpc-java#3522)). The `grpc-context` artifact now exists solely as a backward-compatible Maven coordinate that pulls in `grpc-api`. `grpc-api:1.78.0` (directly declared in `esql-datasource-grpc`) transitively pulls `grpc-context:1.78.0` into the plugin bundle. Lintian flags this empty jar with a `codeless-jar` warning, failing the packaging test. A lintian override was added in #142667 but proved ineffective on CI Ubuntu 2204/2404 (lintian reported `0 hints overridden; 9 unused overrides`). The correct fix is to not ship the jar at all. This mirrors the approach already used in `esql-datasource-gcs`, which excludes `grpc-context` for the same reason. Closes #142819
elasticsearchmachine
pushed a commit
that referenced
this pull request
Sep 18, 2026
… test (#154075) (#159696) ## Summary - Excludes `grpc-context` from the `esql-datasource-grpc` plugin's runtime classpath so it is not bundled into the deb package - Removes the now-unnecessary lintian override for `codeless-jar` on that path - Unmutes `DebMetadataTests#test05CheckLintian` ## Background Since gRPC Java 1.57.0, `grpc-context` is intentionally an empty JAR — all classes were moved to `grpc-api` to fix a JPMS split-package issue ([grpc/grpc-java#3522](grpc/grpc-java#3522)). The `grpc-context` artifact now exists solely as a backward-compatible Maven coordinate that pulls in `grpc-api`. `grpc-api:1.78.0` (directly declared in `esql-datasource-grpc`) transitively pulls `grpc-context:1.78.0` into the plugin bundle. Lintian flags this empty jar with a `codeless-jar` warning, failing the packaging test. A lintian override was added in #142667 but proved ineffective on CI Ubuntu 2204/2404 (lintian reported `0 hints overridden; 9 unused overrides`). The correct fix is to not ship the jar at all. This mirrors the approach already used in `esql-datasource-gcs`, which excludes `grpc-context` for the same reason. Closes #142819 (cherry picked from commit b7b9dcd) # Conflicts: # muted-tests.yml Co-authored-by: Rene Groeschke <rene@elastic.co>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Extend the external data source framework with a request/response
Connector SPI for services that return data over a network protocol.
Implement the first connector for Apache Arrow Flight over gRPC.
connection-oriented external sources
connector on a background thread with backpressure
ExternalSourceFactory hierarchy
with schema resolution and streaming result cursor
Flight server
Developed using AI-assisted tooling