fix(inkless:ci): green the upstream test suite (diskless test fixes + flaky quarantine) by jeqo · Pull Request #700 · aiven/inkless

jeqo · 2026-07-14T22:25:08Z

Upstream Kafka tests broken on main (see https://github.com/aiven/inkless/actions/runs/29324563467/job/87077973795) because they were not adapted to Inkless changes or are flaky.

This PR makes the JUnit test workflow a trustable signal: green unless a non-flaky test breaks or infrastructure fails. Three independent concerns, one commit each:

Three upstream Kafka unit tests were failing deterministically because they were never adapted to Inkless (Diskless Topics) changes. Fixed the stale expectations. Test-only, no production code touched.
A set of upstream integration tests that are documented-flaky on Apache Kafka (several with currently-open flaky-test JIRA tickets) were reddening the noflaky-nonew CI bucket independent of any Inkless change. Tagged them @Flaky so they run only in the flaky bucket while tracking the upstream tickets.
The flaky/quarantine job itself was going red whenever a quarantined test flaked, which is expected noise rather than a real break. Made that job report-only (fail only on real infrastructure problems), so the workflow is trustable.

Commit 1 - `fix(inkless:ci): update upstream tests for diskless additions`

Test	Root cause	Fix
`TopicImageNodeTest.testChildPartitionId`	`PartitionRegistration.toString()` now includes the diskless fields (`classicToDisklessStartOffset`, `disklessProducerStates`, `disklessLeaderEpoch`)	Update expected string
`NodeApiVersionsTest.testUnknownApiVersionsToString`	The "unknown" apiKey `337` no longer sorts last once the diskless client APIs (`INIT_DISKLESS_LOG`=500, `ALTER_DISKLESS_SWITCH`=501) exist, so `toString()` no longer ends with it	Use `Short.MAX_VALUE` as the unknown key
`MetricsTest.testGeneralBrokerTopicMetricsAreGreedilyRegistered`	All-topics `BytesIn`/`BytesOut` meters are registered for both `topicType=classic` and `topicType=diskless`; the diskless variants share a metric name with their classic counterparts, so they are extra JMX MBeans but not extra `metricMapKeySet()` entries (22 vs 24)	Account for the diskless-tagged duplicates in the assertion; capture `topicMetrics(None)` once

I verified against apache/trunk that none of these have an upstream fix to adopt: the upstream tests are byte-identical and pass there because upstream lacks the diskless fields / API keys / dual classic+diskless meters. The divergence is entirely Inkless-introduced, so the fixes belong in the fork.

Commit 2 - `test(inkless:ci): quarantine known-flaky upstream integration tests with @Flaky`

Each tagged test maps to an upstream Apache Kafka flaky-test JIRA ticket; in several cases the exact method (and parameter) is named, and the ticket is still open. These are not affected by this PR's diff.

Test	Ticket	Status
`MirrorConnectorsIntegrationBaseTest.testReplicateSourceDefault` (inherited by 5 subclasses)	KAFKA-15926 (also KAFKA-15927)	open
`MirrorConnectorsIntegrationBaseTest.testReplicateTargetDefault` (inherited by 5 subclasses)	KAFKA-15926 (no dedicated ticket; same MM2 harness)	open
`MirrorConnectorsIntegrationBaseTest.testSyncTopicConfigs` (inherited by 5 subclasses)	KAFKA-15945 (also KAFKA-15523, KAFKA-14971)	resolved (recurs)
`MirrorConnectorsIntegrationBaseTest.testReplication` (inherited by IdentityReplication)	KAFKA-16488 (also KAFKA-12566)	resolved (recurs)
`MonitorableSinkIntegrationTest.testMonitorableSinkConnectorAndTask`	KAFKA-18952	resolved (recurs)
`TaskAssignorConvergenceTest.randomClusterPerturbationsShouldConverge[balance_subtopology]`	KAFKA-16491	open
`ShareConsumerTest.testPollThrowsInterruptExceptionIfInterrupted`	KAFKA-19961	open
`SaslOAuthBearerSslEndToEndAuthorizationTest.testNoConsumeWithoutDescribeAclViaSubscribe`	KAFKA-9655	open

Notes on placement:

The MM2 family is tagged once on the base methods in MirrorConnectorsIntegrationBaseTest; JUnit propagates the tag to the inherited executions in all subclasses (IdentityReplication, ExactlyOnce, SSL, Transactions, CustomForwardingAdmin).
testNoConsumeWithoutDescribeAclViaSubscribe is defined in the shared EndToEndAuthorizationTest base and inherited by ~10 auth-mechanism subclasses. To avoid silencing it for all mechanisms, it is overridden solely in the OAUTHBEARER subclass to carry the tag; every other SASL variant keeps upstream coverage.
build.gradle: added testImplementation project(':test-common:test-common-util') (home of the @Flaky annotation) to :streams and :connect:mirror; the other affected modules already depend on it. The annotation is a thin meta-annotation for @Tag("flaky"), which KafkaPostDiscoveryFilter excludes from the main suite.

Commit 3 - `fix(inkless:ci): make the flaky test job report-only so CI is trustable`

The flaky/quarantine matrix job runs only @Flaky-tagged tests. junit.py previously failed the job on any non-zero Gradle exit code, so a single quarantined test flaking turned the whole workflow red even when nothing at the Kafka coverage level was broken. This defeats the purpose of quarantine and makes the signal untrustworthy.

Change: junit.py now treats a non-zero Gradle exit code as non-fatal when running the flaky job (RUN_FLAKY=true, passed from the matrix in build.yml), while still failing on real infrastructure problems - timeouts (exit 124), thread dumps (hangs), and a missing exit code. Flaky failures are still reported in the job summary tables. The noflaky and new jobs are unaffected and remain the trustable gating signal.

Net effect: the workflow is green unless a non-flaky test breaks or infrastructure fails.

Verification

compileTestJava / compileTestScala + checkstyleTest pass for every touched module.
The three diskless test fixes were confirmed green in CI (run 29402697743): they no longer appear in the failure set; the remaining reds were exactly the flaky integration tests quarantined in commit 2.
The flaky filter was confirmed to exclude the tagged tests from the noflaky bucket on two representative cases (an inherited MM2 method and the SASL override), both reporting "No tests found for given includes" in default mode; they still run in the flaky bucket with retries.
The MM2 config-sync tests pass locally in isolation, confirming they are load-sensitive flakes (not a code regression), consistent with their upstream flaky tickets.
junit.py exit behavior verified locally: flaky + test failures -> exit 0; noflaky + test failures -> exit 1; flaky + timeout (124) -> exit 1; clean / unset RUN_FLAKY -> unchanged.

Copilot

Pull request overview

Updates upstream Kafka unit tests in the Inkless fork to reflect Diskless Topics-related behavioral changes (string formatting, ApiKey ordering, and additional JMX MBeans due to topicType-tagged metrics), restoring CI green without modifying production code.

Changes:

Adjust PartitionRegistration expected toString() output to include diskless-related fields.
Make NodeApiVersionsTest use a guaranteed-last unknown apiKey (Short.MAX_VALUE) now that higher-valued diskless ApiKeys exist.
Update broker topic metrics test to account for diskless-tagged all-topics BytesIn/BytesOut being registered as additional JMX MBeans.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
metadata/src/test/java/org/apache/kafka/image/node/TopicImageNodeTest.java	Updates expected partition registration string to include diskless fields.
clients/src/test/java/org/apache/kafka/clients/NodeApiVersionsTest.java	Ensures the “unknown apiKey sorts last” assertion remains stable with new ApiKeys.
core/src/test/scala/unit/kafka/metrics/MetricsTest.scala	Adjusts assertion to account for diskless-tagged duplicate all-topics metrics in JMX.

jeqo · 2026-07-16T07:40:34Z

CI run green: https://github.com/aiven/inkless/actions/runs/29422114572

viktorsomogyi · 2026-07-24T14:56:30Z

    }

    @Test
+    @Flaky(value = "KAFKA-16488", comment = "Chronically flaky MM2 integration test across all subclasses; see also KAFKA-12566.")


As far as I know this annotation can't be inherited as the annotation processor doesn't walk the method hierarchy. Did your test flag this as flaky though?

Good catch. I confirmed against the green run (29422114572, Java 17): the flaky bucket ran and the main gate excluded the four base methods (testReplication, testReplicateSourceDefault, testReplicateTargetDefault, testSyncTopicConfigs) across Identity, SSL, Transactions and CustomForwardingAdmin — so inheritance does carry the tag for non-overridden methods.
But this one was missing: MirrorConnectorsIntegrationExactlyOnceTest overrides testReplication() without re-declaring @Flaky, so it ran in the gating suite instead of the flaky bucket.
Pushed a fixup adding @Flaky to the override

New run on CI confirms results: https://github.com/aiven/inkless/actions/runs/30467415276/job/90632800023

Three upstream Kafka unit tests broke on main because they were not adapted to Inkless changes: - TopicImageNodeTest.testChildPartitionId: PartitionRegistration.toString now includes the diskless fields (classicToDisklessStartOffset, disklessProducerStates, disklessLeaderEpoch); update expected string. - NodeApiVersionsTest.testUnknownApiVersionsToString: the "unknown" apiKey 337 no longer sorts last once the diskless client APIs (500/501) exist, so toString no longer ends with it; use Short.MAX_VALUE instead. - MetricsTest.testGeneralBrokerTopicMetricsAreGreedilyRegistered: all-topics BytesIn/BytesOut are registered for both topicType=classic and diskless. The diskless variants share a metric name with their classic counterparts, so they are extra JMX MBeans but not extra metricMapKeySet() entries; account for the diskless-tagged duplicates in the assertion. Test-only changes; no production code touched.

@flaky

…ith @flaky These integration tests are documented-flaky on upstream Apache Kafka (several with currently-open flaky-test JIRA tickets naming the exact methods/params) and reddened the noflaky-nonew CI bucket independent of any Inkless change. Tag them @flaky so they are excluded from the main suite and run only in the flaky bucket (with retries), while tracking the upstream tickets. MM2 integration family (tagged once on the base methods in MirrorConnectorsIntegrationBaseTest; inherited by IdentityReplication, ExactlyOnce, SSL, Transactions, CustomForwardingAdmin): - testReplicateSourceDefault -> KAFKA-15926 (open; also KAFKA-15927) - testReplicateTargetDefault -> KAFKA-15926 (no dedicated ticket; same harness) - testSyncTopicConfigs -> KAFKA-15945 (also KAFKA-15523, KAFKA-14971) - testReplication -> KAFKA-16488 (also KAFKA-12566) Standalone: - MonitorableSinkIntegrationTest.testMonitorableSinkConnectorAndTask -> KAFKA-18952 - TaskAssignorConvergenceTest.randomClusterPerturbationsShouldConverge -> KAFKA-16491 (open) - ShareConsumerTest.testPollThrowsInterruptExceptionIfInterrupted -> KAFKA-19961 (open) - SaslOAuthBearerSslEndToEndAuthorizationTest.testNoConsumeWithoutDescribeAclViaSubscribe -> KAFKA-9655; overridden solely to tag the OAUTHBEARER variant so other SASL mechanisms keep upstream coverage. build.gradle: add testImplementation test-common-util to :streams and :connect:mirror (source of the @flaky annotation; the other affected modules already depend on it). Verified: compile + checkstyle pass for all touched modules, and the flaky filter excludes the tagged tests from the noflaky bucket (inherited MM2 method and the SASL override both report "No tests found" in default mode).

The flaky/quarantine matrix job runs only @Flaky-tagged tests (known-unreliable upstream tests, e.g. the MM2 config-sync integration tests that pass in isolation but flake under CI load). Previously junit.py failed the job on any non-zero Gradle exit code, so a flaked quarantined test turned the whole workflow red even though nothing at the Kafka coverage level was broken - defeating the point of quarantine and making the signal untrustworthy. Make junit.py treat a non-zero Gradle exit code as non-fatal when running the flaky job (RUN_FLAKY=true), while still failing on real infrastructure problems: timeouts (exit 124), thread dumps (hangs), and a missing exit code. Flaky failures are still reported in the job summary tables. The noflaky and new jobs are unaffected and remain the trustable gating signal. Net effect: the workflow is green unless a non-flaky test breaks or infra fails. Verified junit.py exit behavior locally: - flaky + test failures -> exit 0 (green) - noflaky + test failures -> exit 1 (red) - flaky + timeout (124) -> exit 1 (infra, still red) - clean / unset RUN_FLAKY -> unchanged

@flaky

…tests with @flaky

@flaky

… flaky quarantine) (#700) * fix(inkless:ci): update upstream tests for diskless additions Three upstream Kafka unit tests broke on main because they were not adapted to Inkless changes: - TopicImageNodeTest.testChildPartitionId: PartitionRegistration.toString now includes the diskless fields (classicToDisklessStartOffset, disklessProducerStates, disklessLeaderEpoch); update expected string. - NodeApiVersionsTest.testUnknownApiVersionsToString: the "unknown" apiKey 337 no longer sorts last once the diskless client APIs (500/501) exist, so toString no longer ends with it; use Short.MAX_VALUE instead. - MetricsTest.testGeneralBrokerTopicMetricsAreGreedilyRegistered: all-topics BytesIn/BytesOut are registered for both topicType=classic and diskless. The diskless variants share a metric name with their classic counterparts, so they are extra JMX MBeans but not extra metricMapKeySet() entries; account for the diskless-tagged duplicates in the assertion. Test-only changes; no production code touched. * test(inkless:ci): quarantine known-flaky upstream integration tests with @flaky These integration tests are documented-flaky on upstream Apache Kafka (several with currently-open flaky-test JIRA tickets naming the exact methods/params) and reddened the noflaky-nonew CI bucket independent of any Inkless change. Tag them @flaky so they are excluded from the main suite and run only in the flaky bucket (with retries), while tracking the upstream tickets. MM2 integration family (tagged once on the base methods in MirrorConnectorsIntegrationBaseTest; inherited by IdentityReplication, ExactlyOnce, SSL, Transactions, CustomForwardingAdmin): - testReplicateSourceDefault -> KAFKA-15926 (open; also KAFKA-15927) - testReplicateTargetDefault -> KAFKA-15926 (no dedicated ticket; same harness) - testSyncTopicConfigs -> KAFKA-15945 (also KAFKA-15523, KAFKA-14971) - testReplication -> KAFKA-16488 (also KAFKA-12566) Standalone: - MonitorableSinkIntegrationTest.testMonitorableSinkConnectorAndTask -> KAFKA-18952 - TaskAssignorConvergenceTest.randomClusterPerturbationsShouldConverge -> KAFKA-16491 (open) - ShareConsumerTest.testPollThrowsInterruptExceptionIfInterrupted -> KAFKA-19961 (open) - SaslOAuthBearerSslEndToEndAuthorizationTest.testNoConsumeWithoutDescribeAclViaSubscribe -> KAFKA-9655; overridden solely to tag the OAUTHBEARER variant so other SASL mechanisms keep upstream coverage. build.gradle: add testImplementation test-common-util to :streams and :connect:mirror (source of the @flaky annotation; the other affected modules already depend on it). Verified: compile + checkstyle pass for all touched modules, and the flaky filter excludes the tagged tests from the noflaky bucket (inherited MM2 method and the SASL override both report "No tests found" in default mode). * fix(inkless:ci): make the flaky test job report-only so CI is trustable The flaky/quarantine matrix job runs only @Flaky-tagged tests (known-unreliable upstream tests, e.g. the MM2 config-sync integration tests that pass in isolation but flake under CI load). Previously junit.py failed the job on any non-zero Gradle exit code, so a flaked quarantined test turned the whole workflow red even though nothing at the Kafka coverage level was broken - defeating the point of quarantine and making the signal untrustworthy. Make junit.py treat a non-zero Gradle exit code as non-fatal when running the flaky job (RUN_FLAKY=true), while still failing on real infrastructure problems: timeouts (exit 124), thread dumps (hangs), and a missing exit code. Flaky failures are still reported in the job summary tables. The noflaky and new jobs are unaffected and remain the trustable gating signal. Net effect: the workflow is green unless a non-flaky test breaks or infra fails. Verified junit.py exit behavior locally: - flaky + test failures -> exit 0 (green) - noflaky + test failures -> exit 1 (red) - flaky + timeout (124) -> exit 1 (infra, still red) - clean / unset RUN_FLAKY -> unchanged * fixup! test(inkless:ci): quarantine known-flaky upstream integration tests with @flaky Co-authored-by: Cursor <cursoragent@cursor.com>

@flaky

… flaky quarantine) (#700) * fix(inkless:ci): update upstream tests for diskless additions Three upstream Kafka unit tests broke on main because they were not adapted to Inkless changes: - TopicImageNodeTest.testChildPartitionId: PartitionRegistration.toString now includes the diskless fields (classicToDisklessStartOffset, disklessProducerStates, disklessLeaderEpoch); update expected string. - NodeApiVersionsTest.testUnknownApiVersionsToString: the "unknown" apiKey 337 no longer sorts last once the diskless client APIs (500/501) exist, so toString no longer ends with it; use Short.MAX_VALUE instead. - MetricsTest.testGeneralBrokerTopicMetricsAreGreedilyRegistered: all-topics BytesIn/BytesOut are registered for both topicType=classic and diskless. The diskless variants share a metric name with their classic counterparts, so they are extra JMX MBeans but not extra metricMapKeySet() entries; account for the diskless-tagged duplicates in the assertion. Test-only changes; no production code touched. * test(inkless:ci): quarantine known-flaky upstream integration tests with @flaky These integration tests are documented-flaky on upstream Apache Kafka (several with currently-open flaky-test JIRA tickets naming the exact methods/params) and reddened the noflaky-nonew CI bucket independent of any Inkless change. Tag them @flaky so they are excluded from the main suite and run only in the flaky bucket (with retries), while tracking the upstream tickets. MM2 integration family (tagged once on the base methods in MirrorConnectorsIntegrationBaseTest; inherited by IdentityReplication, ExactlyOnce, SSL, Transactions, CustomForwardingAdmin): - testReplicateSourceDefault -> KAFKA-15926 (open; also KAFKA-15927) - testReplicateTargetDefault -> KAFKA-15926 (no dedicated ticket; same harness) - testSyncTopicConfigs -> KAFKA-15945 (also KAFKA-15523, KAFKA-14971) - testReplication -> KAFKA-16488 (also KAFKA-12566) Standalone: - MonitorableSinkIntegrationTest.testMonitorableSinkConnectorAndTask -> KAFKA-18952 - TaskAssignorConvergenceTest.randomClusterPerturbationsShouldConverge -> KAFKA-16491 (open) - ShareConsumerTest.testPollThrowsInterruptExceptionIfInterrupted -> KAFKA-19961 (open) - SaslOAuthBearerSslEndToEndAuthorizationTest.testNoConsumeWithoutDescribeAclViaSubscribe -> KAFKA-9655; overridden solely to tag the OAUTHBEARER variant so other SASL mechanisms keep upstream coverage. build.gradle: add testImplementation test-common-util to :streams and :connect:mirror (source of the @flaky annotation; the other affected modules already depend on it). Verified: compile + checkstyle pass for all touched modules, and the flaky filter excludes the tagged tests from the noflaky bucket (inherited MM2 method and the SASL override both report "No tests found" in default mode). * fix(inkless:ci): make the flaky test job report-only so CI is trustable The flaky/quarantine matrix job runs only @Flaky-tagged tests (known-unreliable upstream tests, e.g. the MM2 config-sync integration tests that pass in isolation but flake under CI load). Previously junit.py failed the job on any non-zero Gradle exit code, so a flaked quarantined test turned the whole workflow red even though nothing at the Kafka coverage level was broken - defeating the point of quarantine and making the signal untrustworthy. Make junit.py treat a non-zero Gradle exit code as non-fatal when running the flaky job (RUN_FLAKY=true), while still failing on real infrastructure problems: timeouts (exit 124), thread dumps (hangs), and a missing exit code. Flaky failures are still reported in the job summary tables. The noflaky and new jobs are unaffected and remain the trustable gating signal. Net effect: the workflow is green unless a non-flaky test breaks or infra fails. Verified junit.py exit behavior locally: - flaky + test failures -> exit 0 (green) - noflaky + test failures -> exit 1 (red) - flaky + timeout (124) -> exit 1 (infra, still red) - clean / unset RUN_FLAKY -> unchanged * fixup! test(inkless:ci): quarantine known-flaky upstream integration tests with @flaky

jeqo requested a review from Copilot July 14, 2026 22:25

Copilot started reviewing on behalf of jeqo July 14, 2026 22:25 View session

Copilot AI reviewed Jul 14, 2026

View reviewed changes

Comment thread core/src/test/scala/unit/kafka/metrics/MetricsTest.scala Outdated

jeqo force-pushed the jeqo/fix-upstream-ci branch from bb94c1d to 1045633 Compare July 15, 2026 08:57

jeqo changed the title ~~fix(inkless:ci): update upstream tests for diskless additions~~ fix(inkless:ci): green the upstream test suite (diskless test fixes + flaky quarantine) Jul 15, 2026

jeqo marked this pull request as ready for review July 16, 2026 07:43

jeqo requested a review from viktorsomogyi July 16, 2026 08:16

viktorsomogyi reviewed Jul 24, 2026

View reviewed changes

jeqo added 4 commits July 29, 2026 18:14

fixup! test(inkless:ci): quarantine known-flaky upstream integration …

6db2a7d

…tests with @flaky

jeqo force-pushed the jeqo/fix-upstream-ci branch from 7607750 to 6db2a7d Compare July 29, 2026 15:43

jeqo requested a review from viktorsomogyi July 29, 2026 15:44

viktorsomogyi approved these changes Jul 30, 2026

View reviewed changes

viktorsomogyi merged commit 86f4bda into main Jul 30, 2026
18 checks passed

viktorsomogyi deleted the jeqo/fix-upstream-ci branch July 30, 2026 09:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(inkless:ci): green the upstream test suite (diskless test fixes + flaky quarantine) - #700

fix(inkless:ci): green the upstream test suite (diskless test fixes + flaky quarantine)#700
viktorsomogyi merged 4 commits into
mainfrom
jeqo/fix-upstream-ci

jeqo commented Jul 14, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

jeqo commented Jul 16, 2026

Uh oh!

viktorsomogyi Jul 24, 2026

Uh oh!

jeqo Jul 29, 2026

Uh oh!

jeqo Jul 29, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

jeqo commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Commit 1 - fix(inkless:ci): update upstream tests for diskless additions

Commit 2 - test(inkless:ci): quarantine known-flaky upstream integration tests with @Flaky

Commit 3 - fix(inkless:ci): make the flaky test job report-only so CI is trustable

Verification

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

jeqo commented Jul 16, 2026

Uh oh!

viktorsomogyi Jul 24, 2026

Choose a reason for hiding this comment

Uh oh!

jeqo Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

jeqo Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jeqo commented Jul 14, 2026 •

edited

Loading

Commit 1 - `fix(inkless:ci): update upstream tests for diskless additions`

Commit 2 - `test(inkless:ci): quarantine known-flaky upstream integration tests with @Flaky`

Commit 3 - `fix(inkless:ci): make the flaky test job report-only so CI is trustable`