Skip to content

[test optimizaton] Use duration buckets for vitest EFD retries#8288

Merged
juan-fernandez merged 1 commit into
masterfrom
juan-fernandez/efd-duration-retries-vitest
May 7, 2026
Merged

[test optimizaton] Use duration buckets for vitest EFD retries#8288
juan-fernandez merged 1 commit into
masterfrom
juan-fernandez/efd-duration-retries-vitest

Conversation

@juan-fernandez
Copy link
Copy Markdown
Collaborator

@juan-fernandez juan-fernandez commented May 6, 2026

What does this PR do?

Updates Vitest Early Flake Detection repeat scheduling to use the slow_test_retries duration buckets returned by the settings API.

Avoids reporting skipped excess EFD repeats once the first execution duration selects a lower retry count, and marks tests with test.early_flake.abort_reason=slow when the selected bucket aborts additional retries.

Motivation

Jest, Mocha, and Cucumber already limit EFD retry attempts based on the duration of the first test execution. Vitest should apply the same API-driven retry bucket behavior so slow tests do not keep scheduling unnecessary EFD retries.

Additional Notes

Stack: 3/5. This is now the base PR for the remaining framework PRs after #8287 merged.

@codecov
Copy link
Copy Markdown

codecov Bot commented May 6, 2026

Codecov Report

❌ Patch coverage is 94.11765% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.74%. Comparing base (8d64c6e) to head (bae88c2).
⚠️ Report is 7 commits behind head on master.

Files with missing lines Patch % Lines
packages/datadog-instrumentations/src/vitest.js 95.23% 3 Missing ⚠️
packages/datadog-plugin-vitest/src/index.js 80.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #8288      +/-   ##
==========================================
+ Coverage   89.63%   89.74%   +0.11%     
==========================================
  Files         836      836              
  Lines       44477    44536      +59     
  Branches     8188     8214      +26     
==========================================
+ Hits        39866    39970     +104     
+ Misses       4611     4566      -45     
Flag Coverage Δ
aiguard-integration-active 40.57% <ø> (ø)
aiguard-integration-latest 40.52% <ø> (ø)
aiguard-integration-maintenance 40.57% <ø> (ø)
aiguard-macos 35.26% <ø> (-0.09%) ⬇️
aiguard-ubuntu 35.36% <ø> (-0.09%) ⬇️
aiguard-windows 35.17% <ø> (-0.09%) ⬇️
apm-capabilities-tracing-macos 49.04% <0.00%> (-0.01%) ⬇️
apm-capabilities-tracing-ubuntu-active 49.07% <0.00%> (-0.19%) ⬇️
apm-capabilities-tracing-ubuntu-latest 49.04% <0.00%> (-0.01%) ⬇️
apm-capabilities-tracing-ubuntu-maintenance 49.07% <0.00%> (-0.19%) ⬇️
apm-capabilities-tracing-ubuntu-oldest 49.06% <0.00%> (-0.19%) ⬇️
apm-capabilities-tracing-windows ?
apm-integrations-aerospike-18-gte.5.2.0 34.25% <ø> (-0.09%) ⬇️
apm-integrations-aerospike-20-gte.5.5.0 34.27% <ø> (-0.09%) ⬇️
apm-integrations-aerospike-22-gte.5.12.1 34.27% <ø> (-0.09%) ⬇️
apm-integrations-aerospike-22-gte.6.0.0 34.27% <ø> (-0.09%) ⬇️
apm-integrations-aerospike-eol- 34.17% <ø> (-0.09%) ⬇️
apm-integrations-child-process 35.28% <ø> (-0.09%) ⬇️
apm-integrations-confluentinc-kafka-javascript-18 41.13% <ø> (-0.09%) ⬇️
apm-integrations-confluentinc-kafka-javascript-20 41.15% <ø> (-0.09%) ⬇️
apm-integrations-confluentinc-kafka-javascript-22 41.15% <ø> (-0.09%) ⬇️
apm-integrations-confluentinc-kafka-javascript-24 41.08% <ø> (-0.09%) ⬇️
apm-integrations-couchbase-18 34.31% <ø> (-0.22%) ⬇️
apm-integrations-couchbase-eol 34.49% <ø> (-0.09%) ⬇️
apm-integrations-dns 34.16% <ø> (-0.09%) ⬇️
apm-integrations-elasticsearch 34.91% <ø> (-0.08%) ⬇️
apm-integrations-http-latest 42.08% <ø> (-0.08%) ⬇️
apm-integrations-http-maintenance 42.14% <ø> (-0.08%) ⬇️
apm-integrations-http-oldest 42.15% <ø> (-0.08%) ⬇️
apm-integrations-http2 39.61% <ø> (-0.09%) ⬇️
apm-integrations-kafkajs-latest 41.03% <ø> (-0.09%) ⬇️
apm-integrations-kafkajs-oldest 41.08% <ø> (-0.09%) ⬇️
apm-integrations-net 34.86% <ø> (-0.09%) ⬇️
apm-integrations-next-11.1.4 29.29% <ø> (-0.08%) ⬇️
apm-integrations-next-13.2.0 30.73% <ø> (-0.08%) ⬇️
apm-integrations-next-gte.10.2.0.and.lt.11 23.10% <ø> (ø)
apm-integrations-next-gte.11.0.0.and.lt.13 30.74% <ø> (-0.08%) ⬇️
apm-integrations-next-gte.13.0.0.and.lt.14 30.99% <ø> (-0.08%) ⬇️
apm-integrations-next-gte.14.0.0.and.lte.14.2.6 30.83% <ø> (-0.06%) ⬇️
apm-integrations-next-gte.14.2.7.and.lt.15 30.80% <ø> (-0.08%) ⬇️
apm-integrations-next-gte.15.0.0 30.87% <ø> (-0.08%) ⬇️
apm-integrations-oracledb 34.54% <ø> (-0.09%) ⬇️
apm-integrations-prisma-18-gte.6.16.0.and.lt.7.0.0 34.87% <ø> (-0.08%) ⬇️
apm-integrations-prisma-latest-all 35.19% <ø> (-0.08%) ⬇️
apm-integrations-restify 36.34% <ø> (-0.09%) ⬇️
apm-integrations-sharedb 33.84% <ø> (-0.09%) ⬇️
apm-integrations-tedious 34.34% <ø> (-0.09%) ⬇️
appsec-express 51.91% <ø> (?)
appsec-fastify 48.60% <ø> (-0.07%) ⬇️
appsec-graphql 48.90% <ø> (-0.07%) ⬇️
appsec-integration-active 37.19% <9.52%> (-0.06%) ⬇️
appsec-integration-latest 37.16% <9.52%> (-0.06%) ⬇️
appsec-integration-maintenance 37.18% <9.52%> (-0.06%) ⬇️
appsec-integration-oldest 37.18% <9.52%> (-0.06%) ⬇️
appsec-kafka 41.42% <ø> (-0.13%) ⬇️
appsec-ldapjs 40.67% <ø> (-0.07%) ⬇️
appsec-lodash 40.78% <ø> (-0.07%) ⬇️
appsec-macos 55.75% <ø> (-0.07%) ⬇️
appsec-mongodb-core 45.04% <ø> (-0.08%) ⬇️
appsec-mongoose 45.94% <ø> (-0.07%) ⬇️
appsec-mysql 47.95% <ø> (-0.07%) ⬇️
appsec-next-latest-11.1.4 29.41% <ø> (-0.07%) ⬇️
appsec-next-latest-13.2.0 30.87% <ø> (-0.09%) ⬇️
appsec-next-latest-gte.10.2.0.and.lt.11 28.71% <ø> (ø)
appsec-next-latest-gte.11.0.0.and.lt.13 30.86% <ø> (-0.09%) ⬇️
appsec-next-latest-gte.13.0.0.and.lt.14 31.10% <ø> (-0.08%) ⬇️
appsec-next-latest-gte.14.0.0.and.lte.14.2.6 30.94% <ø> (-0.09%) ⬇️
appsec-next-latest-gte.14.2.7.and.lt.15 30.94% <ø> (-0.09%) ⬇️
appsec-next-latest-gte.15.0.0 30.94% <ø> (-0.09%) ⬇️
appsec-next-oldest-11.1.4 29.42% <ø> (-0.09%) ⬇️
appsec-next-oldest-13.2.0 31.15% <ø> (-0.09%) ⬇️
appsec-next-oldest-gte.10.2.0.and.lt.11 28.84% <ø> (ø)
appsec-next-oldest-gte.11.0.0.and.lt.13 30.87% <ø> (-0.09%) ⬇️
appsec-next-oldest-gte.13.0.0.and.lt.14 31.34% <ø> (-0.08%) ⬇️
appsec-next-oldest-gte.14.0.0.and.lte.14.2.6 31.19% <ø> (-0.09%) ⬇️
appsec-next-oldest-gte.14.2.7.and.lt.15 31.19% <ø> (-0.09%) ⬇️
appsec-next-oldest-gte.15.0.0 31.19% <ø> (-0.09%) ⬇️
appsec-node-serialize 40.00% <ø> (-0.07%) ⬇️
appsec-passport 43.62% <ø> (-0.08%) ⬇️
appsec-postgres 47.50% <ø> (-0.07%) ⬇️
appsec-sourcing 39.34% <ø> (-0.07%) ⬇️
appsec-stripe 41.40% <ø> (-0.08%) ⬇️
appsec-template 40.16% <ø> (-0.07%) ⬇️
appsec-ubuntu 55.82% <ø> (-0.07%) ⬇️
appsec-windows 55.68% <ø> (-0.07%) ⬇️
debugger-ubuntu-active 43.19% <ø> (ø)
debugger-ubuntu-latest 43.14% <ø> (ø)
debugger-ubuntu-maintenance 43.21% <ø> (ø)
debugger-ubuntu-oldest 43.69% <ø> (ø)
instrumentations-instrumentation-bluebird 29.28% <ø> (-0.09%) ⬇️
instrumentations-instrumentation-body-parser 36.88% <ø> (-0.08%) ⬇️
instrumentations-instrumentation-child_process 34.77% <ø> (-0.09%) ⬇️
instrumentations-instrumentation-cookie-parser 31.10% <ø> (-0.08%) ⬇️
instrumentations-instrumentation-express 31.32% <ø> (-0.08%) ⬇️
instrumentations-instrumentation-express-mongo-sanitize 31.22% <ø> (-0.08%) ⬇️
instrumentations-instrumentation-express-session 36.57% <ø> (-0.08%) ⬇️
instrumentations-instrumentation-fs 28.97% <ø> (-0.09%) ⬇️
instrumentations-instrumentation-generic-pool 29.88% <ø> (ø)
instrumentations-instrumentation-http 36.24% <ø> (-0.09%) ⬇️
instrumentations-instrumentation-knex 29.25% <ø> (-0.09%) ⬇️
instrumentations-instrumentation-light-my-request 36.51% <ø> (-0.08%) ⬇️
instrumentations-instrumentation-mongoose 30.31% <ø> (-0.08%) ⬇️
instrumentations-instrumentation-multer 36.67% <ø> (-0.08%) ⬇️
instrumentations-instrumentation-mysql2 34.76% <ø> (-0.09%) ⬇️
instrumentations-instrumentation-passport 40.25% <ø> (-0.08%) ⬇️
instrumentations-instrumentation-passport-http 40.04% <ø> (-0.08%) ⬇️
instrumentations-instrumentation-passport-local 40.53% <ø> (-0.08%) ⬇️
instrumentations-instrumentation-pg 34.31% <ø> (-0.09%) ⬇️
instrumentations-instrumentation-promise 29.22% <ø> (-0.09%) ⬇️
instrumentations-instrumentation-promise-js 29.22% <ø> (-0.09%) ⬇️
instrumentations-instrumentation-q 29.25% <ø> (-0.09%) ⬇️
instrumentations-instrumentation-url 29.22% <ø> (-0.09%) ⬇️
instrumentations-instrumentation-when 29.23% <ø> (-0.09%) ⬇️
instrumentations-integration-esbuild-active 19.47% <9.52%> (-0.03%) ⬇️
instrumentations-integration-esbuild-latest 19.45% <9.52%> (-0.03%) ⬇️
instrumentations-integration-esbuild-maintenance 19.47% <9.52%> (-0.03%) ⬇️
instrumentations-integration-esbuild-oldest 19.46% <9.52%> (-0.03%) ⬇️
llmobs-ai 37.72% <ø> (-0.08%) ⬇️
llmobs-anthropic 37.77% <ø> (-0.08%) ⬇️
llmobs-bedrock 36.48% <ø> (-0.08%) ⬇️
llmobs-google-genai 36.79% <ø> (-0.08%) ⬇️
llmobs-langchain 36.46% <ø> (-0.07%) ⬇️
llmobs-openai 40.54% <ø> (-0.08%) ⬇️
llmobs-sdk-active 45.35% <ø> (-0.09%) ⬇️
llmobs-sdk-latest 45.29% <ø> (-0.09%) ⬇️
llmobs-sdk-maintenance 45.35% <ø> (-0.09%) ⬇️
llmobs-sdk-oldest 45.34% <ø> (-0.09%) ⬇️
llmobs-vertex-ai 36.95% <ø> (-0.08%) ⬇️
openfeature-macos 37.94% <ø> (+0.05%) ⬆️
openfeature-ubuntu 38.02% <ø> (ø)
openfeature-unit-active 48.62% <ø> (ø)
openfeature-unit-latest 48.46% <ø> (ø)
openfeature-unit-maintenance 48.62% <ø> (ø)
openfeature-unit-oldest 48.62% <ø> (ø)
openfeature-windows 37.82% <ø> (+0.05%) ⬆️
platform-core 37.45% <ø> (ø)
platform-esbuild 40.80% <ø> (ø)
platform-instrumentations-misc 30.71% <ø> (ø)
platform-integration-active 46.94% <ø> (-0.01%) ⬇️
platform-integration-latest 46.91% <ø> (+<0.01%) ⬆️
platform-integration-maintenance 46.96% <ø> (ø)
platform-integration-oldest 47.12% <ø> (ø)
platform-shimmer 42.46% <ø> (ø)
platform-unit-guardrails 36.24% <ø> (ø)
platform-webpack 20.57% <11.53%> (-0.02%) ⬇️
plugins-azure-durable-functions 36.78% <ø> (ø)
plugins-azure-event-hubs 34.79% <ø> (ø)
plugins-azure-service-bus 35.24% <ø> (ø)
plugins-bullmq 40.05% <ø> (-0.20%) ⬇️
plugins-cassandra 34.55% <ø> (-0.21%) ⬇️
plugins-cookie 26.66% <ø> (ø)
plugins-cookie-parser 26.47% <ø> (ø)
plugins-crypto 27.72% <ø> (ø)
plugins-dd-trace-api 34.70% <ø> (-0.09%) ⬇️
plugins-express-mongo-sanitize 26.60% <ø> (ø)
plugins-express-session 26.43% <ø> (ø)
plugins-fastify 38.66% <ø> (-0.08%) ⬇️
plugins-fetch 34.99% <ø> (-0.08%) ⬇️
plugins-fs 35.05% <ø> (-0.09%) ⬇️
plugins-generic-pool 25.58% <ø> (ø)
plugins-google-cloud-pubsub 42.37% <ø> (-0.08%) ⬇️
plugins-grpc 37.41% <ø> (-0.08%) ⬇️
plugins-handlebars 26.64% <ø> (ø)
plugins-hapi 36.55% <ø> (-0.09%) ⬇️
plugins-hono 36.83% <ø> (-0.08%) ⬇️
plugins-ioredis 35.14% <ø> (-0.09%) ⬇️
plugins-knex 26.32% <ø> (ø)
plugins-langgraph 34.41% <ø> (-0.09%) ⬇️
plugins-ldapjs 24.20% <ø> (ø)
plugins-light-my-request 26.06% <ø> (ø)
plugins-limitd-client 29.52% <ø> (-0.15%) ⬇️
plugins-lodash 25.66% <ø> (ø)
plugins-mariadb 36.04% <ø> (-0.03%) ⬇️
plugins-memcached 34.79% <ø> (-0.09%) ⬇️
plugins-microgateway-core 35.67% <ø> (-0.09%) ⬇️
plugins-modelcontextprotocol-sdk 33.69% <ø> (-0.09%) ⬇️
plugins-moleculer 37.50% <ø> (-0.09%) ⬇️
plugins-mongodb 35.93% <ø> (-0.08%) ⬇️
plugins-mongodb-core 35.59% <ø> (-0.06%) ⬇️
plugins-mongoose 35.57% <ø> (+0.02%) ⬆️
plugins-multer 26.43% <ø> (ø)
plugins-mysql 35.52% <ø> (-0.22%) ⬇️
plugins-mysql2 35.81% <ø> (-0.09%) ⬇️
plugins-node-serialize 26.70% <ø> (ø)
plugins-opensearch 34.42% <ø> (-0.09%) ⬇️
plugins-passport-http 26.48% <ø> (ø)
plugins-pino 31.24% <ø> (-0.09%) ⬇️
plugins-postgres 33.75% <ø> (-0.08%) ⬇️
plugins-process 27.72% <ø> (ø)
plugins-pug 26.66% <ø> (ø)
plugins-redis 35.14% <ø> (-0.09%) ⬇️
plugins-router 39.00% <ø> (-0.08%) ⬇️
plugins-sequelize 25.36% <ø> (ø)
plugins-test-and-upstream-amqp10 35.03% <ø> (+0.05%) ⬆️
plugins-test-and-upstream-amqplib 40.27% <ø> (-0.06%) ⬇️
plugins-test-and-upstream-apollo 35.88% <ø> (-0.08%) ⬇️
plugins-test-and-upstream-avsc 34.37% <ø> (-0.09%) ⬇️
plugins-test-and-upstream-bunyan 30.60% <ø> (-0.09%) ⬇️
plugins-test-and-upstream-connect 37.21% <ø> (-0.09%) ⬇️
plugins-test-and-upstream-graphql 36.82% <ø> (-0.06%) ⬇️
plugins-test-and-upstream-koa 36.81% <ø> (-0.08%) ⬇️
plugins-test-and-upstream-protobufjs 34.58% <ø> (-0.09%) ⬇️
plugins-test-and-upstream-rhea 40.29% <ø> (-0.09%) ⬇️
plugins-undici 35.71% <ø> (-0.08%) ⬇️
plugins-url 27.72% <ø> (ø)
plugins-valkey 34.80% <ø> (-0.09%) ⬇️
plugins-vm 27.72% <ø> (ø)
plugins-winston 31.10% <ø> (-0.22%) ⬇️
plugins-ws 38.20% <ø> (-0.22%) ⬇️
profiling-macos 42.74% <ø> (-0.07%) ⬇️
profiling-ubuntu 43.15% <ø> (-0.07%) ⬇️
profiling-windows 39.97% <ø> (-0.07%) ⬇️
serverless-aws-sdk-latest-aws-sdk 34.76% <ø> (-0.07%) ⬇️
serverless-aws-sdk-latest-bedrockruntime 33.28% <ø> (-0.08%) ⬇️
serverless-aws-sdk-latest-client 22.37% <ø> (ø)
serverless-aws-sdk-latest-dynamodb 35.71% <ø> (?)
serverless-aws-sdk-latest-eventbridge 28.78% <ø> (-0.08%) ⬇️
serverless-aws-sdk-latest-kinesis 38.64% <ø> (-0.08%) ⬇️
serverless-aws-sdk-latest-lambda 35.91% <ø> (-0.08%) ⬇️
serverless-aws-sdk-latest-s3 33.84% <ø> (-0.08%) ⬇️
serverless-aws-sdk-latest-serverless-peer-service 40.03% <ø> (-0.09%) ⬇️
serverless-aws-sdk-latest-sns 39.89% <ø> (-0.08%) ⬇️
serverless-aws-sdk-latest-sqs 39.04% <ø> (-0.08%) ⬇️
serverless-aws-sdk-latest-stepfunctions 34.47% <ø> (-0.08%) ⬇️
serverless-aws-sdk-latest-util 47.86% <ø> (ø)
serverless-aws-sdk-oldest-aws-sdk 34.81% <ø> (-0.07%) ⬇️
serverless-aws-sdk-oldest-bedrockruntime 33.32% <ø> (-0.08%) ⬇️
serverless-aws-sdk-oldest-client 22.74% <ø> (ø)
serverless-aws-sdk-oldest-dynamodb 35.76% <ø> (-0.08%) ⬇️
serverless-aws-sdk-oldest-eventbridge 28.82% <ø> (-0.08%) ⬇️
serverless-aws-sdk-oldest-kinesis 38.76% <ø> (-0.08%) ⬇️
serverless-aws-sdk-oldest-lambda 35.95% <ø> (-0.20%) ⬇️
serverless-aws-sdk-oldest-s3 33.90% <ø> (-0.08%) ⬇️
serverless-aws-sdk-oldest-serverless-peer-service 40.10% <ø> (-0.09%) ⬇️
serverless-aws-sdk-oldest-sns 40.07% <ø> (-0.08%) ⬇️
serverless-aws-sdk-oldest-sqs 39.08% <ø> (-0.08%) ⬇️
serverless-aws-sdk-oldest-stepfunctions 34.52% <ø> (-0.08%) ⬇️
serverless-aws-sdk-oldest-util 48.18% <ø> (ø)
serverless-azure-functions-eventhubs 38.43% <ø> (ø)
serverless-azure-functions-servicebus 38.49% <ø> (ø)
serverless-lambda 32.80% <ø> (-0.08%) ⬇️
test-optimization-cucumber-latest-7.0.0 50.22% <ø> (+0.09%) ⬆️
test-optimization-cucumber-latest-latest 52.94% <ø> (+0.12%) ⬆️
test-optimization-cucumber-oldest-7.0.0 50.25% <ø> (+0.12%) ⬆️
test-optimization-cypress-latest-12.0.0-commonJS 48.24% <ø> (+0.10%) ⬆️
test-optimization-cypress-latest-12.0.0-esm 48.25% <ø> (+0.07%) ⬆️
test-optimization-cypress-latest-14.5.4-commonJS 48.07% <ø> (+0.11%) ⬆️
test-optimization-cypress-latest-14.5.4-esm 48.12% <ø> (+0.10%) ⬆️
test-optimization-cypress-latest-latest-commonJS 48.57% <ø> (+0.07%) ⬆️
test-optimization-cypress-latest-latest-esm 48.60% <ø> (+0.07%) ⬆️
test-optimization-cypress-oldest-12.0.0-commonJS 48.26% <ø> (+0.07%) ⬆️
test-optimization-cypress-oldest-12.0.0-esm 48.29% <ø> (+0.07%) ⬆️
test-optimization-cypress-oldest-14.5.4-commonJS 48.11% <ø> (+0.07%) ⬆️
test-optimization-cypress-oldest-14.5.4-esm 48.14% <ø> (+0.07%) ⬆️
test-optimization-jest-latest-latest 54.36% <ø> (+0.08%) ⬆️
test-optimization-jest-latest-oldest 53.53% <ø> (+0.08%) ⬆️
test-optimization-jest-oldest-latest 54.37% <ø> (+0.08%) ⬆️
test-optimization-jest-oldest-oldest 53.46% <ø> (+0.08%) ⬆️
test-optimization-mocha-latest-latest 53.18% <ø> (+0.08%) ⬆️
test-optimization-mocha-latest-oldest 50.80% <ø> (+0.08%) ⬆️
test-optimization-mocha-oldest-latest 53.27% <ø> (+0.11%) ⬆️
test-optimization-mocha-oldest-oldest 50.74% <ø> (+0.08%) ⬆️
test-optimization-playwright-latest-latest-playwright-active-test-span 43.66% <ø> (+0.28%) ⬆️
test-optimization-playwright-latest-latest-playwright-atr 42.41% <ø> (+0.11%) ⬆️
test-optimization-playwright-latest-latest-playwright-efd 42.67% <ø> (+0.09%) ⬆️
test-optimization-playwright-latest-latest-playwright-final-status 42.90% <ø> (+0.11%) ⬆️
test-optimization-playwright-latest-latest-playwright-impacted-tests 42.30% <ø> (ø)
test-optimization-playwright-latest-latest-playwright-reporting 42.45% <ø> (+0.09%) ⬆️
test-optimization-playwright-latest-latest-playwright-test-management 44.39% <ø> (+0.10%) ⬆️
test-optimization-playwright-latest-oldest-playwright-active-test-span 43.69% <ø> (+0.28%) ⬆️
test-optimization-playwright-latest-oldest-playwright-atr 42.59% <ø> (+0.11%) ⬆️
test-optimization-playwright-latest-oldest-playwright-efd 42.69% <ø> (+0.09%) ⬆️
test-optimization-playwright-latest-oldest-playwright-final-status 42.90% <ø> (+0.09%) ⬆️
test-optimization-playwright-latest-oldest-playwright-impacted-tests 42.32% <ø> (ø)
test-optimization-playwright-latest-oldest-playwright-reporting 42.49% <ø> (+0.09%) ⬆️
test-optimization-playwright-latest-oldest-playwright-test-management 44.44% <ø> (+0.11%) ⬆️
test-optimization-playwright-oldest-latest-playwright-active-test-span 43.69% <ø> (+0.28%) ⬆️
test-optimization-playwright-oldest-latest-playwright-atr 42.44% <ø> (+0.11%) ⬆️
test-optimization-playwright-oldest-latest-playwright-efd ?
test-optimization-playwright-oldest-latest-playwright-final-status 42.91% <ø> (+0.11%) ⬆️
test-optimization-playwright-oldest-latest-playwright-impacted-tests 42.35% <ø> (ø)
test-optimization-playwright-oldest-latest-playwright-reporting 42.46% <ø> (+0.09%) ⬆️
test-optimization-playwright-oldest-latest-playwright-test-management 44.40% <ø> (+0.10%) ⬆️
test-optimization-playwright-oldest-oldest-playwright-active-test-span 43.73% <ø> (+0.28%) ⬆️
test-optimization-playwright-oldest-oldest-playwright-atr 42.63% <ø> (+0.11%) ⬆️
test-optimization-playwright-oldest-oldest-playwright-efd 42.70% <ø> (+0.09%) ⬆️
test-optimization-playwright-oldest-oldest-playwright-final-status 42.94% <ø> (+0.11%) ⬆️
test-optimization-playwright-oldest-oldest-playwright-impacted-tests 42.36% <ø> (ø)
test-optimization-playwright-oldest-oldest-playwright-reporting 42.50% <ø> (+0.09%) ⬆️
test-optimization-playwright-oldest-oldest-playwright-test-management 44.45% <ø> (+0.11%) ⬆️
test-optimization-selenium-latest 45.47% <ø> (+0.30%) ⬆️
test-optimization-selenium-oldest 44.95% <ø> (+0.07%) ⬆️
test-optimization-testopt-active 46.98% <11.76%> (+0.03%) ⬆️
test-optimization-testopt-latest 46.94% <11.76%> (+0.03%) ⬆️
test-optimization-testopt-maintenance 46.98% <11.76%> (+0.03%) ⬆️
test-optimization-testopt-oldest 47.87% <ø> (?)
test-optimization-vitest-latest 50.91% <94.11%> (+0.30%) ⬆️
test-optimization-vitest-oldest 47.64% <94.11%> (+0.57%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 6, 2026

Overall package size

Self size: 5.78 MB
Deduped: 6.62 MB
No deduping: 6.62 MB

Dependency sizes | name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.0.1 | 82.56 kB | 817.39 kB | | dc-polyfill | 0.1.10 | 26.73 kB | 26.73 kB |

🤖 This report was automatically generated by heaviest-objects-in-the-universe

@datadog-official
Copy link
Copy Markdown

datadog-official Bot commented May 6, 2026

Tests

Fix all issues with BitsAI or with Cursor

⚠️ Warnings

🧪 3 Tests failed

Plugin kafkajs with kafkajs >=1.4 (1.4.0) data stream monitoring "before each" hook for "Should set a message payload size when consuming a message" from checkpoints   View in Datadog   (Fix with Cursor)
This server does not host this topic-partition

KafkaJSProtocolError: This server does not host this topic-partition
    at createErrorFromCode (versions/kafkajs@1.4.0/node_modules/kafkajs/src/protocol/error.js:472:10)
    at Object.parse (versions/kafkajs@1.4.0/node_modules/kafkajs/src/protocol/requests/metadata/v0/response.js:56:11)
    at Connection.send (versions/kafkajs@1.4.0/node_modules/kafkajs/src/network/connection.js:250:35)
    at process.processTicksAndRejections (node:internal/process/task_queues:104:5)
    at async Broker.metadata (versions/kafkajs@1.4.0/node_modules/kafkajs/src/broker/index.js:144:12)
    at async retryOnLeaderNotAvailable.delay (versions/kafkajs@1.4.0/node_modules/kafkajs/src/admin/index.js:98:55)
    at async callback (versions/kafkajs@1.4.0/node_modules/kafkajs/src/admin/index.js:20:14)
...
Plugin kafkajs with kafkajs >=1.4 (2.2.4) data stream monitoring "before each" hook for "Should set a checkpoint on produce" from checkpoints   View in Datadog   (Fix with Cursor)
This server does not host this topic-partition

KafkaJSProtocolError: This server does not host this topic-partition
    at createErrorFromCode (versions/kafkajs@>=1.4/node_modules/kafkajs/src/protocol/error.js:581:10)
    at Object.parse (versions/kafkajs@>=1.4/node_modules/kafkajs/src/protocol/requests/metadata/v0/response.js:55:11)
    at Connection.send (versions/kafkajs@>=1.4/node_modules/kafkajs/src/network/connection.js:433:35)
    at process.processTicksAndRejections (node:internal/process/task_queues:104:5)
    at async [private:Broker:sendRequest] (versions/kafkajs@>=1.4/node_modules/kafkajs/src/broker/index.js:904:14)
    at async Broker.metadata (versions/kafkajs@>=1.4/node_modules/kafkajs/src/broker/index.js:177:12)
    at async retryOnLeaderNotAvailable.delay (versions/kafkajs@>=1.4/node_modules/kafkajs/src/admin/index.js:175:55)
...
Plugin kafkajs with kafkajs >=1.4 (1.4.0) without configuration "before each" hook for "should be instrumented" from producer   View in Datadog   (Fix with Cursor)
This server does not host this topic-partition

KafkaJSProtocolError: This server does not host this topic-partition
    at createErrorFromCode (versions/kafkajs@1.4.0/node_modules/kafkajs/src/protocol/error.js:472:10)
    at Object.parse (versions/kafkajs@1.4.0/node_modules/kafkajs/src/protocol/requests/metadata/v0/response.js:56:11)
    at Connection.send (versions/kafkajs@1.4.0/node_modules/kafkajs/src/network/connection.js:250:35)
    at process.processTicksAndRejections (node:internal/process/task_queues:104:5)
    at async Broker.metadata (versions/kafkajs@1.4.0/node_modules/kafkajs/src/broker/index.js:144:12)
    at async retryOnLeaderNotAvailable.delay (versions/kafkajs@1.4.0/node_modules/kafkajs/src/admin/index.js:98:55)
    at async callback (versions/kafkajs@1.4.0/node_modules/kafkajs/src/admin/index.js:20:14)
...

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

🎯 Code Coverage (details)
Patch Coverage: 74.60%
Overall Coverage: 86.58% (-0.02%)

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: bae88c2 | Docs | Datadog PR Page | Give us feedback!

@juan-fernandez juan-fernandez force-pushed the juan-fernandez/efd-duration-retries-vitest branch from 5ee02f1 to 20768e7 Compare May 6, 2026 13:04
@pr-commenter
Copy link
Copy Markdown

pr-commenter Bot commented May 6, 2026

Benchmarks

Benchmark execution time: 2026-05-07 18:42:40

Comparing candidate commit bae88c2 in PR branch juan-fernandez/efd-duration-retries-vitest with baseline commit 8d64c6e in branch master.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 1749 metrics, 95 unstable metrics.

Base automatically changed from juan-fernandez/efd-duration-retries-cucumber to master May 7, 2026 15:47
@juan-fernandez juan-fernandez marked this pull request as ready for review May 7, 2026 15:51
@juan-fernandez juan-fernandez requested a review from a team as a code owner May 7, 2026 15:51
@juan-fernandez juan-fernandez changed the title fix(ci-visibility): use duration buckets for vitest EFD retries [test optimizaton] Use duration buckets for vitest EFD retries May 7, 2026
@juan-fernandez juan-fernandez force-pushed the juan-fernandez/efd-duration-retries-vitest branch from d4fc7a8 to bae88c2 Compare May 7, 2026 15:52
@juan-fernandez juan-fernandez enabled auto-merge (squash) May 7, 2026 15:52
@chatgpt-codex-connector
Copy link
Copy Markdown

💡 Codex Review

const firstExecutionDurationMs = testCaseStartedAt === undefined ? 0 : performance.now() - testCaseStartedAt
efdRetryCount = status === 'skip'
? 0
: getEfdRetryCount(firstExecutionDurationMs, earlyFlakeDetectionSlowTestRetries)

P2 Badge Avoid choosing Cucumber retries from coordinator timing

When Cucumber runs in parallel, this code executes in the coordinator and measures the time between receiving testCaseStarted and testCaseFinished messages, not the worker's actual test duration. The worker independently chooses how many EFD retries to run from its local duration in getWrappedRunTestCase, so for tests close to a bucket boundary (for example a 4.99s worker execution with {'5s': 2, '10s': 0} plus IPC/scheduling delay) the coordinator can choose 0 retries and mark the suite finished after the first result while the worker is still emitting retry results. This can produce premature or incorrect suite status in parallel Cucumber runs; the coordinator should use the worker's chosen retry count or avoid recomputing bucket decisions from IPC timing.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@juan-fernandez juan-fernandez merged commit d422180 into master May 7, 2026
1003 of 1004 checks passed
@juan-fernandez juan-fernandez deleted the juan-fernandez/efd-duration-retries-vitest branch May 7, 2026 18:45
@dd-octo-sts dd-octo-sts Bot mentioned this pull request May 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants