Skip to content

[test optimization] Prevent Jest --bail from stopping on quarantined test failures#8015

Merged
juan-fernandez merged 1 commit into
masterfrom
juan-fernandez/fix-jest-bail-quarantine
Apr 16, 2026
Merged

[test optimization] Prevent Jest --bail from stopping on quarantined test failures#8015
juan-fernandez merged 1 commit into
masterfrom
juan-fernandez/fix-jest-bail-quarantine

Conversation

@juan-fernandez
Copy link
Copy Markdown
Collaborator

@juan-fernandez juan-fernandez commented Apr 16, 2026

What does this PR do?

Suppresses event.test.errors for quarantined tests at test_done time so Jest doesn't count them as failures. This prevents --bail from stopping the test run when only quarantined tests fail.

Errors are only suppressed on the final execution to avoid interfering with ATR/ATF retries. For ATF/EFD tests (which use atrSuppressedErrors), errors are skipped during restoration at run_finish if the test is quarantined.

Quarantine suppression info is sent from workers to the main process via IPC (new payload code JEST_WORKER_QUARANTINE_PAYLOAD_CODE) so the session summary remains accurate in parallel mode.

Motivation

Jest's --bail flag stops execution on the first test failure. Quarantined tests that fail left event.test.errors intact, so bail would trigger even though the failure was expected to be ignored. The exit code flip at session level happened too late to prevent this.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Apr 16, 2026

Overall package size

Self size: 5.5 MB
Deduped: 6.34 MB
No deduping: 6.34 MB

Dependency sizes | name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.0.1 | 82.56 kB | 817.39 kB | | dc-polyfill | 0.1.10 | 26.73 kB | 26.73 kB |

🤖 This report was automatically generated by heaviest-objects-in-the-universe

@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 16, 2026

Codecov Report

❌ Patch coverage is 4.87805% with 39 lines in your changes missing coverage. Please review.
✅ Project coverage is 73.59%. Comparing base (b5ded05) to head (d844bdd).
⚠️ Report is 21 commits behind head on master.

Files with missing lines Patch % Lines
packages/datadog-instrumentations/src/jest.js 2.50% 39 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #8015      +/-   ##
==========================================
- Coverage   73.83%   73.59%   -0.24%     
==========================================
  Files         773      773              
  Lines       35983    36083     +100     
==========================================
- Hits        26567    26557      -10     
- Misses       9416     9526     +110     
Flag Coverage Δ
aiguard-macos 36.53% <100.00%> (+0.01%) ⬆️
aiguard-ubuntu 36.64% <100.00%> (+0.01%) ⬆️
aiguard-windows 36.44% <100.00%> (+0.01%) ⬆️
apm-capabilities-tracing-macos 48.17% <100.00%> (-0.25%) ⬇️
apm-capabilities-tracing-ubuntu 48.21% <100.00%> (-0.25%) ⬇️
apm-capabilities-tracing-windows 48.00% <100.00%> (-0.25%) ⬇️
apm-integrations-child-process 36.16% <100.00%> (+<0.01%) ⬆️
apm-integrations-couchbase-18 35.18% <100.00%> (+0.08%) ⬆️
apm-integrations-couchbase-eol 35.23% <100.00%> (+0.08%) ⬆️
apm-integrations-oracledb 35.20% <100.00%> (+0.08%) ⬆️
appsec-express 52.89% <100.00%> (+<0.01%) ⬆️
appsec-fastify 49.38% <100.00%> (+0.04%) ⬆️
appsec-graphql 49.63% <100.00%> (-0.06%) ⬇️
appsec-kafka 42.17% <100.00%> (+<0.01%) ⬆️
appsec-ldapjs 41.48% <100.00%> (+0.02%) ⬆️
appsec-lodash 41.51% <100.00%> (+0.02%) ⬆️
appsec-macos 56.82% <100.00%> (+0.01%) ⬆️
appsec-mongodb-core 46.14% <100.00%> (+0.04%) ⬆️
appsec-mongoose 46.71% <100.00%> (+0.04%) ⬆️
appsec-mysql 48.86% <100.00%> (+0.06%) ⬆️
appsec-node-serialize 40.69% <100.00%> (+0.02%) ⬆️
appsec-passport 44.68% <100.00%> (+0.04%) ⬆️
appsec-postgres 48.44% <100.00%> (+0.04%) ⬆️
appsec-sourcing 40.18% <100.00%> (+0.05%) ⬆️
appsec-stripe 42.40% <100.00%> (+0.02%) ⬆️
appsec-template 40.85% <100.00%> (+0.02%) ⬆️
appsec-ubuntu 56.90% <100.00%> (+0.01%) ⬆️
appsec-windows 56.71% <100.00%> (+0.01%) ⬆️
instrumentations-instrumentation-bluebird 29.85% <100.00%> (-0.03%) ⬇️
instrumentations-instrumentation-body-parser 37.76% <100.00%> (+0.02%) ⬆️
instrumentations-instrumentation-child_process 35.54% <100.00%> (+<0.01%) ⬆️
instrumentations-instrumentation-cookie-parser 31.77% <100.00%> (-0.04%) ⬇️
instrumentations-instrumentation-express 31.99% <100.00%> (-0.04%) ⬇️
instrumentations-instrumentation-express-mongo-sanitize 31.90% <100.00%> (-0.04%) ⬇️
instrumentations-instrumentation-express-session 37.39% <100.00%> (+0.03%) ⬆️
instrumentations-instrumentation-fs 29.52% <100.00%> (-0.03%) ⬇️
instrumentations-instrumentation-generic-pool 30.99% <100.00%> (-0.15%) ⬇️
instrumentations-instrumentation-http 36.99% <100.00%> (+0.02%) ⬆️
instrumentations-instrumentation-knex 29.82% <100.00%> (-0.03%) ⬇️
instrumentations-instrumentation-mongoose 30.93% <100.00%> (-0.04%) ⬇️
instrumentations-instrumentation-multer 37.53% <100.00%> (+0.03%) ⬆️
instrumentations-instrumentation-mysql2 35.51% <100.00%> (+0.04%) ⬆️
instrumentations-instrumentation-passport 41.27% <100.00%> (+0.02%) ⬆️
instrumentations-instrumentation-passport-http 40.96% <100.00%> (+0.02%) ⬆️
instrumentations-instrumentation-passport-local 41.46% <100.00%> (+0.02%) ⬆️
instrumentations-instrumentation-pg 35.03% <100.00%> (+0.04%) ⬆️
instrumentations-instrumentation-promise 29.78% <100.00%> (-0.03%) ⬇️
instrumentations-instrumentation-promise-js 29.79% <100.00%> (-0.03%) ⬇️
instrumentations-instrumentation-q 29.82% <100.00%> (-0.03%) ⬇️
instrumentations-instrumentation-url 29.78% <100.00%> (-0.03%) ⬇️
instrumentations-instrumentation-when 29.80% <100.00%> (-0.03%) ⬇️
llmobs-ai 38.47% <100.00%> (+<0.01%) ⬆️
llmobs-anthropic 37.92% <100.00%> (-0.01%) ⬇️
llmobs-bedrock 37.17% <100.00%> (-0.01%) ⬇️
llmobs-google-genai 37.61% <100.00%> (-0.01%) ⬇️
llmobs-langchain 37.05% <100.00%> (-0.12%) ⬇️
llmobs-openai 41.29% <100.00%> (-0.01%) ⬇️
llmobs-vertex-ai 37.78% <100.00%> (-0.01%) ⬇️
platform-core 30.01% <ø> (-1.09%) ⬇️
platform-esbuild 32.83% <ø> (-1.14%) ⬇️
platform-instrumentations-misc 40.24% <ø> (-0.73%) ⬇️
platform-shimmer 35.73% <ø> (-1.27%) ⬇️
platform-unit-guardrails 31.38% <ø> (-1.11%) ⬇️
platform-webpack 20.83% <4.87%> (-0.02%) ⬇️
plugins-azure-durable-functions 25.21% <ø> (-0.66%) ⬇️
plugins-azure-event-hubs 25.36% <ø> (-0.67%) ⬇️
plugins-azure-service-bus 24.78% <ø> (-0.63%) ⬇️
plugins-bullmq 40.81% <100.00%> (+<0.01%) ⬆️
plugins-cassandra 35.34% <100.00%> (+0.08%) ⬆️
plugins-cookie 26.33% <ø> (-0.73%) ⬇️
plugins-cookie-parser 26.14% <ø> (-0.71%) ⬇️
plugins-crypto 25.66% <ø> (-0.90%) ⬇️
plugins-dd-trace-api 35.47% <100.00%> (-0.04%) ⬇️
plugins-express-mongo-sanitize 26.28% <ø> (-0.71%) ⬇️
plugins-express-session 26.10% <ø> (-0.71%) ⬇️
plugins-fastify 39.44% <100.00%> (+<0.01%) ⬆️
plugins-fetch 35.85% <100.00%> (+0.05%) ⬆️
plugins-fs 35.76% <100.00%> (-0.01%) ⬇️
plugins-generic-pool 25.25% <ø> (-0.67%) ⬇️
plugins-google-cloud-pubsub 43.12% <100.00%> (+0.02%) ⬆️
plugins-grpc 38.12% <100.00%> (+0.01%) ⬆️
plugins-handlebars 26.32% <ø> (-0.72%) ⬇️
plugins-hapi 37.36% <100.00%> (+0.03%) ⬆️
plugins-hono 37.60% <100.00%> (-0.12%) ⬇️
plugins-ioredis 35.80% <100.00%> (+0.10%) ⬆️
plugins-knex 26.01% <ø> (-0.67%) ⬇️
plugins-langgraph 35.15% <100.00%> (+<0.01%) ⬆️
plugins-ldapjs 23.88% <ø> (-0.58%) ⬇️
plugins-light-my-request 25.75% <ø> (-0.67%) ⬇️
plugins-limitd-client 30.07% <100.00%> (-0.03%) ⬇️
plugins-lodash 25.33% <ø> (-0.68%) ⬇️
plugins-mariadb 36.67% <100.00%> (+0.06%) ⬆️
plugins-memcached 35.45% <100.00%> (+0.09%) ⬆️
plugins-microgateway-core 36.44% <100.00%> (+0.04%) ⬆️
plugins-moleculer 38.15% <100.00%> (+0.03%) ⬆️
plugins-mongodb 36.54% <100.00%> (+0.08%) ⬆️
plugins-mongodb-core 36.26% <100.00%> (+0.09%) ⬆️
plugins-mongoose 36.13% <100.00%> (-0.06%) ⬇️
plugins-multer 26.10% <ø> (-0.71%) ⬇️
plugins-mysql 36.54% <100.00%> (+0.09%) ⬆️
plugins-mysql2 36.52% <100.00%> (+0.09%) ⬆️
plugins-node-serialize 26.37% <ø> (-0.73%) ⬇️
plugins-opensearch 35.10% <100.00%> (+0.08%) ⬆️
plugins-passport-http 26.16% <ø> (-0.70%) ⬇️
plugins-postgres 34.49% <100.00%> (+0.06%) ⬆️
plugins-process 25.66% <ø> (-0.90%) ⬇️
plugins-pug 26.33% <ø> (-0.73%) ⬇️
plugins-redis 35.96% <100.00%> (+0.05%) ⬆️
plugins-router 40.01% <100.00%> (-0.05%) ⬇️
plugins-sequelize 25.03% <ø> (-0.66%) ⬇️
plugins-test-and-upstream-amqp10 35.77% <100.00%> (+0.04%) ⬆️
plugins-test-and-upstream-amqplib 40.96% <100.00%> (+0.03%) ⬆️
plugins-test-and-upstream-apollo 36.62% <100.00%> (+0.03%) ⬆️
plugins-test-and-upstream-avsc 35.62% <100.00%> (+<0.01%) ⬆️
plugins-test-and-upstream-bunyan 31.22% <100.00%> (-0.04%) ⬇️
plugins-test-and-upstream-connect 37.95% <100.00%> (+0.03%) ⬆️
plugins-test-and-upstream-graphql 37.30% <100.00%> (+0.04%) ⬆️
plugins-test-and-upstream-koa 37.56% <100.00%> (+0.03%) ⬆️
plugins-test-and-upstream-protobufjs 35.83% <100.00%> (+<0.01%) ⬆️
plugins-test-and-upstream-rhea 41.07% <100.00%> (+0.03%) ⬆️
plugins-undici 36.63% <100.00%> (+0.02%) ⬆️
plugins-url 25.66% <ø> (-0.90%) ⬇️
plugins-valkey 35.47% <100.00%> (+0.09%) ⬆️
plugins-vm 25.66% <ø> (-0.90%) ⬇️
plugins-winston 31.65% <100.00%> (-0.04%) ⬇️
plugins-ws 39.08% <100.00%> (+0.07%) ⬆️
profiling-macos 38.03% <100.00%> (+<0.01%) ⬆️
profiling-ubuntu 38.65% <100.00%> (+0.46%) ⬆️
profiling-windows 39.55% <100.00%> (+<0.01%) ⬆️
serverless-azure-functions-client 25.10% <ø> (-0.65%) ⬇️
serverless-azure-functions-eventhubs 25.10% <ø> (-0.65%) ⬇️
serverless-azure-functions-servicebus 25.10% <ø> (-0.65%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@datadog-datadog-prod-us1
Copy link
Copy Markdown

datadog-datadog-prod-us1 Bot commented Apr 16, 2026

Tests

Fix all issues with BitsAI or with Cursor

⚠️ Other Violations

🧪 1 Test failed

Instrumentation when with when >=3 (3.0.0) "before all" hook for "should run the then() callbacks in the context where then() was called" from with when >=3 (3.0.0)   View in Datadog   (Fix with Cursor)
Timeout of 5000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/home/runner/work/dd-trace-js/dd-trace-js/packages/datadog-instrumentations/test/when.spec.js)

Error: Timeout of 5000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/home/runner/work/dd-trace-js/dd-trace-js/packages/datadog-instrumentations/test/when.spec.js)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

🎯 Code Coverage (details)
Patch Coverage: 4.76%
Overall Coverage: 68.02% (-0.06%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: d844bdd | Docs | Datadog PR Page | Was this helpful? React with 👍/👎 or give us feedback!

@pr-commenter
Copy link
Copy Markdown

pr-commenter Bot commented Apr 16, 2026

Benchmarks

Benchmark execution time: 2026-04-16 09:45:07

Comparing candidate commit d844bdd in PR branch juan-fernandez/fix-jest-bail-quarantine with baseline commit b5ded05 in branch master.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 233 metrics, 27 unstable metrics.

@juan-fernandez juan-fernandez force-pushed the juan-fernandez/fix-jest-bail-quarantine branch from 3bccbc2 to d0e5aa8 Compare April 16, 2026 08:57
…est --bail from stopping execution

Quarantined test failures left `event.test.errors` intact, so Jest's --bail flag
would stop the run on the first quarantined failure. This suppresses errors for
quarantined tests at test_done time (after capturing status for dd-trace reporting)
so Jest doesn't see the failure. Errors are only suppressed on the final execution
to avoid interfering with ATR retries.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@juan-fernandez juan-fernandez force-pushed the juan-fernandez/fix-jest-bail-quarantine branch from d0e5aa8 to d844bdd Compare April 16, 2026 09:30
@juan-fernandez juan-fernandez changed the title fix(test-optimization): prevent Jest --bail from stopping on quarantined test failures [test optimization] Prevent Jest --bail from stopping on quarantined test failures Apr 16, 2026
@juan-fernandez juan-fernandez marked this pull request as ready for review April 16, 2026 09:48
@juan-fernandez juan-fernandez requested a review from a team as a code owner April 16, 2026 09:48
@juan-fernandez juan-fernandez merged commit 91205a5 into master Apr 16, 2026
946 of 951 checks passed
@juan-fernandez juan-fernandez deleted the juan-fernandez/fix-jest-bail-quarantine branch April 16, 2026 10:34
dd-octo-sts Bot pushed a commit that referenced this pull request Apr 17, 2026
@dd-octo-sts dd-octo-sts Bot mentioned this pull request Apr 17, 2026
juan-fernandez added a commit that referenced this pull request Apr 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants