feat(test-optimization): create final_status tag on test event for playwright#8190
Conversation
Overall package sizeSelf size: 5.7 MB Dependency sizes| name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.0.1 | 82.56 kB | 817.39 kB | | dc-polyfill | 0.1.10 | 26.73 kB | 26.73 kB |🤖 This report was automatically generated by heaviest-objects-in-the-universe |
🎉 All green!❄️ No new flaky tests detected 🎯 Code Coverage (details) 🔗 Commit SHA: 4a193e0 | Docs | Datadog PR Page | Give us feedback! |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e66d125687
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
04a7a83 to
cfcdb4f
Compare
BenchmarksBenchmark execution time: 2026-05-04 13:42:44 Comparing candidate commit 4a193e0 in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 1747 metrics, 97 unstable metrics. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0b149cc3ac
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
…t finished last) is the one reporting the final status
juan-fernandez
left a comment
There was a problem hiding this comment.
great job! Only one sneaky edge case uncovered
What does this PR do?
Create a
@test.final_statustag that is added to every playwright test event that represents the final execution of that test (there can be multiple executions of a single test due to retry mechanisms being active)Motivation
Tracers send each test execution as a span to the backend. When retry mechanisms are enabled a single test can run several times with different outcomes. These mechanisms differ in behavior. Some stop once the test reaches the desired state such as ATR ending when a test passes. Others always run a fixed number of attempts such as EFD or Attempt to fix.
To ensure the correct final result for the customer some intermediate outcomes are suppressed and the testing framework receives an adjusted final status. For example if an early EFD attempt passes but later attempts fail the test is still reported as passed and the failing attempts are hidden to avoid failing CI pipelines.
Both external and internal users have requested a way to query tests in Datadog by this final status. Their main goal is to build monitors and alerts for hard failures on default branches.
To support this a new tag will be added to the last retry of a test named:
@test.final_status:pass|fail|skip