test(e2e): fix four CI flakes at root cause and make the Stripe block… - #3360
Conversation
…-checkout failure diagnosable
The nightly develop e2e run has been red since 2026-07-31. Four of the failures are
test-side defects and are fixed here; the thirteen stripe-express ones are NOT fixed,
because their cause is still unproven and papering over them would hide a real signal.
Every fix below is a race/wait/selector fix. No assertion was weakened and no test was
skipped.
Four flakes, all the same shape — a one-shot read of a React surface taken before it
settles, then compared against a later value:
* customer: addDefaultProductToCartAndAssert raced
`Promise.all([waitForLoadState('load'), click()])`. A click that triggers navigation
already auto-waits for scheduled navigations, so pairing it with a second load wait
made the click itself time out at 30s under CI load. Click, then assert arrival
web-first. Both navigation sites in the method.
* abuse-reports: waitForListReady() returns on the first render that has any rows, which
is not the final render. The TC10 baseline captured a transient 20 rows that
re-rendered to the real 2. Adds getStableRowCount() (same count twice running) and a
web-first expectRowCount().
* vendor-auction / request-for-quotes / vendor-support: parseTabCount cannot distinguish
a genuine "(0)" from a tab whose count has not rendered yet, because DataViews paints
"All" before "All (3)". Adds expectTabCountAtLeast() to utils/dataViews.ts and applies
it to the three specs with that shape. Expected-zero assertions deliberately keep the
plain read — polling for zero would just burn the timeout.
* vendor-reports: reports/helper.ts::shouldBlockNavigation() redirects report paths via
document.location.href. When that beats DOMContentLoaded the browser cancels the
original navigation and goto() rejects with net::ERR_ABORTED even though the page loads
fine. Adds gotoReportUrl(), which swallows ONLY ERR_ABORTED and then re-waits, and
routes gotoReports/gotoAnalytics/testPageLoadPerformance through it. Also strengthens
the perf test: it measured elapsed time but never asserted the page rendered, so an
instant error page would have passed the 30s budget.
Stripe block checkout — two real defects fixed, root cause still open:
* Impossible wait budget. The post-click waits were 60s (waitForURL) + 120s (settle poll)
= 180s minimum, inside specs configured at 150s and before any setup cost. The poll,
which is what decides pass/fail, could never run to completion on CI: the test died
mid-poll and reported the poll's placeholder ("none") rather than a verdict. The SPA
redirect is the known-unreliable path on CI, so it no longer gets 60s of that budget
there (15s on CI, full 60s locally where it is the fast path), and the two affected
specs move to 240s to match the sibling money specs.
* The suite was blind to its own failure mode. After clicking Place Order it only polled
the REST API, so "the Store API rejected the payment" and "the click was a no-op because
the block was not submittable" both surfaced as the same "none". placeBlockOrderExpect-
Received now records every POST /wc/store/v1/checkout and reports it in the failure.
That is what distinguishes product-side from suite-side on the next run.
Evidence for leaving the thirteen alone rather than patching them green: a local A/B
passes both without and with #3345 (28s / 26s), the suite's own SIMULATE_CONFIRM_BLOCK
repro also passes (35s, the fallback works), and the failing shards' databases contain
zero dokan_stripe_express orders while saved-token payments on the same shard succeed.
So the order is genuinely never created, and the cause is not yet established.
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughThe Playwright end-to-end tests now use retrying count assertions, web-first interactions, redirect-tolerant report navigation, Stripe checkout diagnostics, valid billing fixtures, longer timeouts, and refreshed shard-duration data. ChangesEnd-to-end test stability
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant placeBlockOrderExpectReceived
participant StoreAPICheckout
participant OrderReceivedRedirect
participant confirmNewPaidStripeOrder
placeBlockOrderExpectReceived->>StoreAPICheckout: submit checkout request
StoreAPICheckout-->>placeBlockOrderExpectReceived: return status and response body
placeBlockOrderExpectReceived->>OrderReceivedRedirect: wait for order-received redirect
alt redirect succeeds
OrderReceivedRedirect-->>placeBlockOrderExpectReceived: return success
else redirect fails
OrderReceivedRedirect-->>placeBlockOrderExpectReceived: return redirect failure
placeBlockOrderExpectReceived->>confirmNewPaidStripeOrder: confirm payment fallback
confirmNewPaidStripeOrder-->>placeBlockOrderExpectReceived: return settlement result
end
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 5
🧹 Nitpick comments (1)
tests/pw/tests/e2e/vendor-reports/vendorReportsPage.ts (1)
170-180: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd the required
@sincetag.This new JSDoc block has no
@sincetag. Add@since DOKAN_SINCEbefore the closing marker. Do not use a guessed release version.As per coding guidelines, for new code documentation, use the literal placeholder
@since DOKAN_SINCE.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/pw/tests/e2e/vendor-reports/vendorReportsPage.ts` around lines 170 - 180, Add the literal `@since` DOKAN_SINCE tag to the new JSDoc block above the closing marker, without guessing a release version or changing the existing documentation.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/pw/tests/e2e/abuse-reports/abuseReportsPage.ts`:
- Around line 534-545: Update getStableRowCount to require a sustained stability
window: reset the window whenever rows.count() changes, and return the count
only after the unchanged window fully elapses. If the overall timeout expires
before stability is achieved, throw instead of returning the last potentially
unstable count.
In `@tests/pw/tests/e2e/stripe-express/stripeExpressPage.ts`:
- Around line 580-583: Update the response handling around storeApiAttempts so
it immediately records a placeholder when Playwright observes the response,
before calling res.text(). Replace that placeholder with the truncated body on
success, or the unavailable-body marker on failure, while preserving the HTTP
status.
In `@tests/pw/tests/e2e/vendor-reports/vendorReportsPage.ts`:
- Around line 505-510: Update assertAnalyticsShell() and its call from the
report performance check to require a reports-specific selector—analyticsRoot or
reportsMenuActive—to be visible, while retaining the PHP-fatal check. Remove
bodyLen-based success logic so login, generic error, or unrelated pages cannot
satisfy the shell assertion.
- Around line 181-188: Update gotoReportUrl to validate navigation to the
expected canonical report URL with page.waitForURL(...) after page.goto,
preserving ERR_ABORTED handling only when that expected URL is reached; re-throw
navigation failures otherwise, then wait for report-specific content instead of
relying solely on waitForLoadState('domcontentloaded').
In `@tests/pw/utils/dataViews.ts`:
- Around line 184-196: Add the literal `@since` DOKAN_SINCE tag to the JSDoc
blocks for expectTabCountAtLeast in tests/pw/utils/dataViews.ts (lines 184-196),
getStableRowCount in tests/pw/tests/e2e/abuse-reports/abuseReportsPage.ts (lines
523-533), and expectRowCount in
tests/pw/tests/e2e/abuse-reports/abuseReportsPage.ts (line 547); do not use a
released or guessed version.
---
Nitpick comments:
In `@tests/pw/tests/e2e/vendor-reports/vendorReportsPage.ts`:
- Around line 170-180: Add the literal `@since` DOKAN_SINCE tag to the new JSDoc
block above the closing marker, without guessing a release version or changing
the existing documentation.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: b0597ff9-1bf0-413f-a145-abb95b1f93d9
📒 Files selected for processing (11)
tests/pw/tests/e2e/abuse-reports/abuseReports.spec.tstests/pw/tests/e2e/abuse-reports/abuseReportsPage.tstests/pw/tests/e2e/customer/customerPage.tstests/pw/tests/e2e/request-for-quotes/newRequestedQuotes.spec.tstests/pw/tests/e2e/stripe-express/stripeExpress.spec.tstests/pw/tests/e2e/stripe-express/stripeExpressPage.tstests/pw/tests/e2e/stripe-express/stripeExpressXss.spec.tstests/pw/tests/e2e/vendor-auction/newAuction.spec.tstests/pw/tests/e2e/vendor-reports/vendorReportsPage.tstests/pw/tests/e2e/vendor-support/newVendorSupport.spec.tstests/pw/utils/dataViews.ts
| async getStableRowCount(timeout = 15000): Promise<number> { | ||
| const rows = this.page.locator(this.adminReact.dataRow); | ||
| let last = -1; | ||
| const deadline = Date.now() + timeout; | ||
| while (Date.now() < deadline) { | ||
| const current = await rows.count(); | ||
| if (current === last) return current; | ||
| last = current; | ||
| await this.page.waitForTimeout(300); | ||
| } | ||
| return last; | ||
| } |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
Wait for a sustained stable period.
Line 540 returns after one unchanged 300 ms interval. The intermediate row count described at Lines 526-530 can persist across that interval and then change on a later render. The helper can still return the transient baseline that this change intends to avoid.
Reset a stability window after each count change. Return only after that window elapses. Throw when the deadline expires instead of returning a potentially unstable count.
Proposed fix
async getStableRowCount(timeout = 15000): Promise<number> {
const rows = this.page.locator(this.adminReact.dataRow);
- let last = -1;
const deadline = Date.now() + timeout;
+ let last = await rows.count();
+ let stableSince = Date.now();
+
while (Date.now() < deadline) {
+ await this.page.waitForTimeout(300);
const current = await rows.count();
- if (current === last) return current;
- last = current;
- await this.page.waitForTimeout(300);
+ if (current !== last) {
+ last = current;
+ stableSince = Date.now();
+ continue;
+ }
+ if (Date.now() - stableSince >= 1000) return current;
}
- return last;
+ throw new Error(`Abuse report row count did not settle within ${timeout} ms`);
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| async getStableRowCount(timeout = 15000): Promise<number> { | |
| const rows = this.page.locator(this.adminReact.dataRow); | |
| let last = -1; | |
| const deadline = Date.now() + timeout; | |
| while (Date.now() < deadline) { | |
| const current = await rows.count(); | |
| if (current === last) return current; | |
| last = current; | |
| await this.page.waitForTimeout(300); | |
| } | |
| return last; | |
| } | |
| async getStableRowCount(timeout = 15000): Promise<number> { | |
| const rows = this.page.locator(this.adminReact.dataRow); | |
| const deadline = Date.now() + timeout; | |
| let last = await rows.count(); | |
| let stableSince = Date.now(); | |
| while (Date.now() < deadline) { | |
| await this.page.waitForTimeout(300); | |
| const current = await rows.count(); | |
| if (current !== last) { | |
| last = current; | |
| stableSince = Date.now(); | |
| continue; | |
| } | |
| if (Date.now() - stableSince >= 1000) return current; | |
| } | |
| throw new Error(`Abuse report row count did not settle within ${timeout} ms`); | |
| } |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/pw/tests/e2e/abuse-reports/abuseReportsPage.ts` around lines 534 - 545,
Update getStableRowCount to require a sustained stability window: reset the
window whenever rows.count() changes, and return the count only after the
unchanged window fully elapses. If the overall timeout expires before stability
is achieved, throw instead of returning the last potentially unstable count.
| void res | ||
| .text() | ||
| .then(body => storeApiAttempts.push(`HTTP ${status}: ${body.slice(0, 300)}`)) | ||
| .catch(() => storeApiAttempts.push(`HTTP ${status}: <body unavailable>`)); |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Record the response before reading its body.
storeApiAttempts gets an entry only after res.text() resolves. If the response body stalls, the failure message says that no checkout POST occurred, even though Playwright observed the response. Push a placeholder entry before calling res.text(). Update that entry when the body becomes available.
Proposed fix
-const storeApiAttempts: string[] = [];
+const storeApiAttempts: Array<{ status: number; body: string }> = [];
const status = res.status();
+const attempt = { status, body: '<body unavailable>' };
+storeApiAttempts.push(attempt);
void res
.text()
- .then(body => storeApiAttempts.push(`HTTP ${status}: ${body.slice(0, 300)}`))
- .catch(() => storeApiAttempts.push(`HTTP ${status}: <body unavailable>`));
+ .then(body => {
+ attempt.body = body.slice(0, 300);
+ });🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/pw/tests/e2e/stripe-express/stripeExpressPage.ts` around lines 580 -
583, Update the response handling around storeApiAttempts so it immediately
records a placeholder when Playwright observes the response, before calling
res.text(). Replace that placeholder with the truncated body on success, or the
unavailable-body marker on failure, while preserving the HTTP status.
| private async gotoReportUrl(url: string): Promise<void> { | ||
| await this.page.goto(url, { waitUntil: 'domcontentloaded' }).catch((err: unknown) => { | ||
| if (!/ERR_ABORTED/.test(String(err))) { | ||
| throw err; | ||
| } | ||
| }); | ||
| await this.page.waitForLoadState('domcontentloaded'); | ||
| } |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
file='tests/pw/tests/e2e/vendor-reports/vendorReportsPage.ts'
if ! rg -n 'waitForURL' "$file"; then
echo "Expected a URL-aware wait in gotoReportUrl" >&2
exit 1
fi
rg -n -C 8 'gotoReportUrl|ERR_ABORTED|waitForLoadState|waitForURL' "$file"Repository: getdokan/dokan
Length of output: 195
🏁 Script executed:
#!/usr/bin/env bash
set -u
file='tests/pw/tests/e2e/vendor-reports/vendorReportsPage.ts'
# Print file metadata and the relevant method context without failing on missing waitForURL.
printf '--- candidate lines ---\n'
sed -n '155,210p' "$file" | cat -n
printf '\n--- related gotoReportUrl usages ---\n'
rg -n 'gotoReportUrl|ERR_ABORTED|waitForLoadState|waitForURL' "$file" || true
printf '\n--- Playwright API references in tests ---\n'
rg -n 'waitForURL\(|waitForLoadState\(' tests/pw/tests/e2e vendor-reports 2>/dev/null || trueRepository: getdokan/dokan
Length of output: 50370
Confirm the correct report URL before returning from gotoReportUrl.
ERR_ABORTED handling does not prove the application redirected to the canonical report URL. page.waitForLoadState('domcontentloaded') can complete for the current document without checking any new URL. Use page.waitForURL(...) with the expected report URL, then wait for the report-specific content; re-throw when the expected navigation does not occur.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/pw/tests/e2e/vendor-reports/vendorReportsPage.ts` around lines 181 -
188, Update gotoReportUrl to validate navigation to the expected canonical
report URL with page.waitForURL(...) after page.goto, preserving ERR_ABORTED
handling only when that expected URL is reached; re-throw navigation failures
otherwise, then wait for report-specific content instead of relying solely on
waitForLoadState('domcontentloaded').
| await this.gotoReportUrl(url); | ||
| const elapsed = Date.now() - start; | ||
| expect(elapsed).toBeLessThan(30000); | ||
| expect(await this.hasNoPhpFatal()).toBe(true); | ||
| expect(elapsed, `${url} should load within the 30s budget`).toBeLessThan(30000); | ||
| // Timing alone is not a load: a page that errored out instantly would "pass" the budget. | ||
| // Assert the reports shell actually rendered (this also re-checks for a PHP fatal). | ||
| await this.assertAnalyticsShell(); |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Require a reports-specific selector for the performance check.
The new call at Line 510 delegates to assertAnalyticsShell(). That helper treats bodyLen > 50 as success even when neither analyticsRoot nor reportsMenuActive is visible. A login page, generic error page, or unrelated document with enough text can pass this check.
Require analyticsRoot or reportsMenuActive, together with the PHP-fatal check. Do not use body length as proof that the reports shell rendered.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/pw/tests/e2e/vendor-reports/vendorReportsPage.ts` around lines 505 -
510, Update assertAnalyticsShell() and its call from the report performance
check to require a reports-specific selector—analyticsRoot or
reportsMenuActive—to be visible, while retaining the PHP-fatal check. Remove
bodyLen-based success logic so login, generic error, or unrelated pages cannot
satisfy the shell assertion.
| /** | ||
| * Web-first "this tab reports at least N" assertion. | ||
| * | ||
| * `parseTabCount` cannot distinguish a real zero from a not-rendered-yet tab: DataViews paints the | ||
| * label ("All") before the counted label ("All (3)"), and both a missing `(n)` and a genuine `(0)` | ||
| * come back as 0. A single `expect(await getTabCount('all')).toBeGreaterThanOrEqual(1)` therefore | ||
| * races the count in — the shape behind the intermittent CI failure "All tab count >= 1 after | ||
| * seeding / Received: 0". | ||
| * | ||
| * Polling fixes the race without weakening anything: the assertion still fails if the count never | ||
| * reaches `min`. Use this wherever a count is expected to be non-zero. An expected-zero assertion | ||
| * (`toBe(0)`) must NOT use this — poll there would just wait out the timeout. | ||
| */ |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Add the required documentation version placeholder.
Each new JSDoc block must include @since DOKAN_SINCE. Do not add a released or guessed version.
tests/pw/utils/dataViews.ts#L184-L196: Add@since DOKAN_SINCEtoexpectTabCountAtLeastdocumentation.tests/pw/tests/e2e/abuse-reports/abuseReportsPage.ts#L523-L533: Add@since DOKAN_SINCEtogetStableRowCountdocumentation.tests/pw/tests/e2e/abuse-reports/abuseReportsPage.ts#L547-L547: Add@since DOKAN_SINCEtoexpectRowCountdocumentation.
As per coding guidelines, new code documentation must use the literal placeholder @since DOKAN_SINCE.
📍 Affects 2 files
tests/pw/utils/dataViews.ts#L184-L196(this comment)tests/pw/tests/e2e/abuse-reports/abuseReportsPage.ts#L523-L533tests/pw/tests/e2e/abuse-reports/abuseReportsPage.ts#L547-L547
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/pw/utils/dataViews.ts` around lines 184 - 196, Add the literal `@since`
DOKAN_SINCE tag to the JSDoc blocks for expectTabCountAtLeast in
tests/pw/utils/dataViews.ts (lines 184-196), getStableRowCount in
tests/pw/tests/e2e/abuse-reports/abuseReportsPage.ts (lines 523-533), and
expectRowCount in tests/pw/tests/e2e/abuse-reports/abuseReportsPage.ts (line
547); do not use a released or guessed version.
Source: Coding guidelines
…reports redirect fix Follow-up to bd210f5, using what that run's diagnostic reported. The Store API instrumentation answered the open question. Across the failing shards it printed 36 times, always the same, and never once the alternative: The Blocks checkout never issued POST /wc/store/v1/checkout at all — the Place Order click was a NO-OP (block not submittable / validation blocked it), so no payment was ever attempted. This is a checkout-submission failure, not a declined payment. So no payment was ever attempted: this is not a product defect and not a declined card. The budget change also did its job — the settle poll now reports "Timeout 120000ms exceeded while waiting on the predicate" instead of dying mid-poll at the 150s test timeout. Root cause of the no-op: fillCardDetails only ever re-read the card NUMBER. The method already knows the Payment Element can re-mount mid-entry, but a re-mount after the number is typed keeps the number populated while silently dropping expiry and CVC. That leaves an INCOMPLETE card, so elements.submit() fails validation, WooCommerce Blocks refuses to submit, and the Place Order click does nothing — no POST, no order, and the settle poll can only ever report "none". It matches the observed signature exactly: zero Store API attempts on CI, while saved-token payments on the same runner succeed because they never touch the card fields. fillCardDetails now verifies all three fields persisted before returning, and names the one that was lost when it retries. Also, when no POST is issued, the failure now reads back the block's own notice banner and the Payment Element's inline error, so the next such failure names the offending field instead of requiring another round trip to find out. vendor-reports: the previous ERR_ABORTED fix was incomplete and CI caught it. Swallowing the abort stopped goto() throwing, but waitForLoadState then resolved instantly against the cancelled, blank document while the app's redirect was still in flight — so the assertion added in the same commit correctly caught an empty page (bodyLen <= 50). It now polls for real rendered content instead of a load event, which still fails if the page genuinely never renders. Verified locally: the six flake-fix specs are 130 passed / 4 skipped (pre-existing skips) / 0 failed, and all 13 previously-failing stripe-express tests pass. Three of those 13 initially failed under --workers=2 on transfer contention; an A/B against the pushed revision showed they pass with and without this change when run serially, which is how CI runs them (playwright.config.ts sets workers to 1 on CI). Note the limit of that evidence: local cannot reproduce the PE re-mount, so local green here proves no regression rather than proving the fix. The added diagnostics mean the next run either goes green or names the field that failed validation.
…n submit Follow-up to d74cbc9, acting on what that run's diagnostic printed. Run 31095722827 reported, on every failing Stripe block-checkout test: The Blocks checkout never issued POST /wc/store/v1/checkout at all — the Place Order click was a NO-OP (block not submittable / validation blocked it), so no payment was ever attempted. Block validation said: Your mobile phone number is invalid. 38 NO-OP diagnostics, zero Store API attempts. The card was NOT the problem — the "card did not fully persist" guard added in the same commit never printed once, so the Payment Element filled correctly and the incomplete-card theory it was testing is dead. The billing phone seeded for these flows was `(555) 555-5555`. 555-555-XXXX is the reserved fictional range, and Stripe Link validates the billing phone as a MOBILE number, so it rejects it with exactly that message. The rejection blocks the WooCommerce Blocks checkout from submitting at all, which is why the settle poll could only ever report "none". Replaced with `+14152367890` in all eight places: a real-format US mobile with a valid area code and exchange, in E.164 so no validator has to infer the country. Nothing asserts on this value — every occurrence is a seed — so this cannot change what any test proves. The `createOrder.billing` copy carries a comment explaining the failure mode so it does not get tidied back to a 555 number. Where the evidence stops, stated plainly: this does NOT reproduce locally, in any configuration tried. Locally the same customer carries the same number and SE-CHK-B-01 passes; the Stripe API accepts `(555) 555-5555` on a PaymentMethod when tested directly; and temporarily un-blocking merchant-ui-api so the Link wallet mounts still passes. The message is emitted by Stripe, not by anything in wp-content — grepped the whole install and the running container. So the reasoning is that Link's mobile validation only engages on the runner, consistent with the runner-IP Link and hCaptcha behaviour already documented in gotoBlockCheckout(). Only CI can confirm it. If CI is still red, the diagnostic will print `Block validation said: …` again. A CHANGED message means the phone was one cause and there is another; an IDENTICAL message means the phone is not the cause and the next thing to attack is Link mounting on the runner at all. Verified locally: tsc --noEmit at the develop baseline of 82, SE-CHK-B-01 green in 28.9s, and the seeded customer really does now carry +14152367890 rather than the value merely changing in source.
…to have content Third attempt at this one. The first two each failed on CI and each narrowed it: 1. Swallowing only ERR_ABORTED stopped goto() throwing, but waitForLoadState then returned instantly against the cancelled, blank document whose load event had already fired. 2. Polling for rendered content passed against the PRE-redirect page; the redirect then blanked it, so assertAnalyticsShell still saw bodyLen <= 50 a moment later (run 31095722827). Content alone is not a settled page. shouldBlockNavigation() can redirect a report path more than once, so the only safe signal is the URL holding still AND the body having content in the same observation — both true twice running before gotoReportUrl returns. On timeout it now reports the final URL and body length rather than leaving the caller to fail on a bare `expect(false).toBe(true)`, which is what made the previous two rounds hard to read. Verified locally: tsc --noEmit at the develop baseline of 82, and the whole vendorReports spec is 41 passed / 4 skipped (pre-existing skips) / 0 failed.
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/pw/tests/e2e/vendor-reports/vendorReportsPage.ts`:
- Around line 190-202: Add the literal `@since` DOKAN_SINCE tag to the
documentation block for gotoReportUrl, preserving the existing documentation and
placement.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 2eba6f4b-8e10-4e10-9770-0e4dca62e906
📒 Files selected for processing (1)
tests/pw/tests/e2e/vendor-reports/vendorReportsPage.ts
| * Wait for the page to STOP MOVING, not merely to have content once. | ||
| * | ||
| * Two earlier attempts failed on CI and each taught something: | ||
| * 1. Swallowing ERR_ABORTED alone — `waitForLoadState` then returned instantly against the | ||
| * cancelled, blank document, whose load event had already fired. | ||
| * 2. Polling for rendered content — that passed against the PRE-redirect page, and the | ||
| * redirect then blanked it, so `assertAnalyticsShell` still saw bodyLen <= 50 a moment | ||
| * later. Content alone is not a settled page. | ||
| * | ||
| * `shouldBlockNavigation()` can redirect a report path more than once, so the only safe | ||
| * signal is the URL holding still AND the body having content on the same observation. | ||
| * Both must be true twice running before this returns. | ||
| */ |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Add the required documentation version placeholder.
Add @since DOKAN_SINCE to the gotoReportUrl documentation block. This new documentation must use the literal placeholder.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/pw/tests/e2e/vendor-reports/vendorReportsPage.ts` around lines 190 -
202, Add the literal `@since` DOKAN_SINCE tag to the documentation block for
gotoReportUrl, preserving the existing documentation and placement.
Source: Coding guidelines
…m error, rebalance shards
Run 31147407922 took the branch from 14 failures to 1. The phone fix landed: every shard that
carried the Stripe block-checkout tests went green, and vendor-reports passed on its third attempt.
This clears the last failure and fixes the sharding.
SE-GUEST-01. The diagnostic message CHANGED, which is what identified this:
before: Block validation said: Your mobile phone number is invalid.
now: Block validation said: Please provide a mobile phone number.
Not invalid any more — absent. A guest has no saved address, so nothing pre-fills the phone the way
ensureCustomerAddress() does for logged-in flows; it has to be typed. fillBlockGuestDetails() had no
phone parameter at all. Added as optional, so existing callers are unaffected, and tolerant of the
field being missing when a store has the phone field switched off.
SE-GUEST-03 was passing for the wrong reason, and would have kept passing. It asserts "an inline
error appears and we never reach order-received" — which is exactly what a MISSING PHONE produces.
The declined card was never submitted, so the test proved nothing about a decline. Two changes:
the guest details now carry a phone so the decline is genuinely exercised, and
placeBlockOrderExpectError() now requires the notice to be about the payment
(/declin|card|payment|insufficient|cvc|expir/), so an address or contact validation error can never
satisfy a decline test again.
Shard rebalancing, regenerated from this run's spec-durations.json across all 12 shards (195 specs,
up from 190). The previous baseline was three weeks old (2026-07-15) and badly wrong:
old: shards ranged 15.7 -> 86.9 min, spread 71.2 min (190% of ideal)
new: shards range 16.7 -> 16.8 min, spread 0.1 min (0.6%)
That matches the wall-clock actually observed on CI, where one shard took 48m13s while another took
23m51s. The 18 paypal-marketplace specs are not in this run's data and fall back to the mean, which
is fine — they skip in seconds without credentials. Regenerate again once PayPal really runs.
One inflated entry to be aware of: stripeExpressGuest is recorded at 9.3 min because SE-GUEST-01
failed and retried three times in the run the data came from. It will drop once this commit is
verified green. Over-estimating a spec only spreads load away from it, so it is safe to ship.
Verified locally: tsc --noEmit at the develop baseline of 82, and the whole guest spec passes
(3/3) with the stricter payment-error assertion in place.
… keywords Reverts a regression I introduced in 8627018. That commit tightened placeBlockOrderExpectError() to require the notice to mention the payment (/declin|card|payment|insufficient|cvc|expir/), which broke all three decline tests on CI run 31162562169: SE-CHK-B-03, SE-EDGE-04 and SE-GUEST-03. A real Stripe decline does not surface with any of those words. WooCommerce Blocks reports it as the generic "Something went wrong. Please contact us to get assistance.", so the assertion rejected a genuine decline. The intent was right, the implementation assumed copy it had no business assuming. Inverted: reject the FORM-VALIDATION family instead (/please provide|please enter|is invalid|is required|enter a valid/) and accept anything else. That still closes the loophole the guard exists for — "Please provide a mobile phone number." satisfied the original bare toBeVisible() check, so a decline test could pass without the card ever being submitted — without depending on how Stripe or Woo happen to word a decline. Also makes the guest phone fill fail loudly. It previously guarded with `if (count())` and swallowed fill errors, so a renamed field was indistinguishable from "this store has no phone field", and the caller only found out much later as a no-op Place Order click with no visible validation message. It now asserts the field is visible and that the value persisted. Verified locally, each run individually: SE-CHK-B-03 (24.0s), SE-EDGE-04 (36.3s) and SE-GUEST-03 (33.8s) all pass. tsc --noEmit at the develop baseline of 82. Still open on this branch, not addressed here: SE-GUEST-01 (the phone message is gone but it now fails with no visible validation message, and it reproduces locally), and four tests that began failing with the shard rebalance in 8627018 — adminDataViewsMigration, productAdvertising, newProducts draft tab and newAuction row-edit. The rebalance itself worked, shards went from 23:51-48:13 down to 15:15-21:33, but regrouping changes which specs share a worker and that can expose cross-test coupling. Those four are undiagnosed.
The four non-Stripe CI failures on this branch were never flakes. Each is a different spec earlier on the same shard leaving global state dirty, so the victim fails on CI while passing in isolation on a clean database: - adminModules derived `wasActive` from catalogue PRESENCE, which is always true, so its "reset" deactivated Store Reviews instead of restoring it and adminDataViewsMigration then found no DataViews tabs. - paymentsPage.togglePaymentMethod blind-flipped the offline gateways despite the "enable" comment, disabling the already-enabled bacs and timing out productAdvertising's checkout. - productsDetails pinned `vendor_product_editor` to 'legacy' and never restored it, so newAuction's Edit action opened the legacy screen. - newProductForm saved a draft product and never deleted it, so newProducts' "empty Draft tab" assertion saw a row. Fix each at the source: capture real state and restore it, set the desired state rather than flipping whatever is there, and clean up seeded products. Also fixes the same bug class in five more specs found by sweeping for mutations without restores. productsDetails additionally flips its seeded product to 'draft' and deleted none of its seven products; setting.spec ends the terms-and-conditions test on 'off' when the seeded default is 'on', which is the leak singleStore carried a defensive re-seed for; vendorSettings kept its catalog and min-max resets INSIDE a test, so a skip or failure left them applied; newWithdrawB15 left the disbursement schedule configured. Restores go through setOptionValue where a merge could not remove an added key. noticeAndPromotion is unrelated: Dokan's promotion feed can serve more than one live promotion, so the slider renders n slides and a bare toBeVisible() throws a strict-mode violation. Assert every match instead. Verified by running each polluter and reading the global state back: modules, offline gateways, product editor, terms-and-conditions, catalog mode, withdraw disbursement and draft count all return to their pre-run values. 285 tests passed across the two runs with no failures. Whether the four victims now pass inside their real shards still needs a CI run.
After a shard finishes, re-run only the tests that failed, without their shard-mates, and let that second pass decide the shard's result. A test that fails in shard context but passes alone is separated from one that fails either way. The main test step becomes continue-on-error and a gate step at the end fails the shard unless the isolation re-run cleared every failure. `outcome` still reports the true first-pass result, so the six later steps that key off steps.e2e-test.outcome are unaffected. Guard rails: - No recorded test failures means no re-run. If the first pass died from a crash or a timeout rather than assertions, `--last-failed` would select zero tests and trivially "pass", turning an infrastructure failure green. The gate keeps the shard red instead. - The re-run overrides the reporters with --reporter=list. The blob reporter would otherwise emit a second set of results for the same tests and merge-reports would double-count them. The merged HTML report therefore keeps showing the first-pass failures, which is what actually happened in shard context. - It runs before the coverage step, which overwrites .last-run.json. - Every rescued test is written to the step summary. A failure that only reproduces alongside its shard-mates is the signature of shard-state pollution, not a timing flake, and the summary says so and points at how to confirm it. Known trade-off: Playwright's in-place `retries` cannot green a deterministic shard-context failure, because all three attempts run with the same dirty state. This can, and the re-run inherits retries, so a test now gets three attempts in context plus three alone. The four state-pollution bugs fixed in 3949b53 would have been reported green with a warning rather than red. Verified locally with a throwaway spec: --last-failed selects only the failed test, a still-failing re-run exits 1 and fails the gate, a recovered one exits 0 and greens the shard, and an empty or corrupt .last-run.json skips the re-run.
| const longUnicodeLast = 'Tëst-Sürñámé-日本語-' + 'Ω'.repeat(12); | ||
| const longUnicodeAddr = 'Straße Ñoño 日本語 番地 ' + 'Ä'.repeat(48); | ||
| const billing = { first_name: longUnicodeFirst, last_name: longUnicodeLast, company: '', address_1: longUnicodeAddr, address_2: '', city: 'New York', state: 'NY', postcode: '10003', country: 'US', email: 'customer1@email.com', phone: '(555) 555-5555' }; | ||
| const billing = { first_name: longUnicodeFirst, last_name: longUnicodeLast, company: '', address_1: longUnicodeAddr, address_2: '', city: 'New York', state: 'NY', postcode: '10003', country: 'US', email: 'customer1@email.com', phone: '+14152367890' }; |
There was a problem hiding this comment.
shouldnt we put this on testdata?
There was a problem hiding this comment.
Agreed — done in 2bd8801.
It turned out to be wider than this one line: the phone was hardcoded in seven places across five files — three billing seeds in payloads.ts, three guest-checkout fills in stripeExpressGuest.spec.ts, the page object's customer fixture and the orders page fixture. All of them now reference a single exported MOBILE_TEST_PHONE in utils/payloads.ts.
I put the reason for the value on the constant rather than leaving it bare. Stripe Link validates the billing phone as a mobile number and rejects the reserved fictional (555) 555-5555 range; when it does, WooCommerce Blocks refuses to submit at all, so no Store API request fires and no order is created — the test fails with no visible card error. Without that note the value looks arbitrary and the obvious tidy-up back to a fictional number would silently break the Blocks checkout again.
Side benefit: +14152367890 is a plausibly-real San Francisco number. It is seed data and nothing is ever sent to it, but if we want to swap it for something safer (libphonenumber's canonical +14155552671, inside the reserved 555 range but with a valid exchange) that is now a one-line change instead of a hunt across five files.
No behaviour change — same value everywhere it was already used. tsc holds at the 82-error develop baseline with a byte-identical error set.
Note: paypalMarketplaceGuest.spec.ts has the same literal but is not in this PR, so I left it for the PayPal branch rather than pulling an unrelated file in here.
Review feedback on #3360: the phone belonged in test data, not inline in a spec. It was hardcoded in seven places across five files — three billing seeds in payloads.ts, three guest-checkout fills, the page object's customer fixture and the orders page fixture. A single exported MOBILE_TEST_PHONE now backs all of them. The constant carries the reason for its value. Stripe Link validates the billing phone as a MOBILE number and rejects the reserved fictional (555) 555-5555 range; when it does, WooCommerce Blocks refuses to submit at all, so no Store API request fires, no order is created, and the checkout test fails with no visible card error to point at. Without that note the value reads like an arbitrary string and the obvious "tidy-up" to a fictional number silently breaks the Blocks checkout again. Keeping it in one place also means a validator change is a one-line edit rather than a hunt across five files. No behaviour change: same value everywhere it was already used. tsc holds at the 82-error develop baseline and the error set is byte-identical before and after.
develop now carries the CI flake fixes and the Stripe block-checkout work as the squashed #3360, so the nine commits this branch already merged from qa/ci-e2e-flake-fixes-and-stripe-diagnostic arrive again with a different history and conflict on content that is already identical. Five files conflicted, all on the same change: develop's MOBILE_TEST_PHONE constant, which landed in 2bd8801 after this branch's merge point. Resolved by taking develop's side in every case; the branch held no changes to those files that develop lacks. The PR diff collapses from 59 files to 33 - the PayPal Marketplace suite plus the shared infra it introduced.
The nightly
develope2e run had been red since 2026-07-31. Two different problems were tangled together in those failures and needed different fixes. Both are now fixed, and the branch also rebalances the shard baseline and adds an end-of-shard isolation re-run to CI.No assertion was weakened and no test was skipped to achieve this. One test that was passing for the wrong reason is now stricter, not looser.
1. Stripe block checkout — 12 failures to 0
Root cause, reported by CI once the suite was made to describe its own failure:
Block validation said: Your mobile phone number is invalid.The seeded billing phone was the reserved fictional
(555) 555-5555. Stripe Link validates the billing phone as a mobile number and rejects that range, so WooCommerce Blocks never submitted the form.POST /wc/store/v1/checkoutnever fired, no order was created, and the settle poll could only ever report"none". The card was never the problem.fillBlockGuestDetails()with fail-loud assertions.placeBlockOrderExpectReceived()now records every Store API checkout POST and reads back the block notice banner.SE-GUEST-03was passing on a missing-phone form error without ever submitting the declined card. The decline assertion now matches by exclusion — the error must not be a form-validation message — because real declines surface as "Something went wrong. Please contact us to get assistance.", which a payment-keyword match rejected.2. Cross-spec state pollution — the four "flakes" that were not flakes
Four tests failed on CI and passed locally in isolation, across two runs. They were never timing flakes: each is a different spec earlier on the same shard leaving global state dirty. A clean-environment isolation run cannot reproduce this by construction, because a pristine database is exactly the condition under which the victim passes. Three earlier attempts that treated them as races did not work.
adminDataViewsMigrationStore Reviews tabsadminModules.spec.tsproductAdvertisingcheckoutpaymentsPage.tstogglePaymentMethod()blind-flipped despite its "enable" comment, disabling an already-enabled offline gatewaynewAuctionReact editorproductsDetails.spec.tsvendor_product_editortolegacyand never restored it, unlike its three sibling specsnewProductsempty Draft tabnewProductForm.spec.tsA sweep for the same bug class found six more, all fixed here:
productsDetails.spec.tsalso flips its seeded product todraftand deleted none of its seven seeded products.setting.spec.tsends the terms-and-conditions test onoffwhen the seeded default ison— the leaksingleStore.spec.tswas carrying a defensive re-seed for. Fixed at both ends.vendorSettings.spec.tskept its catalog and min-max resets inside a test, so a skip, a failure or a--grepleft them applied.newWithdrawB15.spec.tsleft the disbursement schedule configured.noticeAndPromotionPage.ts— unrelated. The promotion feed can serve more than one live promotion, so the slider renders n slides and a baretoBeVisible()throws a strict-mode violation. It now asserts every match, which is stricter than the single-element assertion it replaces.Two traps worth knowing when fixing this class:
dbUtils.updateOptionValuedeep-merges, so it can never remove a key a spec added — usesetOptionValuefor a true restore. And a restore placed inside a test only runs if that test runs and passes.3. Shard rebalance
utils/shard-durations.jsonregenerated (the baseline was from 2026-07-15, 190 specs, now 195). Measured spread went from 71.2 min to roughly 0.1 min estimated, confirmed against real CI wall-clock: shards now finish in 15–21 min rather than 24–48 min.4. CI — end-of-shard isolation re-run
After a shard finishes, only the tests that failed are re-run without their shard-mates, and that second pass decides the shard result.
--last-failedwould select zero tests and trivially pass; the gate keeps the shard red instead.merge-reportswould double-count them. The merged HTML report therefore keeps showing the first-pass failures..last-run.json.Trade-off, stated deliberately. Playwright's in-place
retriescannot green a deterministic shard-context failure, because all three attempts run against the same dirty state. This can, and the re-run inherits retries. The four pollution bugs in section 2 would have been reported green-with-a-warning rather than red under this policy. That was an explicit decision, not an oversight.Verification
43ab2b728626a8fe4a3949b5306b077d3878Each state-restore fix was verified by running the polluter and reading the global state back — modules, offline gateways, product editor, terms-and-conditions, catalog mode, withdraw disbursement and draft count — rather than by re-running the victim, which proves nothing.
tsc --noEmitholds at the develop baseline of 82 throughout.Known gaps
productsDetails"vendor can create product tags" is a pre-existing flake measured at 20% raw failure locally, and it consumed both CI retries in one run. Not fixed here.newProducts"vendor can view product list page" is a new retry-masked flake, first seen inside a green run.All Submissions:
Changes proposed in this Pull Request:
Test-only and CI-only. No plugin source is touched. See sections 1–4 above.
Related Pull Request(s)
Closes
How to test the changes in this Pull Request:
npx playwright test --project=e2e_tests --workers=1 tests/e2e/admin/adminModules.spec.tsthen confirm the Store Reviews module is still active. Repeat for
payments,productsDetailsandnewProductFormagainst their respective globals. Re-running the victim test proves nothing, because it passes on a clean database either way.Changelog entry
Fix e2e suite failures caused by Stripe block-checkout phone validation and cross-spec shard state leaks
Previously the nightly e2e run was red: the Stripe block checkout never submitted because the seeded billing phone was rejected as a non-mobile number, and four further tests failed because other specs on the same shard left global options, modules and seeded products dirty. Now the checkout submits, every spec restores the global state it changes, the shard baseline is balanced, and a shard's failures are re-run in isolation before the shard is called red.
Before Changes
Nightly
develope2e red since 2026-07-31, 17 failures at its worst. Shard wall-clock spread 15.7–86.9 min.After Changes
12/12 shards green on two consecutive runs. Shard wall-clock 15–21 min.
PR Self Review Checklist:
FOR PR REVIEWER ONLY: