Skip to content

fix(wix-manage): correct the store-wide coupon scope - #859

Open
danshw wants to merge 2 commits into
mainfrom
fix/coupon-store-wide-scope
Open

fix(wix-manage): correct the store-wide coupon scope#859
danshw wants to merge 2 commits into
mainfrom
fix/coupon-store-wide-scope

Conversation

@danshw

@danshw danshw commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

The defect

ecom-pricing-create-coupon.md documented the "all store products" coupon scope as

"scope": { "namespace": "stores", "group": { "name": "product" } }

with the entityId omitted — in both store-wide request examples, the Scope values for Wix
Stores
table, the SITE row of the recommendation mapping, and the fix column of the error table.

The Coupons API rejects that combination for a percentage coupon. Verbatim, from a recorded run
transcript (POST https://www.wixapis.com/stores/v2/coupons):

Wix API error (400): {"message":"The provided combination of scope and coupon type is invalid.
Provided combination:
namespace=stores, group=product, entityId=empty, coupon type=PercentOff
Supported combinations:
- namespace=stores, group=empty, entityId=empty and coupon type is one of BuyXGetY
- namespace=stores, group=product, entityId=some-id and coupon type is one of BuyXGetY
- namespace=stores, group=collection, entityId=some-id and coupon type is one of BuyXGetY
- nam…

(the recorded error body is truncated at that point)

group is valid only together with its entityId; a store-wide coupon carries the namespace
alone. The Valid scope values
reference this recipe already links to says exactly that — Group is "optional - if not listed, the
coupon will apply to all products/services/events in the namespace"
, and Entity ID is "required
only when Group is listed"
. The recipe contradicted the page it cites.

Blast radius

Reproduced on five consecutive nightly runs of the same scenario. In each one the agent read
…/skills/pricing-create-coupon, sent the documented store-wide scope on its first create call,
took the 400, then retried with group removed and succeeded. The outcome judge scored 10/10 every
time — the user does get the coupon — while the path judge scored 6–7 and named the scope
combination as the gap.

The recovery is what the recipe cost: one rejected mutating call plus a retry, on the most common
coupon request there is.

The fix

Correct the shape everywhere the recipe already restates it (no new prose about API shape, per
Orchestration, not API shape), and add the 400 to the existing error table:

  • both store-wide request examples and the query-response example → "scope": { "namespace": "stores" }
  • Scope values for Wix Stores → "All store products" omits group and group.entityId
  • SITE mapping row → { "namespace": "stores" }
  • error table: the "scope or minimumSubtotal is not used" fix no longer suggests the invalid shape
  • error table: one new row for "The provided combination of scope and coupon type is invalid."

+7 / −15 on the recipe. The front-matter description is unchanged — this corrects a shape, it does
not extend what the recipe covers, so the retrieval key should not move.

Routing

The failing agent read this recipe — ReadFullDocsArticle on
https://dev.wix.com/docs/api-reference/business-solutions/e-commerce/skills/pricing-create-coupon
was the call immediately before the rejected create, and the recipe is the only source of the wrong
shape. The upstream reference is already correct, so there is nothing to fix in the API docs, and
the API's own 400 is already explicit about the supported combinations. The defect is the recipe
restating a shape its own cited reference contradicts, so it is fixed here and only here.

Coverage

New scenario yaml/wix-manage-evals/ecommerce/pricing-promotions/ecom-pricing-create-coupon-store-wide-scope.yml,
three assertions per docs/eval-scenarios.md: ReadFullDocsArticle coverage on the recipe's doc
URL, an llm_judge on the outcome, and a gating llm_judge on the tool-call path that fails a
create call rejected with this 400 even when a retry recovers. The trigger prompt is the unmodified
request that produced the finding — it names no scope, so an agent with the old content still has
every reason to go wrong.

Validated against packages/evalforge-core/src/schema.ts (parseScenario), and the articleUrl
against the gate's canonicalDocUrl.

Note on checks

GitHub Actions is disabled repository-wide (actions/permissionsenabled: false), so no check
will appear on this PR. The gate was run manually against this branch; results are in a comment
below.

The Create Coupon recipe documented an all-products scope as
`{ "namespace": "stores", "group": { "name": "product" } }` with the
`entityId` omitted, in the two request examples, the scope table, the
`SITE` recommendation-mapping row and the error-handling fix column.

The Coupons API rejects that combination for a percentage coupon:

    HTTP 400 {"message":"The provided combination of scope and coupon
    type is invalid.\nProvided combination:\nnamespace=stores,
    group=product, entityId=empty, coupon type=PercentOff\n..."}

`group` is only valid together with its `entityId`; a store-wide coupon
carries the namespace alone. The Valid scope values reference this
recipe already links to says the same — group is optional and "Entity ID
is required only when Group is listed".

An agent following the recipe takes the 400 on its first create call and
has to retry with the group removed. Correct the shape everywhere it is
restated, and add the 400 to the error table.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@danshw

danshw commented Aug 5, 2026

Copy link
Copy Markdown
Contributor Author

EvalForge YAML Gate: eval comparison

GitHub Actions is disabled repository-wide, so the gate does not run as a check here. It was run
manually against this branch's head commit, in eval mode with auto-approve off — a manual run
uses a human token, so auto-approve would post a human-looking approval. Internal run links have
been removed from this comment.

Group verdict: required. Both comparison runs completed with no infrastructure errors.

Scenario Required Winner Cost (PR / prod) Tokens (PR / prod) Time (PR / prod)
ecommerce/pricing-promotions/create-coupon-store-wide-scope (new, this PR) ✅ yes PR, high confidence $0.521 / $0.519 255.3K / 255.7K 52.9s / 54.0s
ecommerce/pricing-promotions/create-coupon (pre-existing; pulled in because it asserts this recipe's doc URL) no no winner $0.507 / $0.723 246.4K / 352.9K 57.6s / 59.4s

create-coupon-store-wide-scope — the discriminating result

PR side — 3/3 passed

  • ✅ Tool called with param (ReadFullDocsArticle on the recipe's doc URL)
  • ✅ LLM judge, outcome — 10/10: "The agent successfully created and confirmed the 20% discount coupon 'SAVE20' for all store products."
  • ✅ LLM judge, path — 8/10: "The run successfully created a coupon but encountered a 400 error during the initial query due to unclear filter syntax documentation."

Production side — 2/3 passed

  • ✅ Tool called with param
  • ✅ LLM judge, outcome — 10/10: "The agent successfully created and confirmed the 20% off store-wide coupon 'SAVE20'."
  • ❌ LLM judge, path — 2/10: "The run failed with a score of 2 due to a 400 error on the first create-coupon call caused by an invalid scope combination, pointing to a potential documentation gap."

required: true was earned on the decision assertion, not on efficiency: the path judge passes on the
PR content and fails on the live content, while cost, tokens and wall-clock are within ~0.5% on all
three ($0.5214 vs $0.5187; 255,298 vs 255,700 tokens; 52.9s vs 54.0s). The pairwise judge scored it
PR wins, high confidence, citing "a 400 error on the initial API call" on the production side.

The coverage assertion passed on both sides, which is what makes the measurement trustworthy —
both runs opened this recipe, so its content was the only thing that differed between them.

Note the residual on the PR side: the path judge gave 8/10 for a 400 on the query call
(POST /stores/v2/coupons/query) over filter syntax. That is a different gap from the one this PR
fixes and is not addressed here.

create-coupon — pre-existing, not-required, unaffected by this change

This scenario was pulled into the run because it asserts the same recipe's doc URL. Its coverage
assertion failed on both sides — neither run called ReadFullDocsArticle for its request shape —
and its outcome judge consequently failed on both sides (1/10 on the PR side, 3/10 on production; both
verdicts read "failed to load the required pricing-create-coupon recipe"). Both runs did create the
coupon correctly.

So it returns not-sure / required: false with "both sides failed assertions … no clear winner", and
the gate's no-winner check reports before the approval block. Because approval is
autoApprove && scenarios.every(s => s.required), this PR would not be auto-approved even with
auto-approve enabled: one pre-existing scenario that this change does not affect is enough to hold it.

That scenario is deliberately left untouched — loosening or retargeting an assertion in a shared
scenario to make this PR pass would be the wrong fix. Its coverage gap is a separate finding worth
raising on its own: its triggerPrompt hands the agent a complete recommendation object, and for that
request shape the agent goes straight to executing without opening the recipe.

Scenarios in this directory carry a single area tag. This one also
carried `aria`, which selects a separate scheduled run over a different
scenario set, so the scenario was being picked up by a sweep it does not
belong to.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@danshw
danshw enabled auto-merge (squash) August 6, 2026 07:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant