STR.7 - Decide Whether and How to Experiment While Preserving Strategic Options
Type: Method pattern Status: Stable
STR.7:1 - Problem frame
Use this pattern when a consequential strategic question remains and someone proposes a trial, pilot or experiment to answer it. Ask whether any attainable result would improve the choice or the grounds for a claim enough to warrant the full cost of obtaining it.
The first useful result may be to use existing evidence, narrow a proposal, retain a possibility without testing it or stop. When an experiment is worthwhile, the result is a bounded design that connects possible observations to the question or decision. Identify the authority, resources, protections and stopping conditions needed before carrying it out.
The comparison shows whether testing would add evidence worth its cost. The scope is strategic use of an experiment; qualified domain methods retain experimental design, measurement, causal inference and any necessary professional assurance.
Do not add an experiment when a sufficient current comparison already answers the question. A required permission cannot be acquired by exposing people or data first and calling the exposure a test. A retained option may need only its description or the maintenance required to keep it usable.
STR.7:2 - Problem
A pilot of the favoured strategy may admit no observation that would distinguish it from a serious alternative. Positive anecdotes justify expansion; adverse results trigger another pilot; no result can end the proposal.
The apparent price also hides preparation, participant time, support, interpretation, restoration and displaced work. A “small” test may consume the only people available for a protected service. Conversely, refusing every experiment because evidence is imperfect can miss an affordable answer that would prevent a much larger mistake.
STR.7:3 - Forces
Further evidence can improve a choice or the grounds for a conclusion, but delay and exposure can make the attainable investigation worse than acting on a sufficient present basis. A narrow experiment is easier to bound, while its result supports a narrower claim.
Preserving several possibilities can protect future choice but incurs carrying cost. Testing one can improve its evidence while closing others through scarce resources or a new obligation. Inquiry and option retention therefore need related but separate comparisons.
STR.7:4 - Solution
Compare the attainable inquiry with feasible alternatives before designing a trial. For a worthwhile experiment, identify what the possible observations could establish, how much the work would cost and the conditions needed to protect those involved. Return the bounded design, an adequate existing answer or the precise obstacle.
STR.7:4.1 - Identify the question an experiment could answer
Identify the subject, horizon and requested conclusion or decision. If a choice is at issue, establish the live alternatives. State the scope of the claim. For example, can the specified service be delivered under the stated customer terms?
Describe materially different possible answers and what each would change. An answer can reject the option, narrow its scope, support a further bounded commitment or leave a decisive gap unresolved. A study can also earn its cost by establishing a warranted explanation or opening a relevant option at a longer research horizon. State that gain and when the answer could be used.
Use a sufficient PSDA.7 uncertainty account directly. If every plausible result leaves the present choice and warranted use unchanged, identify another useful question or stop. Additional precision is useful when it materially improves the receiving use.
STR.7:4.2 - Compare the attainable inquiry with other feasible continuations
Check whether existing records, calculations or qualified domain results already answer the question. Compare the proposed experiment with using that basis, asking a narrower question, making a reversible bounded choice, deferring or stopping. FPF C.11 governs this choice; C.11.DUA helps expose the work hidden in a demand for more evidence.
Include the work and consequences added by the inquiry, including design, execution, participation, checking, interpretation, support, recovery, delay and displaced work where material. Keep costs imposed on customers or other participants visible even when they fall outside the sponsor’s budget.
Use qualitative comparison when it is enough. A numerical value-of-information model needs defensible inputs; uncertain probabilities need not be invented to obtain a decision. If a plausible burden or benefit difference reverses the recommendation, establish that difference if feasible and worthwhile, or qualify the recommendation before commissioning the inquiry.
A binding evidence or protection condition remains in force. Its merits may warrant reconsideration by the responsible authority, but an attractive experiment does not remove it.
STR.7:4.3 - Preserve the alternative being displaced
Name the serious rival to experimenting now. It may be continued operation, a smaller investigation or another use of the same people or resources. State what is forgone, delayed or made harder to recover.
Keep retention separate from testing. Preserving a short account of an unchosen direction is different from paying to retain its capability, contract or interface. STR.6 supplies the retention reason and exercise conditions; STR.9 can compare the resulting flexibility. A stepping stone can remain untested until its next contribution warrants work.
If the question requires an explicit policy for a still-live collection of exploration lines, use C.19 to decide which remain worth exploring, which warrant retention without new tests and which no longer warrant a place in that collection. State the chosen policy, its intended contribution and comparison basis, and the change that would justify reconsideration. This answers the continuing search-policy question; sections 4.1–4.2 compare the worth of one proposed experiment.
STR.7:4.4 - Design the smallest experiment that can supply the useful answer
Start from the needed observation and work backwards to what must actually be offered, used or performed. Choose a form that preserves those conditions with the least full burden. A minimum viable test is the smallest sufficient way to obtain that answer, not necessarily a smaller version of the final product. The following examples show how different questions can call for different tests.
| Question the result must answer | A possible test construction | What its result can support |
|---|---|---|
| Will these people take the offered next step? | Show a comprehensible offer or demonstration and provide an observable opportunity to accept or decline that step. | A response to that offer under those conditions. Viewing or joining a waiting list alone does not establish paid or continued use. |
| Will they use or pay for the proposed service under these terms? | Provide a limited real service, using competent manual work where that is permitted, and observe use, payment and delivery effort. | The service with that actual support. Human delivery does not establish the cost or performance of a future automated arrangement. |
| Can the proposed technical mechanism perform the required work? | Exercise that mechanism under the relevant conditions, or use qualified evidence that answers the same claim. | Performance within the tested or qualified conditions. A human substituting for the mechanism cannot establish that mechanism’s performance. |
Remove a feature or substitute a simpler component only while the required observation and its interpretation remain possible. Retain the work needed to collect the observation, protect participants and close the activity. Explain what participants will receive and who may handle their information; identify whether the service is simulated or will actually be delivered.
Use the appropriate domain design to connect observations to the bounded claim. State the participants or systems exposed, contribution being tried, conditions of use, observations, interpretation and limits. Specify comparison or control conditions when the inference requires them.
When the decision specifically needs the effect of changing an offer or feature, one possible design is a concurrent randomized comparison. Construct it as follows, with competent domain and statistical support where the inference needs it:
- State the change and comparator, the people or systems to which the answer must apply, the outcome and any material harms. Choose a meaningful difference and observation period from the receiving decision. Keep the other conditions compatible with the intended comparison.
- Define the units to be assigned, then allocate eligible units to the variants by a random procedure under stated probabilities before exposure. Keep a unit’s assignment consistent unless the design explicitly handles switching. Repeated visits by one person are not automatically independent units; interaction between participants or competition for shared resources can require a different design. Sending each offer to half the recipients does not establish random assignment.
- Specify the sample size needed for decision-relevant precision, whether it is attainable, the outcome calculation, missing-observation treatment and analysis or stopping rule before inspecting the effects. Retain assignment, actual exposure and outcome evidence for both variants, including departures from the plan. Check whether observed group sizes and missingness are compatible with the allocation and collection design; resolve a material unexplained discrepancy before relying on the effect. Judge the resulting difference with its uncertainty and supported population, configuration and period.
This design supports a controlled comparison. Apply C.28’s conditions if the inference attributes the observed difference to the intervention. Use another qualified design or an adequate existing result when it answers the question better, and return an unattainable inference rather than exposing participants to an uninformative test.
Before dependent activity, establish any permissions required for participation, access, spending and changes to existing provision. Use a valid authorization already in force or seek a needed decision from its holder. Set the resource ceiling, duration or stopping event, exception response and withdrawal or restoration conditions. The ceiling includes the work needed to close the experiment responsibly.
Specify what a positive, negative or unresolved result would support. Agree on the interpretation with other participants where the design depends on their agreement. A successful local delivery may support a larger bounded comparison, not a general demand forecast. A failed result may reject this configuration without disproving every possible direction.
Keep missing conditions explicit. If the design cannot be completed until permitted use or support is established, return that bounded gap and state which independent work can still proceed if needed.
STR.7:4.5 - Return the design or sufficient stop
Return the supported comparison, any proposed or selected next contribution and its conditions. If further testing is unwarranted or infeasible, finish with the answer or evidence limit already established.
If a decision on a proposed design is needed, the authorized chooser can approve, reject or return it for a change. Capable participants carry out the authorized work using appropriate professional methods. Assess what their observations establish about the stated question, accounting for departures and limits. If a commitment needs reconsideration, STR.12 can use the result. Continuing service or recovery duties remain in force under their own terms.
Reopen when the decision, attainable evidence, affected exposure, authority, burden or option value changes. Judge continuation by what the remaining work could contribute and cost.
STR.7:5 - Archetypal Grounding
STR.7:5.1 - SensorCo can obtain a sufficient negative answer without a trial
In this constructed case, SensorCo’s existing contracts can support current service for twelve months. Two of six interviewed customers may discuss a bounded paid service trial next quarter at a price ceiling, subject to agreement and permitted use. The responses identify two possible conversations. Estimating market conversion would require different evidence.
Before the Board’s next trial decision, four already-funded internal days can complete delivery costing from existing records and specify a possible trial. The work needs no new purchase or new customer-data use. The alternative use of those days is a device-diagnostic improvement.
The Board’s supplied criterion protects current service and reserve while seeking a repeat paid contribution less exposed to generic-inspection price competition. Under the stated conditions, costing can change whether the service is worth pursuing. The Board is willing to postpone the diagnostic work for that attainable answer. This justifies preparation, not yet trial execution or a twelve-month service direction.
The interpretation depends on the preparation result.
| Attainable preparation result | Supported strategic return |
|---|---|
| Complete travel and incident-support costing shows no relevant permitted service configuration can cover cost at attainable customer terms | Recommend viable device-only continuation for this horizon without a trial |
| A feasible bounded configuration remains and an attainable trial could change the next commitment | Return the bounded trial design, full burden and missing authorizations for separate choice |
| A decisive cost component or required permitted-use answer is missing | Return that gap; do not treat the unknown as zero cost or permission |
If the customers later agree to the offer, a trial could test their use and payment under those terms and the specified delivery arrangement. It cannot establish general market demand or substitute for a required data-use determination. The four preparation days also do not approve the later eighteen-person-day development configuration.
STR.7:5.2 - A small experiment design with a bounded claim
Consider a separate constructed professional-practice case. A practitioner already has the capability to provide a diagnostic consultation and eight discretionary hours this week after current obligations. Two prospective clients have agreed to consider one paid session each. Their ability to enter that agreement, permitted inputs and the practitioner’s scope are supplied conditions of this example.
The live decision is whether to offer the same bounded format for the following four weeks. The practitioner proposes at most four hours this week: two hours per client including preparation, the session, recording the result and agreed follow-up. The plan allows seven days to complete both offered sessions. Each two-hour allowance reserves its last fifteen minutes for returning the result or an explicit limitation and closing the agreed activity. Stop exploratory work while that closure time remains. No additional spending, continuing support or consequential change to either client’s operations is promised in this test.
Before starting, state the bounded claim: each participating client will choose to pay for this format at the stated offered price, and the practitioner can deliver the agreed diagnostic result within the two-hour allowance. Observe acceptance or refusal, actual work time and whether the promised result was delivered. Include a refusal in the result for the original two prospective clients.
If both clients choose to pay on the stated terms and both deliveries meet the intended conditions, the result supports considering the same limited offer next month. If delivery exceeds the allowance, revise the format or decline continuation; willingness to pay does not fix capacity. If inputs prevent delivery, return that limitation rather than reporting a successful test. Two sessions support only these bounded claims, not a population conversion rate.
The rival is using those four hours to improve diagnostic guidance for existing clients. Compare the contribution of the bounded answer to the coming offer with the value of that displaced work.
STR.7:5.3 - A demonstration, a manual service or a working mechanism?
In this constructed case, a small research service is considering a daily briefing of public reports. The immediate decision is whether to offer the same manually supported format for one further week, not whether to finance automation. Two clients have agreed to consider the pilot under stated prices and terms. Permitted inputs, competent staff and authority for the bounded offer are supplied premises.
A video can show the proposed format and invite a response, but the decision needs evidence of actual delivery and a later order on the stated paid terms. The team therefore proposes five daily briefings for each client, produced manually, followed by a real opportunity to order the same format for the next week. If the pilot is carried out, observe delivery against the agreed content, staff effort and acceptance or refusal of that next offer. If no next offer was made, the repeat-order question remains unanswered.
The proposal allows ten staff hours over seven days: two for preparation, five for delivery, one for interpretation and two for participant follow-up and closure. Stop taking on further activity while the remaining allowance still covers closure. The rival is ten hours of improving existing clients’ guidance; proceed only if the attainable answer warrants that sacrifice. Neither unused time nor permission alone establishes that worth.
If both clients receive the agreed briefings within the allowance and order again on the stated terms, the result supports considering this small manual offer with its measured burden. It does not establish population demand or automated delivery. If the actual decision instead requires briefings produced without staff research, a manual service cannot answer it. Use a qualified result about the mechanism or design a bounded technical test if the evidence is missing and the test is worthwhile.
STR.7:6 - Bias-Annotation
Sponsors can prefer observations that preserve their idea. Specify the adverse and unresolved returns before the experiment and report what refusals, departures and missing observations establish or leave unknown.
A low sponsor cost can conceal unpaid participant labour or exposure. Ask who bears the burden and who can decline. State how the participants and sample size limit the inference.
STR.7:7 - Conformance Checklist
- The inquiry has an attainable finding, explanatory gain or option-creating contribution at a named horizon.
- Materially different results change the choice or warranted use.
- Existing evidence and other feasible continuations have been compared.
- Full burden, displaced work and the serious rival remain visible.
- Retention of a possibility is distinct from testing or selecting it.
- A proposed design states exposure, authority, resource and time limits, interpretation and responsible closure.
- The proposed test can supply the needed observation. Interpret any obtained result under the actual components and support used.
- Results apply only within their supported conditions.
- A sufficient current answer or negative result can finish without another experiment.
STR.7:8 - Common Anti-Patterns and How to Avoid Them
The pilot that cannot fail. Every observation leads to expansion or another pilot. Name the result that rejects or narrows the proposal before proceeding.
Preparation presented as commitment. Four days of costing become approval for service delivery. Preserve the separate decision and conditions for execution.
Cheap because others pay. Customers supply extensive data preparation and support. Include their burden and agreement in the comparison.
A test that creates its own missing permission. Data are used before the right is established. Establish the applicable right before using the data. If it is absent, obtain it from the authorized source or leave the dependent test unperformed.
STR.7:9 - Consequences
A strategic team can stop weak proposals with sufficient evidence and recommend stronger inquiries or commission them within its authority. Useful diversity remains available without obligating work on every possibility.
The result may still be inconclusive. State what remains unresolved when the inquiry reaches the limit of its evidence. Additional work needs a new attainable contribution; sunk effort is not that contribution.
STR.7:10 - Rationale
An experiment can improve a strategic conclusion or a decision based on it. The relevant comparison concerns attainable evidence and its whole cost, including delay and alternative uses of scarce means.
Design, authorization, performance and inference remain separate because each can fail independently. Keeping them distinct lets a practitioner finish a useful design or negative answer without overstating what has happened.
STR.7:11 - SoTA-Echoing
When is experimental learning worth further commitment? Camuffo and colleagues’ 2024 replication and extension studies a scientific approach in four randomized trials involving 759 firms and reports more idea termination with a non-linear pattern of pivots. Adapt the decision-linked hypothesis and termination contribution in sections 4.1 and 4.5. Sufficient existing evidence can justify stopping. The studies report effects for the training interventions and firms examined. A revenue claim in another setting requires its own evidence.
For accountable experimentation, adapt the OECD’s 2024 policy brief and 2025 discussion of policy experimentation. Their attention to authority, capacity, evaluation and later disposition changes section 4.4’s bounded design and section 4.5’s separate continuation. In a particular application, use the participation and assurance methods required by its domain.
For constructing a purpose-fit test, adapt the question-first move in Ries’s The Lean Startup (2011, Part Two introduction and chapters 5–6) and his MVP guide, alongside current GOV.UK prototyping guidance. Section 4.4 works backwards from the needed observation and varies what must actually function; section 5.3 makes that choice usable. Reducing a final product’s feature list is a weaker default when the remaining features cannot answer the question. A qualified existing result remains cheaper when it suffices. The startup accounts are historical illustrations, not general effectiveness estimates; the government guidance distinguishes prototype interaction from production readiness in its service-design setting. Neither supplies a technical performance or participation result for this experiment. Reopen when the inference needs different fidelity, exposure or support.
For attributing a change to an intervention, adapt the concurrent comparison in Ries’s chapter 7, with NIST’s randomized-design guidance. Section 4.4 makes allocation an actual random procedure, not a label for successive customer groups. The 2023 review by Larsen and colleagues, especially sections 1.2 and 4–6, qualifies inference through the experimental unit, observation horizon, analysis and interference. Its online-experiment Methods need their stated assumptions; the book’s reported successes do not establish general effectiveness. Reopen the design when those conditions no longer support the required effect claim.
C.11.DUA supplies the full-burden comparison against a sufficient answer or cheaper continuation. This preserves a useful experiment when its attainable contribution to the choice or grounds for the conclusion warrants the work. Reopen when evidence, exposure or attainable means change that comparison.
STR.7:12 - Relations
STR.6 supplies option and retention questions; STR.9 and STR.10 compare flexibility and commitment consequences. STR.7 helps establish whether the attainable inquiry is worth its cost and, where useful, design it. STR.11 distinguishes retained alternatives, selected joint use and authorized commitments; STR.12 uses the actual result in reconsideration.
PSDA.7 supplies qualified uncertainty when needed. FPF C.11 governs local choice and C.11.DUA the burden of evidence demands. C.19 supplies the continuing exploration-policy result described in section 4.3. The direct experimental and domain methods supply their qualified evidence and performance results.