C.40:4.8 - Search for a way to act through a model of its consequences
Use this branch when actual trials of many action rules are costly, slow or harmful, but an obtainable model can help select a smaller number of worthwhile candidates. The search develops a policy: a rule choosing an action from information available when acting. A predictor instead estimates what follows from an action in a context. Choosing the prediction with the best number is not yet constructing, performing or validating that action rule. A small direct comparison, an adequate known policy or an exact solution can finish the work without this arrangement.
Connect the action rule to the consequence account. Name the contexts, allowed actions, outcome meanings and receiving choice. A context contains observations or remembered information available before the action. Distinguish controllable settings from conditions the practitioner cannot choose. For repeated decisions, preserve the transition, observation timing, resource use and accumulated consequences in the policy’s state and update account. A fixed sequence and a rule reacting to later observations can produce different results.
Write the two roles as a = π(c) and o_hat = P(c,a). The first returns an action; the second returns a modeled outcome, distribution or bound. Give π a representation that can be applied and varied: a table, conditional procedure, rule set, parameterized function or another suitable construction. Include an incumbent whose operation is understood. The professional Method must supply how to perform each returned action; changing a policy description does not make new equipment, access or skill available.
Choose what P must return for the intended comparison. A one-step response can be iterated only with a sufficient update of context and relevant uncertainty. A predictor of total return already summarizes a continuation under its training conditions; repeatedly adding its outputs can double-count that return. A terminal score cannot supply an omitted trajectory constraint. For several outcomes, retain their trade-offs and any protected conditions. A penalty permits violations; an action restriction excludes them only insofar as its enforcing operation works. An aggregate improvement can conceal a loss in a context or for an affected party.
Obtain a model for the receiving use. Recover observations as matched context, action taken and resulting outcome, including timing, selection, censoring and changes of regime. Train or fit P with an applicable learning or modeling Method, retaining its required input meanings, fitting criterion and qualification conditions. When replacing an expensive simulator or source model, construct or obtain a cheaper response that preserves the comparison needed; state the substitution limits and the condition that returns an unsupported query to the source. Keep a source call when no adequate substitution is available. Preserve this distinction: agreement with a simulator supports the simulator’s response, while agreement with observations addresses the modeled phenomenon. Neither automatically identifies what an untried intervention would do. Use C.28 and the corresponding subject model when that intervention claim matters.
Keep the inputs available during policy use separate from information used only for assessment. For a human or expert-generated proposal, store a proposed action separately from the action performed. Fit on the latter’s outcome. An action chosen only in favorable conditions may appear better because of those conditions; merely adding more such records can preserve the error. Where the intervention consequence is unidentified, the useful output can remain a conditional model comparison or a question for a discriminating trial.
Assess the model where the search will use it. Compare relevant response errors, constraint crossings and ordering of candidate actions, not just an overall prediction average. Include withheld contexts, histories or regimes according to the claimed further use. For an iterated model, test the horizon and feedback operation as well as one-step error. Optimization can drive π toward poorly supported combinations even when P predicted historical behavior well. Restrict that use, obtain an informative source response, or retain its uncertainty; repeating the optimizer does not repair the missing relation.
Develop and compare complete policies. Fix a common set or distribution of contexts and a common model version for a comparison. For each candidate π, compute its actions and feed those actions with the corresponding contexts to P. In sequential use, carry the modeled state forward and apply π to the information then available; include the chosen horizon and terminal consequences. Calculate the agreed outcome profile and action costs. Retain nondominated candidates when the receiver has not selected one trade-off. C.18 supplies the stronger front and archive claims; a finite search usually returns the best alternatives found, not an exhaustive optimum.
Vary promising policies using the representation’s feasible operations, and repeat within an affordable allowance. A rule for one context can be combined with a rule for another; a joint action can combine operations inside the same context. Those are different search spaces. Check their joint conditions: two separately allowed actions may share a resource or interfere. Keep a sufficient incumbent and useful diversity so that the next model update does not leave only closely related candidates. Direct enumeration is preferable when the candidate family is small enough; evolutionary variation is one obtaining mechanism for a large family, not a condition for model-assisted development.
flowchart LR
D[Context, actual action and observed outcome] --> M[Fit or revise consequence model]
M --> S[Apply and compare candidate policies in the model]
E[Compatible expert behavior, when useful] --> S
S --> Q[Retain a qualified policy or trade-off set]
Q --> T[Choose permitted actual use]
T --> D
Q --> U[Query the source model when it supplies the response]
U --> M
Q --> R[Return a bounded model-based recommendation]
These arrows show result use. The return through actual use is conditional on permission, feasibility and the needed evidence; a recommendation can be the current endpoint. A simulator test supplies a source-model outcome, not the observed-world data represented by D.
Use uncertainty to change the continuation. Decide whether the uncertain comparison warrants a source query, a limited trial, a conservative alternative or an unresolved answer. With supported response bounds, propagate them into the policy comparison. With fitted predictive distributions, compare calibration and proper scores on relevant observations and retain the population, horizon and selection conditions. A nominal interval at one fixed query is not a simultaneous guarantee over all policies searched. Common model omissions can survive an ensemble, and a narrow fitted interval can be wrong outside its supported regime.
For a retained point predictor, a residual model is one possible addition. Pair its predictions with known outcomes, form residuals r = y − P(x), and fit a model for those residuals using the context and the original prediction. The corrected point is P(x) + estimated residual mean; the residual model supplies a conditional uncertainty account. RIO realizes this with a Gaussian process whose kernel is the sum of an input kernel and a prediction-output kernel, fitting its parameters to residual data. It leaves the original predictor unchanged. Its primary method, §§3–5 and Algorithm 1, supplies that numerical construction and its assumptions. A practitioner using it must distinguish uncertainty of a latent response from variation of a future observed outcome, choose the corresponding prediction, and assess the resulting intervals. Near-zero fitted training residuals do not establish small further-use error. A simpler empirical correction, a justified bound or a direct source call can be preferable.
For a multistep forecast, propagate the joint uncertainty through the update. Sampling a modeled next outcome and feeding it into the next step is one possible rollout; sampling independently at each step would lose a persistent shared disturbance unless that disturbance is retained. Quantiles of such rollouts summarize that model, not all possible failures of it. When the result changes a consequential action, examine the omitted dependence or regime before relying on numerical precision.
Use expert material without claiming to transfer expertise. This optional branch is useful when several people or existing programs already supply diverse workable policies. First agree on the input meanings, available observations, action interface and outcome comparison. Obtain their prescribed actions on contexts that expose both ordinary behavior and consequential differences. Keep permission and confidential information conditions in the collection arrangement. The submitted program or action record is reusable material; the person’s capability is not copied.
When the representations are incompatible, fit each policy’s behavior into a shared evolvable representation with an applicable fitting procedure, or translate it exactly when the finite cases and semantics permit. Compare the original and translated actions, the resulting outcome profiles and important exceptions before using the translation as a seed. If the original uses unavailable information, repair the common inputs, keep that expert as a separate callable supplier, or preserve the failed approximation. An average imitation score cannot establish preservation of a rare safety condition. Ask the expert or the receiving use which distinctions must survive; use those in the comparison. When the learned translation changes the histories it subsequently encounters, use :4.7 to obtain and fit targets on those histories, then compare the resulting behavior again.
Seed the search with the usable translations and preserve the originals for comparison and possible return. Recombination can retain different specialists in different contexts or combine useful action parts within one context. Mutation can explore beyond the submitted behavior. A mixture that only chooses whole expert outputs cannot express every such combination; it can still be the cheaper adequate alternative. If direct translation already returns a sufficient policy, stop there. Track derivation when the receiver needs to inspect origins, but re-evaluate the child: its ancestry does not establish its behavior, safety or the causal value contributed by an expert.
Return to use, then revise what changed. Return an applicable policy or a qualified set, the observations it needs, its action realization, supported outcome profile, material model assumptions and fallback. The decision maker may choose a trade-off or change an action because of a fact absent from the model. Recheck the changed action’s feasibility and modeled consequences where useful. Preserve the reason for the change so that the next learning pass does not mistake it for the original recommendation.
Choose an actual trial or deployment only within its permitted conditions. Observe what was done and what followed; compare that result with the prior prediction and incumbent under the claim being tested. A failed source-model approximation returns to the operation that constructed the substitute and its comparison with the source; a failed account of the phenomenon returns to the model construction and its subject evidence. A changed objective returns to the receiving choice, not automatically to retraining. If a model is revised, re-evaluate retained contenders on a common relevant basis before comparing their scores across versions. Continued policy evolution can reuse material; it cannot make the old and new scores commensurate by itself.
Count model construction, expert querying and translation, policy search, uncertainty calculations, actual trials, oversight and recurring use. Stop with a sufficient known policy, a conditional recommendation or a statement of which action remains unsupported when the remaining search cannot justify its cost or risk. This arrangement develops policies and their consequence models together; repair of what counts as a valuable outcome remains the separate evaluation-development question under E.23 and A.19.ECS.
C.40:4.8.1 - Try a continuation rule against a bounded recorded history
Use this branch when earlier search has left costly, reusable observations and the immediate question is which continuation or stopping rule deserves a new actual trial. A recorded history can supply a cheaper, restricted response mechanism for :4.8. It does not predict an unrecorded outcome. Prefer a direct comparison when a few affordable trials settle the choice, or when preparing and maintaining the replay would cost more than the trials it could usefully avoid.
Construct the history that the rule can encounter. Recover the initial material and the attempts, with their predecessor states, performed actions, observations, examination results and costs. Preserve the conditions that gave each result its meaning: the performer and available means, evaluation, observation interface, relevant environment and timing. A final leaderboard without this connection cannot reconstruct sequential choice. Mark what was available before each decision separately from what became known only after an attempt.
Choose a finite action interface whose responses the records supply. One simple construction opens root branches in their recorded order and permits a branch to continue only to its recorded next child. The new rule may choose among already revealed branches, request the next root branch, or stop. This changes the allocation and order of recorded attempts, not their content. Other recorded structures can support a different interface, but every permitted transition needs its own corresponding response and conditions. Do not manufacture a result for an action merely because a similar action occurs elsewhere in the archive.
Implement the replay as successive revelation. Start each run with the same initial observations, undisclosed outcomes and reset rule state. Ask the candidate rule for its next action using only the information then available; reveal the matching recorded response, charge its recorded trial cost, update the visible history and ask again. The person or program selecting the action must not consult future outcomes, final branch ranks or a summary derived from those outcomes. Where a recorded limit itself reveals information unavailable in use, do not expose that limit to the rule as a predictive feature. A rule may have been developed using other histories, but its decision inputs still obey the declared interface.
An action outside the supplied transitions, an exhausted branch, or a changed material condition returns no supported response. This is not a failed trial, a zero payoff or proof that continuation is unpromising. Stop the supported replay there, narrow the question, or obtain the missing response through an allowed actual trial or a separately justified consequence model. Do not let one candidate exploit an archive boundary as a free oracle. A partial run cannot be ranked as a completed equal-horizon run without a justified comparison of that missing continuation.
Check what can be reordered. A saved branch state can support independent continuation when the changed order leaves all consequential conditions fixed or represented. If earlier actions change a shared resource, another participant, a future observation or the performer’s learned state, simply replaying the old outcome in the new order drops that dependence. Restrict the permitted order, supply an adequate transition model under :4.8, or return to actual trials. Likewise, a batch of separately recorded attempts is not a recorded joint attempt when its members can interfere. Changing the working performer, its instructions, tools or evaluation usually changes the response question; the old history then needs a new justification, not just a new selector.
Compare at the scope the history supports. Fix the history set, initial information, trial allowance, horizon and outcome comparison. Include an adequate incumbent, and run every candidate against the same restricted response mechanism. Keep useful trade-offs between outcome, recorded trial cost and other protected conditions. Charge replay construction and execution, development and selection of rules, examination and the intended actual trials separately from the trial costs taken from the history. Reducing recorded trial calls does not by itself reduce total development cost.
Keeping the incumbent among candidates can ensure that the selected recorded value is no worse on that same fixed comparison. It says nothing by itself about a new search. Repeatedly selecting against a known history can fit its accidents. Use histories held out from rule development when the intended claim needs them, and return the selected rule to a bounded new actual search before claiming improved real performance. An actual result can extend the history; once responses or evaluation change, compare retained rules on a common relevant basis again. E.23:4.1a governs a changed evaluation and the evidence that its earlier choices helped produce.
Worked construction. An organization has archived four independent, unit-cost prototype tests. Opening the first root branch produces A with value 4; continuing A produces A2 with value 9. Opening the next root branch produces B with value 6; continuing B produces B2 with value 5. Values use the same stipulated evaluation, and the useful result of a run is its best tested prototype. Root openings must retain the order A then B. The table below is the replay designer’s complete record; a candidate rule sees a row’s value only after requesting that row.
| Recorded attempt | Predecessor and action | Revealed value | Trial cost |
|---|---|---|---|
| A | Initial state: open next root branch | 4 | 1 |
| B | Initial state: open next root branch after A | 6 | 1 |
| A2 | Continue A | 9 | 1 |
| B2 | Continue B | 5 | 1 |
With allowance 3, both candidate rules first open A and B. The incumbent then continues the higher-valued branch, B, and returns best value 6 at cost 3. A candidate rule reserves its third test for the earliest opened branch, A, and returns 9 at cost 3. That is a reproducible difference on this record. The candidate’s instruction must be expressible without reading the hidden 9; choosing A because the designer already knows that number is not an information-respecting decision. Even an expressible rule may have been fitted to this example, so this result selects a further trial rather than proving its advantage on future prototypes. Stopping after two tests returns 6 at cost 2 and remains an alternative when that result is sufficient. Requesting A3 has no recorded response and cannot be charged as an observed failure.
If the prototypes compete for a shared rig whose temperature changes between tests, the independent unit-cost construction no longer answers the reordered-use question. Retain a permitted order or obtain the affected outcomes. A lower replay score cannot settle that missing premise.
Human self-development. A learner can use an old problem-solving trace to practise choosing the next diagnostic question before its recorded answer is revealed. The usable result is an explicit choice rule, a justified comparison of its choices on the supplied trace, or a question for new practice. Remembering the solution leaks future information; a different trace or an external participant may be needed for that exercise. The replay cannot restore the learner’s former knowledge, fatigue or learning trajectory. It therefore does not show how that person would have learned under a different sequence. Test the proposed way on new appropriate work, and keep its support and acquisition costs under C.36.RP rather than treating the person’s capability as a saved branch state.