A per-species atol built from initial values still misses a species that decays below it: expose CVodeWFtolerances (the over-time half of #196) #213

Summary

#196 gave the state axis a per-species absolute tolerance (CVodeSVtolerances), which removes the
cross-species compromise: a model whose species span decades no longer has to pick one number
that is a tolerance for the largest and the smallest at once.

It does not touch the within-species, over-time version of the same problem. A tolerance built
from a species' magnitude at t=0 is still one number for that species for the whole run, and a
species that starts at order one and decays to something tiny outgrows it. Below its own atol the
species stops being error-controlled — the same failure #196 documents, arrived at through time
rather than through the species list.

CVODE's construct for this is CVodeWFtolerances, a user-supplied error-weight function: instead of
handing the integrator a fixed atol vector, you hand it a callback that computes ewt(y) at the
state actually being integrated. #196 named it explicitly as the right tool and explicitly deferred
it:

That is a within-species, over-time mismatch rather than a cross-species one, and the construct
for it is CVodeWFtolerances (a user-supplied error-weight function), which the binary also links.
Worth knowing about; not worth doing first.

The symbol is present in the built extension, so this is reachable without touching the vendored
SUNDIALS build:

$ nm .../bngsim/_bngsim_core.cpython-312-darwin.so | grep CVodeWFtolerances
00000000000873d8 T _CVodeWFtolerances

Evidence

Bruno_JExpBot2016 is the measured case, and it was chosen because no initial-value heuristic
touches it: its initial conditions are all order one, so atol="auto" derives essentially one
number for every species and the #196 vector is a no-op. Its objective still moves by 1.3e-05
between atol=1e-8 and a converged tail near 1e-16.

That is the whole shape of the gap: a model the cross-species fix cannot help by construction, still
sensitive to atol, because the sensitivity is to when rather than to which species.

Why it is not urgent

Stated plainly so this is not over-prioritized:

• The cross-species case is the one that made models fail (mxstep, unintegrable) or return
  qualitatively wrong values (in Simulator.run takes a scalar atol, so a model spanning ten decades of species scale has no usable tolerance: expose CVodeSVtolerances (the state axis already has its vector twin on the sensitivity axis) #196's wide_dynamic_range.net fixture the small species comes back
  negative where the answer is +6.7e-12). This one is a 1.3e-05 accuracy drift, not a failure.
• The workaround is real: tighten atol for the affected species specifically. Simulator.run takes a scalar atol, so a model spanning ten decades of species scale has no usable tolerance: expose CVodeSVtolerances (the state axis already has its vector twin on the sensitivity axis) #196's vector makes
  that cheap, since tightening one species no longer taxes the rest.
• An error-weight callback is on the hot path — it is called at every step — so a Python callback is
  almost certainly the wrong implementation, and the design question ("what weight function, computed
  where") is the actual work here, not the CVODE plumbing.

Suggested direction

Deliberately loose; the design is the open part.

• Decide what the weight function should be. The obvious candidate is running-magnitude:
  ewt_i = 1 / (rtol*|y_i| + rtol*max(|y_i| seen so far, floor)), i.e. the Simulator.run takes a scalar atol, so a model spanning ten decades of species scale has no usable tolerance: expose CVodeSVtolerances (the state axis already has its vector twin on the sensitivity axis) #196 rule re-evaluated
  against the trajectory rather than the initial state. Whether a running max, a running RMS, or
  something with decay is right is an empirical question and should be measured, not argued.
• Implement it in C++ against CVodeWFtolerances, not as a Python callback — see the hot-path
  note above.
• Keep it opt-in and orthogonal to Simulator.run takes a scalar atol, so a model spanning ten decades of species scale has no usable tolerance: expose CVodeSVtolerances (the state axis already has its vector twin on the sensitivity axis) #196. A scalar atol must stay on CVodeSStolerances and a
  vector on CVodeSVtolerances, both bit-for-bit; this becomes a third mode.
• Check the interaction with the sensitivity axis. Simulator.run takes a scalar atol, so a model spanning ten decades of species scale has no usable tolerance: expose CVodeSVtolerances (the state axis already has its vector twin on the sensitivity axis) #196 made atolS read the per-species state
  tolerance; if the state tolerance becomes a function rather than a vector, that derivation
  needs re-deciding rather than inheriting silently.
• Measure on Bruno_JExpBot2016 against the 1.3e-05 baseline above, and confirm the corpus does
  not move where it should not — the same "nothing else moves" check Simulator.run takes a scalar atol, so a model spanning ten decades of species scale has no usable tolerance: expose CVodeSVtolerances (the state axis already has its vector twin on the sensitivity axis) #196 ran.

Not in scope

Anything the per-species vector already solves. This is specifically for models where the vector is
a no-op because every species starts at a similar magnitude, and the run is still atol-sensitive.

Related: #196 (which deferred this by name), lanl/PyBNF#549 (adopting the cross-species half).