Add per-phase step timeouts: step, waiting_for_signals, waiting_for_permits, running - #264

Pull Request type

• Bugfix
• Feature
• Refactoring (no functional changes, no api changes)
• Build related changes (Please run ./gradlew build --write-locks to refresh dependencies)
• Other (please describe):

NOTE: Please remember to run ./gradlew spotlessApply to fix any format violations.

Changes in this PR

A step can now set a separate timeout for each phase of its life:

timeouts:
  step: 24h                # from step creation, whatever the status
  waiting_for_signals: 4h  # only while waiting for signals
  waiting_for_permits: 2h  # only while waiting for permits
  running: 8h              # from start, same clock as today's `timeout`

Today a step has one timeout, measured from its start time. A step stuck waiting for a signal or a permit is never timed out, and there is no way to bound the whole step end to end. With timeouts, a step can be bounded in each phase independently, and a step that times out says which phase fired in its timeline: Step instance is timed out in phase [WAITING_FOR_SIGNALS] after [4h].

The existing timeout field is unchanged and keeps its meaning. A step sets either timeout or timeouts, not both; the validator rejects a definition with both. Steps with neither behave exactly as today: only the running clock is bounded, at the 10 day default.

Defaults are server configuration instead of constants. maestro.step-timeout takes a default for each phase and a default-timeout-phase that says which phase the single timeout field applies to. This repo ships with the running default at 10 days and default-timeout-phase: running, which is today's behavior. A deployment that wants the single timeout to bound the whole step, or wants every step bounded while waiting for signals, sets that in its yaml.

Step instances expose timeouts_in_millis next to timeout_in_millis, so clients can see the resolved limits. Steps already running when this is deployed carry only timeout_in_millis and keep being checked against the running clock.