Skip to content

feat: add --slice flag for worker scenario grouping#4617

Draft
cidrblock wants to merge 5 commits into
mainfrom
slice
Draft

feat: add --slice flag for worker scenario grouping#4617
cidrblock wants to merge 5 commits into
mainfrom
slice

Conversation

@cidrblock

@cidrblock cidrblock commented Feb 28, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds a --slice flag (and config file support) to control how scenarios are grouped into work units when using --workers. With --slice=1 (the default), scenarios are grouped by their top-level directory segment so all CRUD states for a resource run sequentially on one worker. With --slice=2, each leaf scenario is dispatched independently (pre-slice behavior).

This is relevant for collections with nested scenario directories (e.g. appliance_vlans/gathered, appliance_vlans/merged) where related scenarios share state and must execute in order within a worker.

Changes by file

src/molecule/click_cfg.py — New slice property on CliOptions (experimental, default "1").

src/molecule/types.pyslice: int added to both CommandArgs and ConfigData TypedDicts.

src/molecule/command/test.py, check.py, destroy.py"slice" added to @common_options and wired into command_args via int(ctx.params["slice"]). No other runtime changes.

src/molecule/config.pyslice property reads from the config dict (self.config.get("slice", 1)), following the same pattern as shared_state. _apply_cli_overrides extended so CLI --slice takes precedence over the config file value.

src/molecule/data/molecule.json — Schema entry for slice (integer, default 1, minimum 1).

src/molecule/scenarios.pyslice property on Scenarios reads from the first scenario's config, mirroring the shared_state aggregation pattern.

src/molecule/worker.py — Core implementation:

  • _slice_key(name, depth) — splits scenario name on / and joins the first depth segments to derive the grouping key.
  • _group_scenarios_by_slice(scenarios, depth) — groups Scenario objects by _slice_key, preserving original sort order.
  • run_scenario_slice(entries, ...) — executes a list of (molecule_file, name) pairs sequentially in one worker process, stopping on first failure within the slice. Returns list[SliceResult].
  • run_scenarios_parallel — refactored to group via _group_scenarios_by_slice and submit one future per group to run_scenario_slice instead of one future per scenario. Fail-fast and continue-on-failure semantics apply at the slice level. Log output reports slice count and depth.
  • validate_worker_args — validates that --slice (non-default) requires --workers > 1.
  • run_one_scenario remains unchanged and exported for backward compatibility but is no longer called internally — run_scenario_slice handles all dispatch.

pyproject.tomlverbosity_assertions changed from 2 (int) to "2" (string). Newer pytest requires this config option to be a string; the int type caused TypeError in all integration tests with assertion failures. Pre-existing issue, not introduced by this PR.

tests/fixtures/integration/test_workers/ — Fixture scenarios restructured from flat (scenario_0..5) to nested (group_a/scenario_0..2, group_b/scenario_3..5) to enable slice testing. config.yml now includes slice: 1.

tests/unit/test_worker.py — New tests for _slice_key (4 cases), _group_scenarios_by_slice (4 cases), run_scenario_slice (success + failure-stops-early), validate_worker_args slice validation (2 cases). Existing run_scenarios_parallel tests updated for the new list[SliceResult] return format.

tests/unit/test_click_cfg.pytest_slice_option verifying the new CLI option properties.

tests/integration/test_workers.py — Scenario references updated for nested names. Two new tests: test_workers_slice_groups_by_resource (verifies 2 slices at depth=1) and test_workers_slice_depth_2 (verifies 6 slices at depth=2).

docs/guides/parallel.md — Documents --slice with a behavior table, CLI examples, config file example, and precedence notes.

Runtime behavior

Existing --workers behavior is preserved. The default --slice=1 groups scenarios by their first path segment, which is a change from pre-slice behavior (individual dispatch) but produces the same results since scenarios within a group still execute sequentially. Users can opt into the previous individual-dispatch behavior with --slice=2.

Sequential mode (--workers 1 or no --workers) is completely unaffected — slice in the config is silently ignored.

Test plan

  • Unit tests pass (_slice_key, _group_scenarios_by_slice, run_scenario_slice, validation, parallel dispatch)
  • Integration tests pass (slice depth 1 and 2, parallel success, verbose, continue-on-failure, fail-fast)
  • tox -e lint clean (ruff, cspell, pydoclint, pylint, mypy, codespell)

@ssbarnea ssbarnea marked this pull request as draft March 5, 2026 13:29
@github-actions github-actions Bot added feat and removed feat labels Mar 5, 2026
@ssbarnea

ssbarnea commented Mar 5, 2026

Copy link
Copy Markdown
Member

@cidrblock bring it out of draft once is passing all CI checks.

…ctations

The ../inventory.yml relative path in config.yml only works for flat
scenarios.  For group_a/scenario_0 it resolves to group_a/inventory.yml
(wrong) instead of molecule/inventory.yml.  Add inventory copies in each
group directory so ansible finds hosts and actually runs the playbooks.

Also fix slice count expectations: default scenario is included in
slicing, so depth=1 yields 3 slices and depth=2 yields 7.

Remove unused 'psql' from dictionary.txt (flagged by cspell hook).

Made-with: Cursor
@github-actions

github-actions Bot commented Jun 4, 2026

Copy link
Copy Markdown

@cidrblock All commits must have verified signatures.\n\nPlease see this Ansible Forum post for more information.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

3 participants