kiwi-scan: A Modular Scan Framework for Commissioning and Diagnostics in EPICS Environments

Actuators, detector PVs, triggers, subscriptions, plugins, and metadata sidecars are configured via YAML. Scan engine (scan type) and scan dimensions are chosen by command line or API. Results are written to timestamped text files. Optional metadata sidecars can record constants and monitored PVs in parallel.

• YAML configuration for actuators, detectors, scan dimensions, triggers, metadata PVs/constants, subscriptions, plots and plugin parameters.
• Pluggable scan engines such as linear, approach, poll, para, and cm, plus externally registered scan types.
• Pluggable runtime extensions - Plugins hook into scan logic and events that can add computed columns or act to monitor events. Base classes are provided for synchonous and asynchronous plugin processing.
• EPICS integration via pyepics wrapper for or a simulated actuator backend for tests and development.
• Structured outputs including the main scan file, optional metadata sidecar logging and waveform support, and post-mortem plotting tools.
• Event handling - Subscriptions route monitored events into defined roles.
• Triggers - Triggers run configured PV writes at named scan phases or on monitor events.
• IOC - pythonSoftIOC-based generic scan IOC.
• Monitors - Flexible machine readable formatted text output and live plotting tool.
• Data Export - Extensible export framework for converting kiwi-scan data to external formats, with built-in support for the SPEC file format.

Install the latest released version from PyPI:

pip install kiwi-scan

Upgrade:

pip install --upgrade kiwi-scan

Install IOC support

pip install "kiwi-scan[ioc]"

Install a specific version

pip install "kiwi-scan[ioc]==0.4.0"

For development, clone the repository and install it in editable mode:

pip install -e ".[dev,ioc]"

kiwi-scan depends on pyepics which depends on Epics Base Version 3.14.12 or higher. Installing the bash completions package (e.g. Debian: sudo apt install bash-completion) is highly recommended for the CLI tools.

The example below runs a tiny detector-free scan with a simulated actuator and writes the output into the current directory.

Create sim_minimal.yaml:

actuators:
  theta:
    type: sim
    pv: THETA
    rb_pv: THETA:RBV
    velocity: 1.0
    dwell_time: 0.0

detector_pvs: []
data_dir: .
output_file: sim_scan.txt
include_timestamps: true

Run a 5-point linear scan:

export KIWI_SCAN_DATA_DIR="$PWD"

scan_runner \
  --scan_type linear \
  --config-file ./sim_minimal.yaml \
  --dim actuator=theta,start=0,stop=1,steps=5

• scan_runner loads the YAML file.
• The --dim arguments define the actual scan range for this run.
• A timestamped file such as sim_scan-20260401123045.txt is created. If the file exists, a unique id is created.
• Even without detectors, the file still contains the scan position and scan timestamp columns.

Command-line tools support a --log-level argument for setting the logging level using MBBO-style values:

0 = NOTSET, 1 = DEBUG, 2 = INFO, 3 = WARNING, 4 = ERROR, 5 = CRITICAL.

kiwi-scan provides a set of built-in scan engines for common beamline and commissioning workflows, including point-by-point scans, continuous motion scans, polling-based acquisition, and nonlinear approach trajectories. In addition, users can implement and register custom scan types through the scan registry API.

For detailed description check the kiwi scan type documentation Kiwi Scan types

The SyncController is used by scan engines to synchronize the scan cycle with multiple synchronized EPICS monitor updates. For detailed description check the SyncController documentation SyncController

The kiwi-scan supports statistics for scan types based on BaseScan and provides a command line tool for live statistics from configured subscriptions. For detailed description check the statistics documentation Stats

kiwi-scan uses YAML files to configure actuators, detector PVs, monitors, triggers, subscriptions, plugins, metadata, and scan-related defaults.

The scan dimensions are normally added by the API or command line arguments (--dim). A scan object always needs at least one scan dimension.

Unknown fields in dataclass-based YAML blocks are generally ignored during parsing. This makes configuration files more forward-compatible across versions, but it also means that misspelled optional fields may not fail immediately.

See the full YAML reference for the description.

The actuator framework provides a small interface for motion-like devices defined in AbstractActuator.

Concrete backends implement the real hardware access, for example EPICS PVs or an undulator-specific multi-axis actuator.

The actuator_runneris a command line tool useful for EPICS monitor based debugging, testing and shell scripting. Typical use:

actuator_runner --config-file ./mono.yaml --move energy=400

The most important files are:

actuator/
  single.py          # common actuator interface and PV event model
  multi.py           # composite actuator for multiple child axes

actuator_concrete/
  single_epics.py    # EPICS implementation of a single actuator
  undulator.py       # two-axis undulator actuator helpers

actuator_runner.py   # command-line tool for direct actuator operation

A more detailed description can be found here.

For details on the print and live queue plotter monitors, including output formats, YAML configuration, and multi-panel plotting, see monitor.md.

kiwi-scan can be embedded directly as a Python library, for example inside a Python IOC or another beamline control application.

The command-line tools use the library API: they build a ScanConfig, load scan/plugin implementations, and then create or execute a scan object.

Check the public API documentation described here.

Plugins are instantiated from plugin_configs and discovered from the built-in plugin package plus any files or directories listed in KIWI_SCAN_PLUGIN_PATH. New plugin classes must be derived from the interface defined in plugin base class or from the async plugin base class

For detailed description check the plugin user and developer documentation:

Plugins - plugin API, plugin discovery, YAML configuration, and built-in plugins such as LoggingPlugin and JogPidPlugin.

Async Plugins - async plugin extension for parallel background processing.

For running kiwi-scan as a generic EPICS soft IOC, see ioc.md.

kiwi2spec converts one or more kiwi-scan data files, including associated metadata, into the SPEC file format for the use with existing analysis and visualization tools. Established EPICS-community tools can further convert SPEC data into formats such as NeXus.

kiwi2spec

After installation, the command line tools are available:

• scan_runner - execute scans from YAML + CLI dimensions + other options
• actuator_runner - actuator commands and run optional monitors + formatted output
• scanplotter_cli - plot scan data, optionally use manifest and file index
• pollstats_cli - online statistics without scan from YAML config
• scantrigger_cli - execute triggers from YAML config
• manifestfiles - list, create, archive, or delete manifest-related files
• scanioc - run the generic scan IOC based on pythonSoftIOC
• kiwi-convert - convert scan data through the generic export framework
• kiwi2spec - export one or more scan data files, including metadata, to SPEC
• scantrigger_cli- execute triggers from YAML config
• manifestfiles - a simple tool to list files referenced in manifests
• kiwi2spec- export one or more scan data files, including metadata

Examples:

scan_runner --help
actuator_runner --help
scanplotter_cli --help
scantrigger_cli --help
pollstats_cli --help
manifestfiles --help
scanioc --help
kiwi-convert --help
kiwi2spec --help

See the Makefile helpers section for information how to install the bash completion scripts.

The manifest writer can be used to track a sequence of scans across independent runs from command line tools or API.

Create or select a new manifest file:

scan_runner --newmanifest [optional_filename.yaml]

Each scan engine appends its configuration and output file reference to the active manifest.

Manifest writing can be controlled from YAML with manifest_mode:

# default: full manifest entry including the full scan config
manifest_mode: full

# smaller manifest entry: data and metadata file references only, no full config block
manifest_mode: small

# do not append manifest entries for this scan
manifest_mode: off

If no filename is given, a timestamped file is created (in KIWI_SCAN_DATA_DIR if set). kiwi_scan.scan.common provides append_to_manifest(self, scan_type: str = None) -> None for external scan types.

A typical run can generate two kinds of files:

1. Main scan file

   • timestamped file name based on output_file
   • position column
   • per-line timestamp
   • detector values and optional detector timestamps
   • plugin-generated columns

2. Metadata sidecar file

   • constants from metadata_constants
   • initial PV snapshots
   • change-driven CA monitor events for the configured metadata_pvs

The post-mortem plotting tools can combine scan files and metadata files for later analysis.

• KIWI_SCAN_DATA_DIR — base directory for output files
• KIWI_SCAN_MANIFEST_FILE – explicitly set the active manifest file
• KIWI_SCAN_MANIFEST_STATE_FILE – override the active manifest state file path
• KIWI_SCAN_REPLACE_* — placeholder replacement values for YAML templates
• KIWI_SCAN_CONFIG_DIR — where preset YAML configs are searched
• KIWI_SCAN_PLUGIN_PATH — extra plugin files/directories to import
• KIWI_SCAN_SCAN_PATH — extra scan-type files/directories to import

See examples/beamline_env.sh for a setup example.

User and developer documentation:

• Actuator framework - small common abstract interface to actuator like devices.
• YAML reference - data model description.
• API reference - public API description.
• Trigger manager - scan trigger phases, YAML configuration, and trigger execution.
• Plugins - plugin API, YAML configuration, and built-in plugins such as LoggingPlugin and JogPidPlugin.
• Scan Types - provided and custom kiwi scan types.
• vim - simple syntax highlighting setup
• Stats - kiwi statistics support
• SyncController - kiwi synced scan loops
• IOC - kiwi scan IOC
• Converter - data export

For repository development, a top-level Makefile and mkvenv.sh helper script are provided.

source ./mkvenv.sh

mkvenv.sh must be sourced, not executed. It:

• creates .venv if it does not exist
• activates .venv
• upgrades pip and installs build helpers on first setup
• installs kiwi-scan in editable mode with development extras
• prefixes the shell prompt with KIWI so the active development shell is obvious

If you want the environment to remain active in your current shell, always use source ./mkvenv.sh directly. make runs recipes in subprocesses and cannot keep your interactive shell activated.

Use the self-documenting help target to see the available development commands:

make help

The development targets are:

• make help - show help
• make lint - run pylint on src/kiwi_scan
• make test - run Python unit tests
• make install_completion - install bash completion snippets from bash-completion/
• make uninstall_completion - remove installed bash completion snippets
• make cscope - build cscope and ctags indexes used by vim.
• make tag - create a timestamp-based tag from HEAD
• make clean - remove .venv, caches, tags, and generated metadata such as *.egg-info

We use a dark launch strategy: features are deployed continuously but activated separately. Timestamped tags (e.g. 0.1.1+20260424.094820) are mapped to a tagged integration state. Public releases use clean semantic versions (X.Y.Z) on PyPI, each mapped to a time stamped tag.

• Read the development setup section.
• Keep changes focused and small.
• Run make test locally.
• Tests should be added in tests/
• Use logging for diagnostics.
• For debugging use the --log-level switch. Include log output and config for reporting bugs.

Cite the Zenodo concept DOI:

DOI 10.5281/zenodo.20662095

The v0.4.0 release has its own version DOI on Zenodo.

DOI 10.5281/zenodo.20662096

• KIWI Scan Presentation
  https://indico.in2p3.fr/event/37441/contributions/171970/attachments/101030/156847/kiwi-scan.pdf

• HASMI: A Configurable Python Framework for Automated Harmonic Analysis and Scan Orchestration at EMIL
  A. Balzer, A. Efimenko, P. Sreelatha Devi, K. Holldack,
  Helmholtz-Zentrum Berlin (HZB), Berlin, Germany,
  Proceedings of ICALEPCS 2025, THAG004 (2025).
  https://proceedings.jacow.org/icalepcs2025/pdf/THAG004.pdf

This project is licensed under the MIT License. See the LICENSE file for details.