Reconfigure Raspberry Pi images with an easy, Docker-like configuration file.

pimod overtakes a given Raspberry Pi image file by mounting a copy and modifying it within a QEMU chroot. This allows the execution of a Pi's ARM code on whatever target, e.g., a x86_64 host.

To ease the usability, a Docker-inspired recipe, called the Pifile, is used to instrument pimod.

# Example Pifile to create a customized version of the Raspberry Pi OS Lite

# Based on a remote image, which will be cached locally, create the altered raspi_example.img file
FROM https://downloads.raspberrypi.org/raspios_lite_arm64/images/raspios_lite_arm64-2023-02-22/2023-02-21-raspios-bullseye-arm64-lite.img.xz
TO rapsi_example.img

# Increase the image by 100 MB
PUMP 100M

# Install an ssh key from local sources
RUN mkdir -p /home/pi/.ssh
INSTALL id_rsa.pub /home/pi/.ssh/authorized_keys

# Enable the serial console and SSH
RUN raspi-config nonint do_serial 0
RUN raspi-config nonint do_ssh 0

# Install the important cowsay util
RUN apt-get update
RUN apt-get install -y cowsay

Usage: pimod.sh [Options] Pifile

Options:
  -c --cache DEST   Define cache location.
  -d --debug        Debug on failure; run an interactive shell before tear down.
  -h --help         Print this help message.
  -r --resolv TYPE  Specify which /etc/resolv.conf file to use for networking.
                    By default, TYPE "auto" is used, which prefers an already
                    existing resolv.conf, only to be replaced by the host's if
                    missing.
                    TYPE "guest" never mounts the host's file within the guest,
                    even when such a file is absent within the image.
                    TYPE "host" always uses the host's file within the guest.
                    Be aware that when run within Docker, the host's file might
                    be Docker's resolv.conf file.
  -t --trace        Trace each executed command for debugging.

There are pre-built images available on Docker Hub:

docker pull nature40/pimod

Alternatively, you can simply build the image yourself locally. This is essential for development, among other things:

git clone https://github.com/Nature40/pimod.git
cd pimod
docker build -t nature40/pimod .

Afterwards, the Docker image can either be used by docker or docker compose:

# Using Docker:
docker run --rm --privileged -v $PWD:/pimod nature40/pimod pimod.sh examples/RPi-OpenWRT.Pifile

# Using Docker Compose:
docker compose run nature40/pimod pimod.sh examples/RPi-OpenWRT.Pifile

Of course, Docker isn't really necessary and pimod can also be used on, e.g., a Debian directly:

sudo apt-get install \
  binfmt-support \
  fdisk \
  file \
  kpartx \
  lsof \
  p7zip-full \
  qemu-user-static \
  rsync \
  unzip \
  wget \
  xz-utils \
  units

sudo ./pimod.sh Pifile

Pimod can also be used as a GitHub Action and is available on the marketplace.

name: tests
on: push

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repository
        uses: actions/checkout@v3
        with:
          submodules: recursive
      - name: Run pimod OpenWRT example
        uses: Natur40/pimod@master
        with:
          pifile: examples/RPi-OpenWRT.Pifile

The Pifile contains commands to modify the image.

Those commands are grouped in stages which pimod executes in their corresponding order.

• First, all setup stage commands are being executed to download the base image and configure the output.
• The prepare stage follows which pre-flight commands, e.g., resizing the output image.
• The action happens in the chroot stage where the QEMU chroot is built, commands are executed within, files are copied and so on.
• Finally, the postprocess stage might clean up some things.

However, as the Pifile being just a Bash script by itself and the commands are functions, which are loaded in different stages, Bash scripting is possible within the Pifile to some extend.

More internals are documented in our our scientific paper. If you stumble upon details there that you think belong in this README, feel free to create an issue or pull request.

$ cat Upgrade.Pifile
FROM 2018-11-13-raspbian-stretch-lite.img

PUMP 100M

RUN raspi-config nonint do_serial 0

RUN apt-get update
RUN bash -c 'DEBIAN_FRONTEND=noninteractive apt-get -y dist-upgrade'
RUN apt-get install -y sl

# The Upgrade.Pifile will create, called by the following command, a new
# Upgrade.img image based on the given Raspbian image. This image's size is
# increased about 100MB, has an enabled UART/serial output, the latest software
# and sl installed.
$ sudo ./pimod.sh Upgrade.Pifile

# Write the new image to a SD card present at /dev/sdc.
$ dd if=Upgrade.img of=/dev/sdc bs=4M status=progress

Further and more expressive examples are available in this repository's ./example directory. Please take a look and feel free to submit your own examples if they are covering a current blind spot.

INCLUDE includes the provided Pifile in the current one for modularity and re-use. The included file has to have a .Pifile extension which need not be specified.

INCLUDE can also accept URLs to remote Pifiles, such as raw GitHub URLs, allowing you to use public modules directly. Remote Pifiles are cached locally (see --cache option) and will be reused on subsequent runs.

# Include a local module
INCLUDE modules/setup

# Include a remote module from GitHub (omits Pifile extension)
INCLUDE https://raw.githubusercontent.com/user/repo/main/modules/module

FROM sets the SOURCE_IMG variable to a target. This might be a local file or a remote URL, which will be downloaded. This file will become the base for the new image.

By default, the Raspberry Pi's default partition number 2 will be used, but can be altered for other targets.

TO sets the DEST_IMG variable to the given file. This file will contain the new image. Existing files will be overridden.

Instead of calling TO, the Pifile's filename can also indicate the output file, if the Pifile ends with ".Pifile". The part before this suffix will be the new DEST_IMG.

If neither TO is called nor the Pifile indicates the output, DEST_IMG will default to rpi.img in the source file's directory.

INPLACE does not create a copy of the image, but performs all further operations on the given image. This is an alternative to FROM and TO.

PUMP increases the image's size about the given amount (suffixes K, M, G are allowed).

PUMP appends a partition of the size (suffixes K, M, G are allowed) using a partion type and file system (ext4, exfat, ...).

INSTALL installs a given file or directory into the destination in the image. The optionally permission mode (chmod) can be set as the first parameter.

EXTRACT copies a given file or directory from the image to the destination.

PATH adds the given path to an overlaying PATH variable, used within the RUN command.

WORKDIR sets the working directory within the image.

ENV either sets or unsets an environment variable to be used within the image. If two parameters are given, the first is the key and the second the value. If one parameter is given, the environment variable will be removed.

An environment variable can be either used via $VAR within another sub-shell (sh -c 'echo $VAR') or substituted beforehand via @@VAR@@.

ENV FOO BAR

RUN sh -c 'echo FOO = $FOO'   # FOO = BAR - substituted within a sh in the image
RUN echo FOO = @@FOO@@        # FOO = BAR - substituted beforehand via pimod

ENV FOO

RUN executes a command in the chrooted image based on QEMU user emulation.

Caveat: because the Pifile is just a Bash script, pipes do not work as one might suspect. A possible workaround could be the usage of bash -c:

RUN bash -c 'hexdump /dev/urandom | head'

HOST executed a command on the local host and can be used to prepare files, cross-compile software, etc.

ZERO fills unused space in the filesystem with zeros. This allows for better compression of the resulting image, resulting in smaller image files. Useful when creating images for distribution.

SHRINK reduces the image file size by shrinking the filesystem and partition. The operation performs a filesystem check, resizes the filesystem, updates the partition table, and truncates the image file.

Without SIZE parameter: Shrinks to the minimum filesystem size plus a 5% safety buffer to prevent the filesystem from being 100% full.

With SIZE parameter: Shrinks the image to the exact target size (suffixes K, M, G are allowed). The target size must be large enough to contain the minimum required filesystem size plus partition overhead. If the current image is already smaller than the target size, no action is taken.

Examples:

SHRINK          # Shrink to minimum size + 5% buffer
SHRINK 2G       # Shrink to exactly 2GB (if possible)
SHRINK 500M     # Shrink to exactly 500MB (if possible)

Note: SHRINK cannot be used with block devices, only image files.

Because the Pifile is just a Bash script, some dirty brilliant hacks and extensions are possible.

Sub shells can be used with the RUN command.

RUN sh -c '
apt-get update
DEBIAN_FRONTEND=noninteractive apt-get -y dist-upgrade
apt-get install -y sl
'

Here documents can also be used to create files inside of the guest system, e.g., by using tee or dd.

RUN tee /bin/example.sh <<EOF
#!/bin/sh

echo "Example output."
EOF

pidiff is a tool that compares two disk images and generates an rsync batch file for incremental updates. This allows you to create update packages that can be applied to running systems without requiring a full image reflash.

Usage: pidiff.sh [OPTIONS] BASE_IMAGE UPDATED_IMAGE

Compare two disk images and generate an rsync batch file for incremental updates.

Arguments:
  BASE_IMAGE      Path to the base image file
  UPDATED_IMAGE   Path to the updated image file

Options:
  --partition=NUM       Rootfs partition number (default: 2)
  --output=PATH         Output batch file path (default: auto-generated)
  --tar                 Create tar archive containing batch and batch.sh files
  --help                Print this help message
  
  Any other options are passed directly to rsync. For example:
  --exclude-from=FILE   Pass --exclude-from to rsync
  --verbose             Pass --verbose to rsync

Examples:
  pidiff.sh base.img updated.img
  pidiff.sh --tar --output=update.batch base.img updated.img
  pidiff.sh --itemize-changes --exclude="*.pyc" base.img updated.img

1. Mount Images: pidiff mounts both the base and updated images, identifying the rootfs partition (default: partition 2).

2. Generate Batch File: It uses rsync --write-batch to create an incremental update file that contains only the differences between the two images.

3. Inject Metadata: The generated batch script includes metadata extracted from both images:

   • Image paths and timestamps
   • OS version information (PRETTY_NAME, VERSION_ID, VERSION_COMMIT)
   • Rsync version and options used
   • Creation timestamp

4. Safety Checks: The generated batch script includes safety checks:

   • Verifies root privileges
   • Validates target directory exists
   • Checks that the target system's OS version matches the base image version

By default, pidiff creates two files:

• *.batch: The rsync batch file containing the incremental changes
• *.batch.sh: An enhanced shell script that applies the update with safety checks

With the --tar option, both files are bundled into a single tar archive.

To apply an update to a running system:

# Extract the archive (if using --tar)
tar -xf update.batch.tar

# Run the update script as root
sudo ./update.batch.sh /path/to/target/rootfs

The script will verify the target system version matches the base image before applying changes.

• OTA Updates: Create incremental update packages for remote systems
• Version Control: Track changes between image versions
• Selective Updates: Update only changed files without full reimaging
• Bandwidth Optimization: Transfer only differences instead of full images

If you happen to use pimod in a scientific project, we would very much appreciate if you cited our scientific paper:

@inproceedings{hoechst2020pimod,
  author = {{Höchst}, Jonas and Penning, Alvar and Lampe, Patrick and Freisleben, Bernd},
  title = {{PIMOD: A Tool for Configuring Single-Board Computer Operating System Images}},
  booktitle = {{2020 IEEE Global Humanitarian Technology Conference (GHTC 2020)}},
  address = {Seattle, USA},
  days = {29},
  month = oct,
  year = {2020},
  keywords = {Single-Board Computer; Operating System Image; System Provisioning},
}

• Debian Wiki, qemu-user-static
• raspberry-pi-chroot-armv7-qemu.md
• chroot-to-pi.sh
• PiShrink
• pi-bootstrap