Summary

Replace esmini's pre-built OSI 3.5.0 binaries (externals/osi/) with source-built OSI 3.8.0 via asam-osi-utilities as a Git submodule. This modernizes the OSI dependency chain, gives esmini access to the latest OSI features (proto3 optional fields, new message types), and provides a reproducible build via vcpkg.

Motivation

• OSI 3.5.0 is outdated: The pre-built binaries shipped in externals/osi/ were frozen at OSI 3.5.0 with protobuf 3.x. Upstream OSI has moved to 3.8.0 with protobuf 5.x, proto3 optional fields, and new message types.
• Pre-built binaries are fragile: Platform-specific .a/.lib files checked into the repo create ABI compatibility issues, complicate CI, and make it impossible to debug or rebuild OSI locally.
• asam-osi-utilities is the canonical build system: The ASAM OSI ecosystem now provides asam-osi-utilities with proper CMake exports, vcpkg integration, and trace file I/O. This PR adopts it as the single source of truth for OSI.

Architecture: Two-Phase Build

The integration uses a two-phase build pattern:

Phase A -- Build OSI dependencies (cached in CI)

cmake -S externals/asam-osi-utilities -B build-osi-deps \
  -DCMAKE_TOOLCHAIN_FILE=<vcpkg>/scripts/buildsystems/vcpkg.cmake \
  -DVCPKG_TARGET_TRIPLET=<triplet>
cmake --build build-osi-deps --config Release
cmake --install build-osi-deps --prefix osi-deps/

Phase A resolves protobuf, abseil, and other transitive dependencies via vcpkg, then builds and installs OSI + utilities into a prefix directory. This is cached in CI -- subsequent runs skip this entirely.

Phase B -- Build esmini (uses Phase A output)

cmake . -B build \
  -DOSI_UTILITIES_PREFIX=osi-deps/ \
  -DCMAKE_PREFIX_PATH="osi-deps/;<vcpkg_installed>/<triplet>" \
  -DProtobuf_PROTOC_EXECUTABLE=<vcpkg_installed>/<triplet>/tools/protobuf/protoc

Phase B consumes the installed OSI libraries via standard CMake find_package(). No vcpkg toolchain needed -- just CMAKE_PREFIX_PATH pointing to the install prefix.

Why two phases?

1. Separation of concerns: vcpkg manages OSI's complex dependency tree (protobuf, abseil, utf8-range, zlib); esmini's own build stays clean with no vcpkg toolchain dependency.
2. Cache efficiency: Phase A output (~50 MB) is cached by hash of externals/asam-osi-utilities/** + triplet files. Rebuilds only when the submodule or triplets change.
3. Protobuf descriptor pool isolation: On Windows, static linking (x64-windows-static-md triplet) ensures each DLL/EXE gets its own protobuf descriptor pool, preventing double-registration crashes when esminiLib.dll and test executables both link OSI.

Commits

Design Decisions

1. Git submodule for asam-osi-utilities

Decision: Add externals/asam-osi-utilities as a Git submodule pointing to the main branch.

Rationale: Submodules provide a pinned, reproducible dependency without vendoring large build systems. The asam-osi-utilities repo includes osi-cpp and mcap as nested submodules, giving esmini the full OSI + trace file stack from a single entry point.

2. Custom vcpkg triplets

• support/vcpkg-triplets/universal-osx.cmake: Builds universal (arm64 + x86_64) fat binaries on macOS so vcpkg-managed protobuf/abseil match esmini's universal builds.
• support/vcpkg-triplets/x64-windows-static-md-v142.cmake: Uses MSVC v142 toolset (VS 2019). esmini's CI uses -T v142 for backward compatibility; vcpkg defaults to v143 whose STL intrinsics aren't available in v142.

3. find_package(Protobuf CONFIG REQUIRED) in osi.cmake

CMake's built-in FindProtobuf module does not resolve transitive dependencies (abseil, utf8-range) from vcpkg's package configs. Using CONFIG mode ensures the vcpkg-installed protobuf-config.cmake is used, which properly declares all transitive targets.

4. MSVC warning suppressions (/wd4141, /wd4267, etc.)

Suppress specific protobuf/abseil header warnings on MSVC, scoped to OSI-consuming targets only. An osi_external_warnings INTERFACE library in osi.cmake carries the 6 /wd flags, and a wrapper osi_with_warnings target propagates them automatically to anything linking OSI_LIBRARIES. Four include-only targets (ScenarioEngine, ViewerBase, PlayerBase, Controllers) explicitly link osi_external_warnings. This keeps the warnings active for esmini's own code while suppressing them where protobuf/abseil headers are compiled. /w14296 remains removed from compiler_warnings.cmake because MSVC's last-flag-wins semantics would override the per-target /wd4296 suppression.

MSVC's /external:W0 flag has a known bug that fails to suppress some warnings from SYSTEM include directories, and template instantiation in esmini's own TUs fires warnings from protobuf headers regardless.

5. Traffic sign enum validation

Add TrafficSign_MainSign_Classification_Type_IsValid() check before static_cast in OSIReporter.cpp. When a traffic sign type is not found in the OpenDRIVE-to-OSI mapping table, osi_type defaults to INT_MAX. In Debug builds, protobuf asserts _IsValid(value) and crashes. The fix validates before casting and falls back to TYPE_OTHER with a log warning.

6. Proper protobuf API for vehicle type in osi2csv.py

Replace str(o.vehicle_classification)[11:-1] with MovingObject.VehicleClassification.Type.Name(type_enum). OSI 3.8.0 changed VehicleClassification fields to proto3 optional (explicit presence tracking), breaking the fragile string-slicing approach.

7. Replace vendored osi3 Python bindings with osi-python

Delete vendored scripts/osi3/*.py files. In CI, osi-python is installed from the submodule tree (pip install externals/asam-osi-utilities/submodules/osi-python). The vendored bindings were generated for OSI 3.5.0 / protobuf 3.x and are incompatible with protobuf 5.x wire format.

8. CI cache strategy (split restore/save)

Use actions/cache/restore@v4 + actions/cache/save@v4 instead of actions/cache@v5 with save-always. The split approach saves the cache immediately after Phase A install (before downstream tests can fail), ensuring the cache is always populated on the first successful Phase A build.

9. fail-fast: false on the test matrix

Disable fail-fast so that a failure on one platform doesn't cancel in-progress jobs on other platforms. During this integration (which touches all 3 platforms), getting results from all platforms simultaneously is critical for efficient debugging.

10. Phase A configuration on Windows vs Linux/macOS

On Windows, Phase A uses matrix.configuration (Debug or Release). On Linux/macOS, Phase A always builds Release. MSVC static libraries encode _ITERATOR_DEBUG_LEVEL and CRT linkage (MD vs MDd), so a Release Phase A library cannot link into a Debug Phase B build. On Linux/macOS, static libraries are configuration-agnostic, so a single Release build serves both configurations. This means Windows has 2 Phase A caches (Debug + Release) while Linux/macOS have 1.

11. Protobuf version management

The C++ protobuf version (currently v5.29.3) is controlled by the vcpkg baseline in externals/asam-osi-utilities/vcpkg-configuration.json. The Python protobuf version (currently >=6.30.2) is declared by osi-python's pyproject.toml. These are not centrally synchronized, but wire-format compatibility across protobuf major versions ensures interoperability between C++ and Python. The requirements.txt no longer pins protobuf directly -- it is pulled transitively via osi-python.

12. Auto-derive vcpkg commit from submodule

The setup_osi_utilities composite action no longer hardcodes the vcpkg commit hash. Instead, it extracts the baseline from externals/asam-osi-utilities/vcpkg-configuration.json via jq at runtime. This eliminates the risk of the action and submodule drifting out of sync when the submodule is updated. An explicit vcpkg-commit input can still override this for special cases.

13. Resilient OSI file size assertions

Unit tests previously used exact EXPECT_EQ(fileStatus.st_size, N) assertions that break on any protobuf serialization change. These are replaced with range-based checks (EXPECT_GT/EXPECT_LT) that validate the test behavior (file was written, grew with frames, mode differences are significant) without depending on exact byte counts. The lane_no_obj test already used this pattern; now all 6 GroundTruth file-size tests follow suit.

Files Changed (77 files, +1372 -1393)

Core CMake integration

• CMakeLists.txt -- Replace DYN_PROTOBUF option with OSI_UTILITIES_PREFIX; remove OSI download step
• support/cmake/external/osi.cmake -- Complete rewrite: find_package(open_simulation_interface) via installed prefix instead of manual lib/include paths
• support/cmake/common/locations.cmake -- Remove EXTERNALS_OSI_PATH and OS-specific OSI paths
• support/cmake/common/definitions.cmake -- Remove DYN_PROTOBUF definitions
• support/cmake/common/flags.cmake -- Protobuf-specific /wd flags moved to per-target osi_external_warnings in osi.cmake; only /wd4127 and /wd4100 remain global
• support/cmake/rule/compiler_warnings.cmake -- /w14296 stays removed (MSVC last-flag-wins would override per-target /wd4296)

CI / GitHub Actions

• .github/actions/setup_osi_utilities/action.yml -- New reusable composite action for two-phase build with caching; auto-derives vcpkg baseline from submodule config via jq
• .github/actions/setup_tools_shared/action.yml -- Add osi-python and asam-osi-utilities Python package installation from submodule tree
• .github/workflows/ci.yml -- Integrate Phase A, custom triplets, fail-fast: false, platform-specific configs
• .github/workflows/memoryleak.yml -- Add Phase A setup
• .github/workflows/sanitize.yml -- Add Phase A setup

Source fixes

• OSIReporter.cpp -- Validate traffic sign enum before set_type(), fall back to TYPE_OTHER
• scripts/osi2csv.py -- Use proper protobuf API for vehicle type extraction
• test/smoke_test.py -- Update OSI assertions for 3.8.0 type aliases (MEDIUM_CAR -> CAR)
• ScenarioEngineDll_test.cpp -- Replace hardcoded OSI file size assertions with resilient range-based checks (EXPECT_GT/EXPECT_LT); increase max_msg_size buffer

Dependency management

• .gitmodules -- Add externals/asam-osi-utilities submodule (recursive)
• externals/asam-osi-utilities -- Submodule at b7c8603 (main)
• support/python/requirements.txt -- Remove standalone protobuf pin (now transitive via osi-python)
• scripts/osi3/ -- Delete all 11 vendored protobuf bindings (replaced by osi-python package)
• support/vcpkg-triplets/universal-osx.cmake -- macOS universal binary triplet
• support/vcpkg-triplets/x64-windows-static-md-v142.cmake -- Windows v142 toolset triplet

Dev Container

• .devcontainer/Dockerfile -- Add system dev packages (lz4-devel, libzstd-devel, libX11-devel, libXrandr-devel, libXinerama-devel, mesa-libGL-devel, fontconfig-devel), vcpkg bootstrap, protobuf version conflict documentation
• .devcontainer/devcontainer.json -- Forward HTTP_PROXY/HTTPS_PROXY/NO_PROXY via containerEnv for vcpkg registry fetch behind corporate proxies
• .gitignore -- Add CMakeFiles/ to ignored paths

Documentation

• README.md -- Update OSI version (3.5.0 -> 3.8.0), protobuf now vcpkg-managed, two-phase build example for Windows and Linux/macOS
• docs/user_guide.adoc -- Rewrite build guide with --recursive and two-phase instructions; replace stale DYN_PROTOBUF section with static/dynamic linking guide and protobuf/OSI version documentation; simplify Python OSI setup; update OSI doc links to latest
• release_notes.md -- Add OSI 3.8.0 migration entry
• .github/copilot-instructions.md -- Build instructions for the new two-phase workflow

Testing

• Unit tests: Updated 6 message size expectations for OSI 3.8.0 (proto3 optional fields change serialization sizes)
• Smoke tests: Updated vehicle type assertions (MEDIUM_CAR -> CAR due to OSI 3.8.0 enum alias resolution)
• All existing tests pass on all 3 platforms in both Debug and Release configurations

Breaking Changes

• DYN_PROTOBUF CMake option removed: Dynamic protobuf linking via externals/OSI/*/lib-dyn is no longer supported. For shared linking scenarios, use LINK_WITH_SHARED_OSI=ON with dynamic vcpkg triplets -- see asam-osi-utilities integration docs.
• externals/osi/ directory no longer used: Pre-built OSI binaries replaced by submodule-based build.
• New build prerequisite: Building with USE_OSI=ON now requires a Phase A pre-build step (automated in CI, documented for local builds in user guide).
• Python protobuf pin removed from requirements.txt: protobuf is now a transitive dependency of osi-python, installed from the submodule.

How to Build Locally

# Clone with submodules
git clone --recursive https://github.com/esmini/esmini.git

# Phase A: Build OSI dependencies (via vcpkg)
cmake -S externals/asam-osi-utilities -B build-deps --preset vcpkg-windows-static-md  # Windows
cmake -S externals/asam-osi-utilities -B build-deps --preset vcpkg                    # Linux/macOS
cmake --build build-deps --config Release
cmake --install build-deps --config Release --prefix osi-deps

# Phase B: Build esmini (set env vars from Phase A output, then use "full" preset)
# Windows (cmd):
set ESMINI_OSI_PREFIX=osi-deps
set ESMINI_CMAKE_PREFIX_PATH=osi-deps;build-deps/vcpkg_installed/x64-windows-static-md
set ESMINI_PROTOC=build-deps/vcpkg_installed/x64-windows-static-md/tools/protobuf/protoc.exe
cmake --preset full
cmake --build build --config Release --target install

# Linux/macOS (bash):
export ESMINI_OSI_PREFIX=osi-deps
export ESMINI_CMAKE_PREFIX_PATH="osi-deps;build-deps/vcpkg_installed/x64-linux"
export ESMINI_PROTOC=build-deps/vcpkg_installed/x64-linux/tools/protobuf/protoc
cmake --preset full
cmake --build build --config Release --target install

# Or build without OSI (no Phase A needed)
cmake --preset default
cmake --build build --config Release --target install

Known Issues (Dev Container)

• Protobuf version conflict: The base image (ghcr.io/bounverif/esmini:latest-devel) ships protobuf 3.21.12 at /opt/bazalt/. Using the vcpkg preset for Phase A causes header conflicts. Use the base preset with -DCMAKE_PREFIX_PATH=/opt/bazalt instead.
• MCAP lz4 + GCC 14: mcap/writer.inl:179 has an implicit int to LZ4F_blockChecksum_t enum conversion rejected by GCC 14 with system lz4 1.8.3. Workaround: add -DCMAKE_CXX_FLAGS=-fpermissive to Phase A configure.
• Proxy required for vcpkg: HTTP_PROXY/HTTPS_PROXY must be set inside the container for vcpkg registry fetch. The updated devcontainer.json forwards these from the host automatically.

Related PRs

• asam-osi-utilities #42 -- CMake export targets (merged, enables find_package())
• asam-osi-utilities #43 -- Writer API + TXTH deprecation
• asam-osi-utilities #44 -- Shared OSI linking CI coverage and vcpkg-shared presets (merged)
• osi-cpp #7 -- Windows shared linking (upstream issue)