Thanks for this interesting initiative. Potentially it can contribute in harmonizing tools.

We'll first look into the ID topic. We understand any ID shall be globally unique, across categories of simulation entities. So far esmini have reused any specified ID in the OpenDRIVE objects, e.g. stationary objects. This traceability from OSI to scenario components is important. We'll now look into whether we can utilize ExternalReference to achieve this instead.

@thomassedlmayer We have an experimental branch which aims to address the issue of overlapping ID's in OSI. Would it be possible for you to verify that branch's functionality against the ASAM OSI Quality Checker? It would be great to get your feedback in case there's still some issues.

The branch in question: feature/global_osi_ids

We have now made a release 2.57.0 which hopefully complies with ASAM OSI Quality Checker. This is a first step for this PR.

Hi @slundel6 and @eknabevcc,
thank you for integrating this so quickly! The new id setup appears to work great for the minimal example of our current test case. I don't see any remaining overlaps between lanes/lane boundaries and moving objects. I didn't test it with more complex scenarios yet but I will have a more detailed look soon.

Though, you probably noticed that the validation step still fails. There are four remaining issues of which 3 will be fixed by some updates on the side of osc-validation. I will update this PR once this is solved on our side.
Optional explanation: During initial development, osc-validation only supported OSI SensorView as input format, so we had to build a workaround to convert ESMini's GroundTruth output to SensorView. This currently generates some unnecessary QC issues because of unset fields. We will refactor the ESMini integration in osc-validation so that it uses the direct output of ESMini as osc-validation now supports GroundTruth as input as well. Then these issues should be gone.

But there is one remaining issue which I unfortunately overlooked to mention in my initial request: OSI's rules require the host_vehicle_id to be set in the GroundTruth message (see OSI docs). ESMini currently does not set the host vehicle id in the GroundTruth message even though one entity in the scenario file is called "Ego". Would this be something that could be adapted, too?

Once this is solved, this should allow more focus on actually validating the OpenSCENARIO part of ESMini (which is the actual goal of osc-validation!

Thank you again for your cooperation and your help regarding the OSI output of ESMini!

We added the host_vehicle_id (seems like a great thing to have available!) to a patch-release v2.57.1

The new id setup appears to work great for the minimal example of our current test case. I don't see any remaining overlaps between lanes/lane boundaries and moving objects.

I accidentally stumbled across this MR while I was having issues with not globally unique IDs in the OSI quality checker. I still don't understand this rule at all, but since it is being enforced in esmini now, I found another issue with the IDs.

Check out this test scenario: xodr_stationary_objects_example.zip

The IDs of traffic signs don't seem to be handled.

Field 'id' value 9 in message 'osi3.TrafficSign' is not globally unique, already used by different message 'osi3.LaneBoundary

Good catch @ClemensLinnhoff , it was indeed missed. Thanks for the included scenario-files, it made it easy to spot.

A new commit 81ec9e62 has been pushed to dev that hopefully fixes it.

Hi @thomassedlmayer,

I've been looking into moving this PR forward. Here's a status summary of where things stand:

Resolved on esmini side (all merged to dev):

• Global OSI IDs — v2.57.0
• host_vehicle_id in GroundTruth — v2.57.1
• Traffic sign ID overlaps — commit 81ec9e6

Resolved on osc-validation side:

• GroundTruth input support appears to have landed in PMSFIT/osc-validation (README now lists "OSI GroundTruth/SensorView output"). This was the main blocker you mentioned in December (the SensorView→GroundTruth refactor).

Remaining work to get this mergeable:

1. Rebase onto current dev — the PR has merge conflicts
2. Update the osc-validation commit ref in action.yml (currently pinned to 44c6a2f) to pick up the GroundTruth support
3. Remove the OSMP_FMU/CMakeLists.txt workaround (as you noted in the PR description)
4. Re-run CI to verify everything passes end-to-end

I'd be happy to help with the rebase and cleanup if you can grant me push access to your fork. Let me know!

I updated the osc-validation reference to the newest state including some new test cases to validate start trigger behavior of some exemplary conditions. Good news: esmini passes all of them so far! (see test (ubuntu-22.04, Release) / Run osc-validation-step; There you can download the artifacts incl. overall html report.)

I recently noticed that esmini has the --traj_filter defaulted to 0.1 which somehow merges close points on a trajectory and thereby slightly manipulates the trajectory. On certain, especially real-world captured trajectories where vehicles are in "almost standstill" this leads to significant increases of the trajectory similarity metric as compared to the input trajectory. For the osc-validation call of esmini I now set this to 0.0. At least for me esmini's output due to the default setting was quite confusing. For the strict follow mode the default setting kind of disregards the OSC spec.

Also, I had to fix some CI issues that occurred out of nowhere because the github actions windows runners seem to report cygwin instead of msys now, causing PYTHON to be unset and smoke_test.py to fail. I don't know if this is just happening on my fork?

@eknabevcc @slundel6 Do you have a timeline for osc-validation to be integrated? Did you have a look if you could agree on the general setup of the github actions for osc-validation? Let me know, if I need to make any changes to comply with your requirements.

Btw, we asked to present the osc-validation project in the OpenSCENARIO CCB, so we get more traction/support also from other scenario engine providers. The discussion will probably take place on May 22.

Side note: I temporarily disabled the nightly runs because I keep getting notifications due to a missing dev branch on my fork. Maybe these could be set up more fork-friendly in general but I didn't look more into that for now.

Thanks @thomassedlmayer for pushing this forward. We'll have a look.

I can right now confirm I had the same issue with sudden cygwin OSTYPE reported for windows images, but only sometimes. We had to update script (commit d144794) to support both "msys" and "cygwin". To my understanding it's just a matter of how the bash environment is launched, it should be no functional difference.

Hi @eknabevcc,

We released the first version of osc-validation with a few new test cases. It now supports test profiles (.github/actions/run_osc_validation/test_profile.toml) which allow you to mark any test cases that are expected to fail for your scenario engine:

[[xfail]]
test = "scenario/trajectories/val_interpolation.py::test_timed_polyline_trajectory_interpolation[constant_acceleration_from_initial_speed-*]"
reason = "ESMini does not currently support timed polyline constant-acceleration interpolation."

For example, we added a test case validating the new interpolation requirement introduced in OSC 1.4.0 (constant acceleration interpolation), which is expected to fail for esmini right now (see OSC Issue #723). The test profile lets you ignore test cases you don't want to tackle immediately.

By the way, you also get a comprehensive html report with detailed error logs etc.: https://github.com/thomassedlmayer/esmini/actions/runs/29261950752/artifacts/8283893034

The other xfail-marked tests in the report validate parts of the edge cases defined in section 6.9 of the OSC specification.

Hello @thomassedlmayer ,
We've looked a bit on this PR (finally), and have some basic questions/comments:

• In test_profile.toml test 1 and 2 seems appropriately skipped for the time being.
• In test_profile.toml test 3 and 4, we can see that esmini jumps to a point on the trajectory (either interpolated between 2 timestamps, or on top of a timestamp depending on the trigger time), which should be the expected behavior. The speed of the vehicle is however not interpolated. Is the speed the root-cause of the failure, or something else?
• To have a smooth start with this, we're considering having this as a separate nightly job (similar to memoryleak and sanitize). This would also make sense if the test-scope grows, and running the checks takes longer and longer time. I've already prepared a branch for this.

Great work so far, feels like a merge is quite close!

Hi @slundel6!
Thanks for having a look!

Test 3 and 4 fail because esmini's behavior diverges from what section 6.9.3 says in my opinion. I'm not saying that 6.9.3 makes sense but I think the following phrasing to me sounds like the teleport to the corresponding position should happen instantly and not with any delay.

[...] the entity behaves as if the trajectory starts at the position interpolated between the most recent time reference and the earliest future time reference. The entity teleports to that position and continues along the trajectory.

Esmini seems to skip that mentioned point and in the subsequent frame jumps to the next trajectory vertex. So from then on the trajectories are in sync again.

I don't want to say that esmini should change any behavior. Instead, our intention was to show that this could be a good process to identify bad definitions in the standard. Similarly, such test cases could be built for new features before publishing them in a new standard version (as we demonstrated for the new acceleration-based interpolation method).

I also created another branch in this repo and removed all the workarounds for the experienced fork CI issues:
master...feature/osc-validation-integration

I don't insist on the way I've set up the CI job. Feel free to adapt it as you like!

Thanks for the detailed reply! We're taking the chance to investigate these errors, and I've ran the framework locally (which was smooth following the guide on your repo).

I don't know this framework very well so I might be doing things wrong, but I find it a bit tricky to find out exactly what went wrong in a failed test. Here's an example (short version) summary of the failed test 3:

   assert metric_result.sample_count == 5, "Trace should have 5 samples at 1 Hz from 0s to 4s inclusive"
   assert metric_result.max_time_error <= 0.01, "Trace should have max time error <= 0.01s"
   assert metric_result.max_xy_error < 0.1, "Trace should have max XY error < 0.1m"
   AssertionError: Trace should have max XY error < 0.1m
   assert 15.0 < 0.1
    +  where 15.0 = ObjectStateMetricResult(reference_object_id=1, tool_object_id=10, sample_count=5, 
   max_xy_error=15.0, max_planar_speed_...ngth_error=0.0, max_width_error=4.768371586472142e-08, max_height_error=0.0, max_dimension_error=4.768371586472142e-08).max_xy_error
   osc_validation/validation/scenario/trajectories/val_follow_trajectory_teleport.py:143: AssertionError

I can see that 0.15 is > 0.1 and that this is a distance, and can guess that in this particular case its probably at timestamp 2.0, but is there a way to be certain? E.g. some verbosity level or similar with extra info on actual vs expected values.

This looks like you most certainly ran the framework correctly! I'm glad it worked fine so far. Let me know, if you're having any issues in the future.

The ObjectState metric is very basic in the way that it just collects a variety of max error values (comparing reference trace and tool-generated trace frame-by-frame) but not the exact moment when each was collected. I agree that it could be a point for improvement to store timestamps as some kind of metadata output of that metric. I think it should be a continuous process to improve these metrics based on the needs we see.

My usual process was to just inspect the generated OSI/mcap traces in a 3D viewer or directly inspect the raw trace content (e.g. object positions). The mcaps are saved to the (probably defaulted) pytest basetemp directory for each validation run. For local validation runs you can use the clickable link in the html report's test case description:

As OSI/mcap viewer I usually use Lichtblick with the asam-osi-converter plugin. It lets you also compare two overlayed OSI traces by drag-and-dropping both into a single Lichtblick instance. You can also inspect the raw content using the "Raw Messages" panel and selecting the desired mcap topic name.

My experience was that you usually need to have an in-depth (frame-by-frame) look at the trace files for failed tests anyways in order to understand exactly what's going wrong. So I didn't see the immediate need to implement a more complex metric than necessary. But I see that some more info would definitely help with the intuitiveness.

The metrics are generally something that should be extended as we develop more test cases. I noticed the same for the trajectory similarity metric: Different simulators seem to handle scheduling fairly different which is fine by OSC rules but has some impact on trigger activation. Hence, this requires quite tolerant metrics in the sense that trajectories could be shifted by one or two frames and still be OSC compliant. So we added a lag-aligned trajectory similarity metric which can be used for trigger-activated trajectory comparison.

Thank you for the detailed explanation, very much appreciated.