Codecov Report

❌ Patch coverage is 78.18182% with 12 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.28%. Comparing base (59c14bf) to head (13e62c4).
⚠️ Report is 7 commits behind head on main.

@mikemhenry from the comment in #758 (comment) . I don't think it's that easy to use our wrapper for these changes. I think eventually we just have to drop support for pymbar 3 at some point.

One thing that I noticed is that this particular test fails with a rather cryptic XLA message when running in GPU with CUDA. The message is as follows

ERROR: Flag 'coordination_agent_recoverable' was defined more than once but with differing types. Defined in files 'external/xla/xla/tsl/distributed_runtime/coordination/coordination_service_agent.cc' and 'external/xla/xla/tsl/distributed_runtime/coordination/coordination_service_agent.cc'.

And just fails without much information. On CPU this works fine, hopefully it's just something with my local environment and nothing deeper to worry about.

The test seems to be rather unstable even doubling the iterations. I guess we could have better results with more states, but that probably requires a bigger refactor and would affect the other tests as well. That said, re-running the failed jobs would probably just work if we want the green checks.

I wonder if we also want better or more robust of checking this instead of the hard 6.0 SIGMA line. For example, taking an average of the higher percentile or similar, and comparing that one with the 6 sigma. That is basically smooth the "worse" results and compare the smooth results.

Hmm, ok, on a second thought, maybe doubling the iterations did help. I see a significantly higher number of CI workflows failing with the original number of iterations...