Improved testing and comparison tooling - #794

I've been working on new tools to help review the result of any changes made to the codebase.

The new comparison tool is essentially an integration test which runs all of the examples using the current source and compares results against the latest release build. It produces a report flagging any:

• runtime errors
• intrinsic changes (mass, intertia, friction etc.)
• extrinsic changes (position, angle, velocity)
• performance changes

Intrinsic properties generally shouldn't change (other than new properties) as this would indicate a likely regression, or at least a breaking change. For example, the mass of the same body should never change between versions.

Extrinsic properties may change as they may have multiple acceptable values due to limited simulation accuracy. They indicate a change to the simulation behaviour, which may be acceptable and intended or may not be. For example, the position of the same body may change between versions if an accuracy improvement was since implemented. Whereas a code refactor for example, should not normally change the resulting extrinsic properties between versions.

It's possible for example that a performance refactor might come with an accuracy trade-off that changes extrinsic properties whilst producing acceptable results. Generally extrinsic changes will require some further manual testing and review before deeming acceptance into the build (the visual comparison tool below can help with judging the difference).

Here's how to run the comparison integration test (included as part of the test suite):

npm run test or npm run test-watch

There is also a new browser-based visual comparison tool that runs both versions side by side and overlays both interactive simulations in real-time for comparing extrinsic changes. Below you can see how the catapult example behaviour quickly diverges from release version when the default friction is changed.

open http://localhost:8000/?compare=120#catapult

To visualise intrinsic changes, the tool can output them into two separate files for each version which can then be diff'd e.g. using vscode.

npm run test-save
code -n -d test/__compare__/examples-build.json test/__compare__/examples-dev.json

This approach is still experimental, so it will take some time to see how useful it is in practice. I might also add code size comparison too. If anybody has ideas on other improvements for this, feel free to post them here. Thanks.