Add screenshot capture to benchmark system for automated shader testing #1220
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Add saveScreenshot() method to capture swapchain images as PPM files - Extend benchmark class to support screenshot capture at specified frame - Add command-line flags --screenshot and --screenshotframe This enables automated differential testing (not included in this diff, but trivial to add on top)
Contributor
Author
|
(note I am not a C++ programmer so feel free to give pointers and corrections) |
Contributor
Author
|
FWIW, here is output from my script that scripts the differential testing. Let me know if you want me to put up the script in a future PR: Testing example: triangle
Running with glsl shaders...
✓ glsl completed
Running with hlsl shaders...
✓ hlsl completed
Running with slang shaders...
✓ slang completed
Running with rust shaders...
✓ rust completed
Comparing outputs...
✓ glsl and hlsl are identical
✓ glsl and slang are identical
✓ glsl and rust are identical
✓ hlsl and slang are identical
✓ hlsl and rust are identical
✓ slang and rust are identical
Testing example: texture
Running with glsl shaders...
✓ glsl completed
Running with hlsl shaders...
✓ hlsl completed
Running with slang shaders...
✓ slang completed
Running with rust shaders...
✓ rust completed
Comparing outputs...
⚠ glsl and hlsl differ
⚠ glsl and slang differ
⚠ glsl and rust differ
✓ hlsl and slang are identical
⚠ hlsl and rust differ
⚠ slang and rust differ
Testing example: computeshader
Running with glsl shaders...
✓ glsl completed
Running with hlsl shaders...
✓ hlsl completed
Running with slang shaders...
✓ slang completed
Running with rust shaders...
✓ rust completed
Comparing outputs...
✓ glsl and hlsl are identical
✓ glsl and slang are identical
⚠ glsl and rust differ
✓ hlsl and slang are identical
⚠ hlsl and rust differ
⚠ slang and rust differ |
Owner
|
Will first need to test if this actually works. I did something similar a few years ago, but ran into issues with sync on some platforms/devices. No need for the differential testing script from my side though. I like to keep CI small and fast. |
Contributor
Author
|
Sounds good, no rush! Looks the differential testing isn't that useful on a lot of the output anyway due to float differences. I only tested this on mac with MoltenVK FWIW (which means some shaders were not tested of course). |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
saveScreenshot()method to capture swapchain images as PPM files--screenshotand--screenshotframeThis enables automated differential testing (not included in this PR, but trivial to add on top. I am doing so in our Rust fork)