Having some common machine-readable format(s) would be helpful for analyzing results. There should be a format that is easy for plotting packages like gnuplot and matplotlib to consume - I expect CSV (which is a subset of the gnuplot text data format) is the right starting point for this.
There might be other formats of interest, maybe JSON that looks something like:
{
"name": "Benchmark name",
"optimize": "ReleaseFast",
"target": "x86-64-linux-gnu",
"summary": {
"mean_wall_time": {
"value": 123438432,
"stddev": 1234,
"min": 12314566,
"max": 2431248383,
},
"other_summary_statistic": {
"value": 12345,
"stddev": 123432
"min": 23432,
"max": 12382132
}
},
"samples": [
{ "wall_time": 123455677, "peak_rss": 1234, "cpu_cycles": 123, "instructions": 1232, "cache_references": 12321, "cache_misses": 234232, "branch_misses": 128234 },
{ "wall_time": 123455677, "peak_rss": 1234, "cpu_cycles": 123, "instructions": 1232, "cache_references": 12321, "cache_misses": 234232, "branch_misses": 128234 },
]
}JSON is pretty verbose for using as a storage format, but it would compress well.
Having some common machine-readable format(s) would be helpful for analyzing results. There should be a format that is easy for plotting packages like gnuplot and matplotlib to consume - I expect CSV (which is a subset of the gnuplot text data format) is the right starting point for this.
There might be other formats of interest, maybe JSON that looks something like:
{ "name": "Benchmark name", "optimize": "ReleaseFast", "target": "x86-64-linux-gnu", "summary": { "mean_wall_time": { "value": 123438432, "stddev": 1234, "min": 12314566, "max": 2431248383, }, "other_summary_statistic": { "value": 12345, "stddev": 123432 "min": 23432, "max": 12382132 } }, "samples": [ { "wall_time": 123455677, "peak_rss": 1234, "cpu_cycles": 123, "instructions": 1232, "cache_references": 12321, "cache_misses": 234232, "branch_misses": 128234 }, { "wall_time": 123455677, "peak_rss": 1234, "cpu_cycles": 123, "instructions": 1232, "cache_references": 12321, "cache_misses": 234232, "branch_misses": 128234 }, ] }JSON is pretty verbose for using as a storage format, but it would compress well.