$p$-values are currently coarsely coded in the tables as the first digit of $\max(\lg(1/p))$, where only the best algorithm is annotated. When we post-process more than two algorithms, we compute all pairwise $p$-values and we could create a table from these which provides additional information and is reasonably informative on its own. As we have less than $k^2/2$ tests to report for $k$ algorithms, we have still $k^2/2$ cells where we could report some other relevant piece of information.

Proposal: The table should report $\pm\log_2(1/p)$ from the two-sided test without Bonferroni correction (which represents the number of coin flips ending up head - probability) and the sign indicates which algorithm is better. Reading out may be easier if the value is put in the lower left part of the matrix, because then we can always start picking an algorithm from the left-most column. The unused cells could give the relative geometric average runtime ratios, and the diagonal could give the runtime ratio w.r.t. the best as in the original tables.

Open decisions

• $p$-values are placed in the upper right or lower left?
• sort algorithms by runtime or alphabetically?
• rounding method for displaying $\log_2(1/p)$ (we don't present decimal places?)
• display also success numbers or rates?

Any other/better ideas what to display?

The feature should create a new html page with one $p$-value table for each function and each dimension.