One should take the results of the TSE 2012 benchmarks with a rather large pinch of salt. Many of the benchmarks in this set are very easily deceived, as most rely solely on an error-free termination of the program, and do not check to see whether the correct output was returned for any of the test cases. If you intend to compare automated repair approaches, we advise the use of the ManyBugs benchmark set instead.
Bug scenarios with correctly functioning Docker images::
- gcd
- zune
- flex
- lighttpd
- look-svr
- look-ultrix
- uniq
- units
- tiff
- php
- deroff
- indent
- atris
Bug scenarios with malfunctioning Docker images:
- ccrypt (requires execution on a real 32-bit platform, otherwise files are ignored)
- lighttpd (permissions problems?)
- imagemagick (missing PNG delegate installation in Dockerfile)
- wuftpd (permissions problems?)
- openldap (requires a very old version of GCC to compile)