Easy and Extensible Tool for Benchmark Construction and Evaluation in Multiple Programming Languages.

You can install PolyEval python package using pip:

$ pip install git+https://github.com/polyeval/polyeval.git

This repository contains tests for example usage, which leverage Nix for reproduction of the runtime enviroment.

$ git clone --recursive https://github.com/polyeval/polyeval
$ cd ./polyeval
$ nix-shell --pure # Will download pdm for project management, as well as dependencies for executing programs in all 34 programming languages. 
$ pdm install
$ pdm run python tests/test_example_evaluation.py # Will execute example tests for 34 programming languagaes.

Download dependencies for all 34 programming langugaes may occupy lots of time and disk space. polyeval-example provides an example that just download dependencies and evaluate programs of 8 mainstream programming languages.

$ git clone --recursive https://github.com/polyeval/polyeval-example
$ cd ./polyeval-example
$ nix-shell --pure
$ pdm install
$ pdm run python test_example_evaluation.py # Will execute example tests for 8 programming languagaes.

PolyEval defines a domain-specific language called PolyEval Description(ped). Here's an example:

def Example
    fun all_above(numbers: list<double>, threshold: double) -> bool
        ([1.0, 2.0, 3.0], 0.5) -> true
        ([1.0, 2.8, 3.0, 4.0, 5.0, 0.5], 0.7) -> false

This defines the behaviour of generated test program.

For example, we can use PolyEval to check whether this C++ function pass the above testcases:

bool allAbove(const vector<double>& numbers, double threshold) {
    return ranges::all_of(numbers, [=](double n) { return n > threshold; });
}

Then evaluate with this Python script:

from polyeval import parse_questions, initialize_template, evaluate

ped_dsl = """
def Example
    fun all_above(numbers: list<double>, threshold: double) -> bool
        ([1.0, 2.0, 3.0], 0.5) -> true
        ([1.0, 2.8, 3.0, 4.0, 5.0, 0.5], 0.7) -> false
"""
question = parse_questions(ped_dsl)[0]

code = """
bool allAbove(const vector<double>& numbers, double threshold) {
    return ranges::all_of(numbers, [=](double n) { return n > threshold; });
}
"""
template = initialize_template("./execution-template", targets=["cpp"])
status, _ = evaluate(template, "cpp", question, code, exist_ok=True)
print(f"Evaluation Result: {status}")

Currently, 34 popular programming languages are supported in main packages: C#, C++, Clojure, CoffeeScript, Crystal, D, Dart, Elixir, Elm, Erlang, F#, Go, Groovy, Hack, Haskell, Java, JavaScript, Julia, Kotlin, Lua, Nim, Objective-C, OCaml, Perl, PHP, PureScript, Python, Racket, ReScript, Ruby, Rust, Scala, Swift, TypeScript. More languages are supported in polyeval-extra package.