Ember Exam is an addon to allow you more control over how you run your tests when used in conjunction with Ember CLI QUnit. It provides the ability to randomize, split, and parallelize your test suite by adding a more robust CLI command.

It started as a way to help reduce flaky tests and encourage healthy test driven development. It's like Head & Shoulders for your tests!

Using Ember Exam is fairly straightforward as it extends directly from the default Ember-CLI test command. So, by default, it will work exactly the same as ember test.

$ ember exam
$ ember exam --filter='acceptance'
$ ember exam --server

The idea is that you should be able to replace ember test with ember exam and never look back.

$ ember exam --random[=<seed>]

The random option allows you to randomize the order in which your tests run. You can optionally specify a "seed" value from which to randomize your tests in order to reproduce results. The seed can be any string value. Regardless of whether you specify a seed or not, Ember Exam will log the seed value used for the randomization at the beginning of the test run:

$ ember exam --random
$ Randomizing tests with seed: liv5d1ixkco6qlatl6o7mbo6r

$ ember exam --random=this_is1337
$ Randomizing tests with seed: this_is1337

Note: You must be using QUnit version 1.23.0 or greater for this feature to work properly.

$ ember exam --split=<num>

The split option allows you to specify a number of partitions greater than one to spread your tests across. Ember Exam will then proceed to run the first batch of tests.

$ ember exam --split=<num> --partition=<num>

The partition option allows you to specify which test group to run after using the split option. It is one-indexed, so if you specifiy a split of 3, the last group you could run is 3 as well.

$ ember exam --split=<num> --parallel

The parallel option allows you to run your split tests across multiple test pages in parallel in Testem. It will use a separate browser instance for each group of tests. So, if you specify a split of 3, then 3 browser instances will be spawned with the output looking something like:

ok 1 PhantomJS 1.9 - Exam Partition #1 - some test
ok 2 PhantomJS 1.9 - Exam Partition #3 - some other other test
ok 3 PhantomJS 1.9 - Exam Partition #2 - some other test

Ember Exam will respect the parallel setting of your Testem config file while running tests in parallel. Note that the default value for parallel in Testem is 1, which means you'll need a non-default value to actually see parallel behavior.

Note: You must be using Testem version 1.5.0 or greater for this feature to work properly.

Integrating ember-exam with ember-try is remarkably easy. Simply define a command in your ember-try.js config that leverages the exam command:

// config/ember-try.js
module.exports = {
  command: 'ember exam --split 3 --parallel',
  // ...
};

Using environmental variables gives you flexibility in how you run your tests, for instance you could also distribute your tests across processes instead of parallelizing them:

module.exports = {
  command: 'ember exam --split 20 --partition ' + process.env.PARTITION,
  // ...
};

If you are working with Travis CI then you can also easily set up seeded-random runs based on PRs. Similar to the following:

var command = [ 'ember', 'exam', '--random' ];
var pr = process.env.TRAVIS_PULL_REQUEST;

if (pr) {
  command.push(pr);
}

module.exports = {
  command: command.join(' '),
  // ...
};