Why?

This would be helpful for testing some types of programs:

• Programs that require a long-running server process to be running in the background, like web services.
• Interpreters that have a relatively long start up time. These could gain performance by feeding test code into a single process and evaling it, rather than starting one instance of the interpreter per test.

Current workaround

Use --jobs 1, and in your !program, start the backend if TXM_INDEX equals TXM_INDEX_FIRST, or stop it if TXM_INDEX equals TXM_INDEX_LAST.

For any other value of --jobs, this workaround introduces a race condition due to non-deterministic ordering.

How it could work

Before running the first test, start the program specified as --backend="program goes here" in the background. After the last test, terminate it.

If the backend program exits at any point, fail all so-far-incomplete tests. In the YAML for the failures, give the backend program's text, stdout, stderr, and exit code.

Limitations

Q: How do we know when the backend is truly "up" and ready to receive the first test?
A: We can't; it's application specific. We should document that it's the user's program's responsibility to deal with it.