I just spent nearly an hour working why a test worked in a local environment, but not remotely. Basically, it's decidedly non-obvious that system calls are run for real by default.
To take a simple example, if you write the following:
:echomsg system('uname')
! uname
$ Linux
~ Linux
... then you have a test that verifies that a system call is made to uname, and you've faked it to return "Linux". (Presumably in a real test, :echomsg system('uname') would actually be something more useful, and the uname command would be called from some internal implementation detail.)
If you accidentally forget the $ hijack line, then (assuming the command is valid on your machine), the tests will still pass, but now they'll be non-hermetic and will execute the command for real.
That might be good in some cases — it's effectively an integration test, and that can be valuable too — but it's far too easy to do by mistake.
I think a better option would be to require an explicit decision in the test to either mock each command (by providing a hijack for stdout and/or the exit status) or to opt-in to run the command for real.
Perhaps this might look something like:
:echomsg system('uname')
! uname (allow-execution-in-test)
~ Linux
#4 is also related, since currently Vroom allows unexpected commands to execute as well.
I just spent nearly an hour working why a test worked in a local environment, but not remotely. Basically, it's decidedly non-obvious that system calls are run for real by default.
To take a simple example, if you write the following:
... then you have a test that verifies that a system call is made to
uname, and you've faked it to return "Linux". (Presumably in a real test,:echomsg system('uname')would actually be something more useful, and theunamecommand would be called from some internal implementation detail.)If you accidentally forget the
$hijack line, then (assuming the command is valid on your machine), the tests will still pass, but now they'll be non-hermetic and will execute the command for real.That might be good in some cases — it's effectively an integration test, and that can be valuable too — but it's far too easy to do by mistake.
I think a better option would be to require an explicit decision in the test to either mock each command (by providing a hijack for stdout and/or the exit status) or to opt-in to run the command for real.
Perhaps this might look something like:
#4 is also related, since currently Vroom allows unexpected commands to execute as well.