A tiny shell-script based TAP-compliant testing framework

 ______   ______     __  __     ______  
/\__  _\ /\  ___\   /\ \_\ \   /\__  _\ 
\/_/\ \/ \ \___  \  \ \  __ \  \/_/\ \/ 
   \ \_\  \/\_____\  \ \_\ \_\    \ \_\ 
    \/_/   \/_____/   \/_/\/_/     \/_/ 

• Installation
  • Per project
  • Per machine
• Usage
  • Specs
  • CLI
  • Extensions
  • Hooks
  • Example
  • Vim integration
  • Pretty Output
• API - core
  • plan
  • fail
  • pass
  • exec_fail
  • exec_ok
  • ok
  • not_ok
  • use
• API - file
  • file_exists
  • not_file_exists
  • not_file_empty
  • equals_file
  • equals_file_file
• API - string
  • equals
  • not_equals
  • match
  • not_match
• API - jq
  • jq_ok
  • jq_ok
• API - diff
  • equals (colordiff)
• API - colordiff
  • equals (colordiff)
• API - shxml

In general you don't have to install tsht, simply add the wrapper script to your project.

1. Create a test directory, e.g. test
2. Download the wrapper script cd test && wget 'https://cdn.rawgit.com/kba/tsht/master/tsht'
3. Create your unit tests
4. Execute all tests using ./tsht or specific tests using ./tsht <path/to/unit-test.tsht>...

The first time you execute the wrapper script, it will clone this repository to .tsht and execute the runner. Whenever you want to update the tsht framework, simply delete the .tsht folder and an up-to-date version of the framework will be cloned when you next run your tests.

To execute tsht scripts without the runner, you will need to have tsht in your $PATH. You can either set $PATH up manually to include the directory that contains the tsht wrapper or clone this repository and use the Makefile.

To install system-wide:

sudo make install

To install to your home directory:

make PREFIX=$HOME/.local install

Tsht unit tests are written in a DSL superset of the Bash shell scripting language. This means that any bash code can be used in a tsht script.

All tsht scripts must end with .tsht.

All tsht scripts should start with #!/usr/bin/env tsht

Tsht scripts are executed in alphabetic order, so prefix the scripts you want to run early with a low number.

Usage: tsht [options...] [<path/to/unit.tsht>...]
    Options:
        --help     -h   Show this help
        --color         Highlight passing/failing tests in green/red
        --update        Update the tsht framework from git
        --version  -V   Show last revision of the runner

In addition to the core functionality, tsht can be extended with extensions. An extension is a subdirectory of tsht that has this structure:

/ext/<name>
ext/
└── <name>
    ├── Makefile
    └── <name>.sh

The Makefile must have an install target that can install necessary binaries into $(PREFIX)/bin.

To use an extension, call the use directive:

use 'jq'

This will call make install in the extension directory and set the PATH variable to let the extension use the locally installed software.

Currently, these extensions are available:

• jq (API, test): A wrapper around the jq CLI JSON query tool
• diff (API, test): Show differences with color-highlighted diff
• diff (API, test): Show differences with diff
• shxml (API, test) Use XML based tools in tests (XSD, XSD, XPATH…)

Some unit tests require setup work before they run and teardown work after they run. To make this easier and reusable, these tasks can be grouped in before and after hooks, which are shell scripts or shell functions.

Hooks are always test-specific and are looked for in three places:

• Shell function after
• Shell scripts named like the test with suffix .before/.after
• Shell scripts in the same directory as the test named .before/.after

Note: before cannot be a shell function in the script because the script must be sourced to declare it and once it has been sourced, it's too late to call the before hook.

#!/usr/bin/env tsht

plan 4

equals $(( 84 / 2 )) 42 "three score and six"
exec_ok "ls /"
match "oo" "foobar"
not_ok $(( 0 / 42 ))  "Nothing divided is nothing"

If you are a vim user, try out the tsht.vim plugin which will detect .tsht files, highlight the builtin functions and execute scripts with the closest wrapper.

There are various TAP consumers that can produce nice output, the tape NodeJS TAP-based framework lists a few.

For example, using the tap-spec TAP reporter can be installed using

npm install -g tap-spec

For the tests of tsht itself, it will produce output like this:

$ ./test/tsht | tap-spec

  Testing ./runner/update/update.tsht

    ✔ Executed: git clone ../../../ /home/kb/build/github.com/kba/tsht/test/runner/update/test-project/.tsht
    ✔ Executed: git checkout master
    ✔ Executed: git reset --hard bd9fbafa643f10087cb24ff0f3b47a9d33a12a26
    ✔ HEAD is bd9fbafa643f10087cb24ff0f3b47a9d33a12a26
    ✔ Executed: ./tsht --update
    ✔ HEAD is 81ce4a381d416855eb37b99a087e8d01edae661c

  Testing ./runner/help/help.tsht

    ✔ Executed: /home/kb/build/github.com/kba/tsht/test/.tsht/tsht-runner.sh --help
    ✔ Executed: /home/kb/build/github.com/kba/tsht/test/.tsht/tsht-runner.sh -h
    ✔ -h == --help
    ✔ Matches '--color': 'Usage: tsht [options...] [...]    Options:        --help     -h   Show this help        --color         Highlight passing/failing tests in green/red        --update '
    ✔ Matches '--update': 'Usage: tsht [options...] [...]    Options:        --help     -h   Show this help        --color         Highlight passing/failing tests in green/red        --update '
    ✔ Matches '--version': 'Usage: tsht [options...] [...]    Options:        --help     -h   Show this help        --color         Highlight passing/failing tests in green/red        --update '
    ✔ Failed as expected (2) '/home/kb/build/github.com/kba/tsht/test/.tsht/tsht-runner.sh --foobar'

  Testing ./runner/color/color-test.tsht


  /home/kb/build/github.com/kba/tsht/test/.tsht/tsht-runner.sh --color thetest

    ✔ Color output as expected

  Testing ./runner/0tshtlib/tshtlib.tsht

    ✔ TSHTLIB is set
    ✔ TSHTLIB is relative to this dir
    ✔ /home/kb/build/github.com/kba/tsht/test/.tsht/tsht-runner.sh is the right tsht-runner.sh

  Testing ./file.tsht

    ✔ Failed as expected (2) 'ls does-not-exist'
    ✔ Executed: touch does-not-exist
    ✔ File exists: does-not-exist
    ✔ Not empty file: does-not-exist
    ✔ (unnamed equals assertion)

  Testing ./api/core/not_ok.tsht

    ✔ Empty string
    ✔ 0
    ✔ "0"

  Testing ./api/core/exec_fail.tsht

    ✔ Failed as expected (2) 'ls does-not-exist'

  Testing ./api/core/ok.tsht

    ✔ Me testing my existence

  Testing ./api/core/exec_ok.tsht

    ✔ Executed: touch does-not-exist

  Testing ./api/file/file_exists.tsht

    ✔ File exists: does-not-exist

  Testing ./api/file/file_not_empty.tsht

    ✔ Not empty file: does-not-exist

  Testing ./api/string/match.tsht

    ✔ Matches '^\d+': '1234'
    ✔ Matches '^\d+$': '1234'
    ✔ Matches '^a\d+$': 'a1234'

  Testing ./api/string/equals.tsht

    ✔ (unnamed equals assertion)
    ✔ (unnamed equals assertion)
    ✔ (unnamed equals assertion)
    ✔ (unnamed equals assertion)

  Testing ./api/string/not_match.tsht

    ✔ Not like '^\d+$': 'string'

  Testing ./api/string/not_equals.tsht

    ✔ 1984 test

  Testing ./ext/shxml/shxml-basic.tsht

    ✔ Executed: shxml --help

  Testing ./ext/colordiff/diff.tsht

    ✔ A colordiff is a colordiff is a colordiff

  Testing ./ext/jq/jq.tsht

    ✔ Executed: jq --version
    ✔ From string
    ✔ From STDIN (1)
    ✔ From STDIN (2)
    ✔ JSON: .foo.bar[1] -> '42'
    ✔ 

  Testing ./ext/diff/diff.tsht

    ✔ A diff is a diff is a diff

  Testing ./issues/issue_8.tsht

    ✔ Matches '--foo': '--foo'

  Testing ./before-after/ba2.tsht

    ✔ File does not exist: DELETEME

  Testing ./before-after/ba.tsht


  Sourcing suffix suffixscript ba.tsht.before for ba.tsht

    ✔ File exists: DELETEME

  Sourcing suffix suffixscript ba.tsht.after for ba.tsht



  total:     51
  passing:   51
  duration:  226ms

This library the core functions of tsht. It is always included and includes the most commonly used libraries:

• string
• file

Specify the number of planned assertions

plan <number-of-tests>

(source, test)

Fail unconditionally

fail <message> [<additional-output>]

The additional output will be prefixed with #.

Succeed unconditionally.

See fail

(source, test)

Execute a command (or function) and succeed when its return code matches the parameter

exec_fail <expected-return> [<cmd-args>...]

Example

exec_fail 2 "ls" "-la" "DOES-NOT-EXIST"

(source, test)

Execute a command (or function) and succeed when it returns zero.

Example

exec_ok "ls" "-la"

Succeed if the first argument is a non-empty non-zero string

Succeed if the first argument is an empty string or zero.

Use an extension library

use 'jq'

Succeed if a file (or folder or symlink...) exists.

file_exists ".git"

Succeed if a file (or folder or symlink...) does not exist.

not_file_exists "temp"

ALIAS: file_not_empty

Succeed if a file exists and is a non-empty file.

Succeed if the first arguments match the contents of the file in the second argument.

Succeed if the contents of two files match, filenames passed as arguments.

This library contains functions testing strings and numbers

Test for equality of strings

equals <expected> <actual> [<message>]

Example:

equals "2" 2 "two equals two"
equals 2 "$(wc -l my-file)" "two lines in my-file"

Inverse of equals.

Succeed if a string matches a pattern

match "^\d+$" "1234" "Only numbers"

Succeed if a string does not match a pattern

not_match "^\d+$" "abcd" "Only numbers"

Extension that allows testing JSON strings.

Enable with

use jq

See jq Github repo.

Test if jq expression validates

Test if jq expression is as exepected

Extension that replaces the builtin equals with a function that shows the difference as a unified diff

Enable with

use diff

Requires diff(1).

Test for equality of strings and output unified diff on fail.

equals <expected> <actual> [<message>]

Example:

equals "2" 2 "two equals two"
equals 2 "$(wc -l my-file)" "two lines in my-file"

Extension that replaces the builtin equals with a function that shows the difference in a colored diff output.

Enable with

use colordiff

Requires perl.

See colordiff Github repo.

Test for equality of strings and output colored diff on fail.

equals <expected> <actual> [<message>]

Example:

equals "2" 2 "two equals two"
equals 2 "$(wc -l my-file)" "two lines in my-file"

Extension that makes shxml available

Enable with

use shxml

See shxml Github repo.