This example project demonstrates a best practices and state of the art features for hermetic end to end web app testing:

It is set up to test a Django "to do" project (in the app folder) but it is designed to such that you can swap out the django app with another containerized app easily. The code in the tests/ folder can remain the same.

Basic:

• One step set up: ./run.sh make.
• Absolute and total environmental consistency and portability (Mac/WSL/Linux) via podman-in-podman containerization and dependency pinning.
• Every project task runnable via one script (./run.sh) - building, running tests, generating docs, re-pinning dependencies.
• Integrated debugging: show all logs for the app under test as well as crystal clear stack traces.
• Easy in-test REPL debugging (via ipython).

Browser testing:

• Built in snapshot testing on browser screenshots.
• Tests can be easily run in headless mode or with VNC so you can interact with a GUI.
• Multibrowser built in - use "given a chromium browser" to "given a firefox browser" on the story and both just work.
• No sleeps only explicit waits.
• On test failures: save artefacts - a screenshot, a video of the failed test, the HTML at the point of failure and a "diff" screenshot in the event of a snapshot failure.

Database:

• Database fixtures on Postgres which can be trivially configured in-story (currently tied to Django, but the code can be changed to use a different database quite easily).
• Story inheritance with fixture merging - the pre-existing todos in the "Login" story are combined with the todos in the "add and retrieve todo" story that inherits from it.
• Database fixtures are cached as podman volumes after being built for faster subsequent test runs.
• Faster testing with postgres by turning data safety features off (don't do this in prod!).

Miscellaneous:

• Simple 3 step github actions config to run all of the tests (& upload any artefacts).
• Explicitly waiting for ports to open before kicking off the test (eliminates a common source of flakiness).
• coverage.py is preconfigured to spit out XML and HTML reports.

State of the art features enabled by hitchstory:

• Type-safe high level specifications in StrictYAML to maintain a separation of concerns between specification and test execution code - all the simplicity and readability of YAML, with none of the headaches.
• Autogeneration of beautiful markdown documentation based upon storytests - complete with screenshots and playable GIFs of the story.
• Storytests which can be programmed to rewrite themselves from program output (see example below).

It is a work in progress. These are more things I'll be adding soon:

• Snapshot testing of emails sent via SMTP.
• Add selectors specified in the story (hidden in the docs). Demonstrate "autogenerated" selectors alongside custom selectors.
• Aggressive caching on github actions while keeping the config as simple as possible.
• Aggressive parallelization end to end test runs on github actions while keeping the config as simple as possible.
• Proxied external API calls in live and recorded mode (via mitmproxy).
• Fixtures that include arbitrary data from python's faker that can still be used for snapshot testing - e.g. "given: 10 arbitrary users".

Podman must be installed on your system first.

Followed by:

$ git clone https://github.com/hitchdev/hitchstory.git
$ cd hitchstory/examples/website
$ ./run.sh make     # builds one local container and volume, with containers inside it

Once ./run.sh make has completed successfully you can start running tests.

The correct my spelling test checks that an error message appears:

Correct my spelling:
  ...
  steps:
  ...
  - should appear:
      text: Did you mean 'buy bread'?
      on: error

If you try changing this error message in the code.

{
    "error_message": "Did you mean '{}'?".format(
        correct_spelling(title)
    )
},

To this:

{
    "error_message": "Are you sure? Did you mean '{}'?".format(
        correct_spelling(title)
    )
},

And run the test that covers it in rewrite mode:

$ STORYMODE=rewrite ./run.sh pytest -k test_correct_my_spelling

This will update the text in the step. e.g.

  - should appear:
      on: error
      text: Are you sure? Did you mean 'buy bread'?

It will also re-record the video, re-take the story screenshots and regenerate the corresponding markdown documentation for the story.

The code that rewrites the story is in the should_appear method in test_integration.py.

$ ./run.sh pytest

This runs "Add and retrieve todo" from story/add-todo.story in normal mode:

$ ./run.sh pytest -k test_add_and_retrieve_todo

This can be useful for debugging a test, using right-click-inspect element or doing some exploratory QA at any point in the story:

$ STORYMODE=vnc ./run.sh pytest -k test_add_and_retrieve_todo

It will pause and launch an ipython prompt once the test passes or fails.

This will generate XML and HTML reports in the artefacts directory.

$ STORYMODE=coverage ./run.sh pytest

If a running test freezes and ctrl-C doesn't work, you can run this command to stop the container:

$ ./run.sh kill

This will regenerate all of the markdown docs for the project from the stories.

$ ./run.sh python tests/docgen.py

There are probably a few ways to do this. The one I find the most useful is to embed ipython in a step like so __import__("IPython").embed(). At this point everything on the Engine is accessible via self - e.g. self.given['browser'].

You can also put a - pause step in any of the stories and it will do the same thing.

You can use this to experiment with playwright until you've got the code you need to put into an actual step.

I find the easiest way with this set up is to import web_pdb ; web_pdb.set_trace() and then to view https://localhost:5555/ with a browser.

Everything runs in one podman container and volume. This deletes them:

$ ./run.sh clean all

These integration tests are run via github actions on every push. The Github Actions YAML is strictly kept as simple as possible so that it doesn't have to be debugged.

See here:

• Github actions YAML
• Runner

The tests in this project are run from a single podman container. The playwright container and the website are run in a container run inside that container:

graph TD;
    TestsContainer-->PodmanCompose;
    PodmanCompose-->WebappContainer;
    PodmanCompose-->PlaywrightContainer;
    PodmanCompose-->PostgresContainer;

This keeps the environment running the testing code completely consistent across environments and enables some sophisticated caching to speed up builds.