We occasionally run into serious problems with banzai deployments even after all unit and functional tests pass. This can happen due to any number of features, but often by the time an issue has been noticed, it has affected the entire network. Here I'm suggesting we perform a live end-to-end test for a single active telescope in each class on a dedicated test worker before deploying any banzai changes to the entire network.

• set up test worker(s)
• choose instruments (make them easily alterable by science support in case of weather)
• set up test-deployment branch or alpha release workflows
• establish persistent logs for test worker(s)
• create robust data quality checklist for test data