We should periodically (e.g. once a day?) run a set of sample circuits to track performance of qFlex, both in terms of memory cost and simulation runtime. Some options for this:

1. Run each circuit that has a hand-crafted ordering, at a reasonable depth (e.g. runtime < 30s)
   • Pros: using a fixed circuit set gives clearer comparisons between runs
   • Cons: might be biased towards these specific circuits
2. Generate circuits and orderings for each grid (via generator.py and order_circuit_simulation.py) and run several for each (e.g. 30 each, runtime < 1s)
   • Pros: more varied coverage for different circuit layouts (but still restricted to QS circuits)
   • Cons: more difficult to compare between runs, still biased towards QS circuits
3. Create a tool for generating truly random circuits (i.e. no fixed placement for 2-qubit gates) and follow the plan in (2), using the new tool to generate circuits
   • Pros: best option for covering various circuit layouts
   • Cons: time cost of writing tool, potential for untested layouts to misbehave (e.g. excessive runtime or memory usage), very difficult to compare between runs