"Watch your sims simmer on the GPU burners.."
Simmer provides a Cogent Lab based GUI for running simulation jobs, and managing and analyzing the resulting data, using the plotcore Editor. It supports both slurm and baremetal job management systems.
The terminal at the bottom runs the interactive goal shell, in the app, with the data and the Simmer itself (as lab.LabBrowser), so you can work with them directly there.
Simmer works on one simulation project at a time, which you open from the Open recent menu, or the list of recent projects that it shows when it starts without one, or with Open. Each project opens in its own simmer app, so you can run the app once, and open the projects of different simulations. Running simmer in the directory of a project opens it.
The settings of a project are in the simmer.json file in the directory of the simulation, which Edit project edits and saves. They include the Server.SSH user@host of the server, which is connected as @1 in goal commands, and connected again after the network is lost.
New project makes a new project for the simulation in a given directory: the local data goes into paths like this:
~/simdata/simname/user with jobs and labscripts subdirectories
with a simdata symbolic link to it in the directory of the simulation, and a simmer.json file.
The server data goes into similar paths but starting with ~/simserver instead.
The slurm backend uses array jobs to distribute computation over many CPU nodes. See slurm.go for the relevant code.
If not using slurm, then the baremetal system manages jobs itself, maintaining a record of the jobs launched relative to the resources available. Simmer auto-launches and reconnects with baremetal, which runs in its own terminal window.
baremetal can be run on localhost to manage jobs on your own local machine, using this config:
[[Servers.Values]]
Name = "lh"
SSH = "localhost"
NGPUs = 4-
Jobsshows current jobs in a Jobs tab. -
Submitruns a new sim job. UseMessagefor general, more stable info, andLabelfor a short unique identifier for this specific job. -
Statusgets status of any running jobs. Anything done running gets status ofFinalizedand is no longer updated by Status. All job metadata is downloaded from host, but not the Results output data, which isFetched separately because it may be large and often needs to be consolidated in a particular way, because multiple runs of a job are executed in parallel. -
Fetchgets result.tsvfiles from server. It can be run on running or Finalized jobs. When run on Finalized, then the status is set toFetchedand it is automatically skipped in any future Fetch actions. -
Resultsgrabs specific result data files into aResultstab, from which further examination and plotting occurs. This step is necessary because there are typically multiple different types of results files, so you need to select which type you want view. -
Plotplots combined data across any selected files inResultstab, allowing you to compare them, using theJobLabelas a legend so each job has its own line color. -
Diffshows a diff browser for any two selected Jobs, or one selected job vs. the current sim working directory -- key for tracking changes made across jobs.
- consat: plot mean function on test epoch is losing the TrainEpoch fields