• Introduction
• Quickstart
• Demo case study
• Bundled scheduler case studies
• Output walkthrough
• Watchmode
• Instant Replay
• Customisation
• Command-line arguments
• Usage tips

qtop's Python version is based on the original qtop bash script written by @fgeorgatos long ago. The main reason for rewriting it from scratch in Python was the drive to add some much-needed functionality, which was getting harder and harder to do in bash. The Python version is much more extensible and can support more Local Resource Management Systems, such as Oracle/Sun Grid Engine (SGE/GE), LSF, LoadLeveler, OAR, SLURM etc. It's also less buggy, with most of the bugs of the old version eradicated ...and new ones introduced ;-)

qtop summarizes, textually and visually, the state of a Resource Management System, along with relevant cluster information. It gathers information from the Resource Management System and organises it in a compact yet illustrative format for easy access. Expandability and configurability were deemed very important and were taken into account from the very early stages of designing qtop. What's more, it is intended that the information gathered by qtop can be reused by other developers or system administrators, either for other purposes or in different visualisation formats/media. For this reason, qtop exports the gathered information to a convenient JSON format. The project also goes to great lengths to keep qtop dependency-free. That means, you don't have to download or install anything extra; qtop works out of the box!

The absolute simplest way to get qtop to display anything is by invoking it in demo mode:

./qtop -b demo

When used, the -b switch must always be followed by one of the supported batch systems (pbs, sge, oar, slurm, demo).

That should create and destroy fictional jobs in fictional machines from fictional users, and display all that in a colorful but not yet very helpful way. The data produced will probably be more than a screenful, so most of it is going to scroll up and away. If you want to start making any sense of it, you're better off trying this:

./qtop -b demo -w

which keeps updating the screen with freshly made-up data, and also keeps you at the beginning of it. This is called the watch mode. You'll notice that this time it's the end that you cannot see, so now would be a good time to introduce keybindings to move around.

Most of what you can see on screen is customisable on-the-fly by editing a configuration file. Some modifications can also be accomplished by using the aforementioned keybindings.

Use demo mode when you want to learn qtop's display without access to a live PBS, SGE, OAR, or Slurm cluster. The demo backend generates fictional worker nodes, jobs, queues, and users, so it is useful for checking the layout and keyboard workflow before pointing qtop at production scheduler data.

Start with a one-shot render:

./qtop -b demo

Then try the same input with common display switches:

./qtop -b demo -FT

This combines two useful view changes:

• -F shows full worker node names instead of compact numbers.
• -T transposes the worker-node matrix so nodes are rows instead of columns.

For a large demo view, let the output overflow horizontally and inspect it with less:

./qtop -b demo -l | less -RS

For an interactive session, use watch mode. The optional value after -w is the refresh interval in seconds:

./qtop -b demo -w 10

While in watch mode, the movement keys from Keyboard shortcuts let you move through the matrix, transpose it, switch node ID display, apply filters, or quit.

The qtop_py/contrib directory contains captured scheduler snapshots for quick checks without access to a live cluster:

./qtop -s qtop_py/contrib -c ON -Fadvv -b sge
./qtop -s qtop_py/contrib -c ON -Frdvvv -b oar
./qtop -s qtop_py/contrib -c ON -raF -b pbs

Use -s for saved scheduler files and -b for the matching parser.

The information that qtop conveys when run can be divided into three sections:

• number of nodes (Total:available (online):Free (non-exclusive))
• number of cores (available/total)
• number of jobs (running/queued)
• queue names, along with running/queued jobs info for every queue

Accounting Summary

If a queue is denoted as blocked in the Resource Management System, it will be marked with an asterisk.

The heart of qtop lies in this sector, which is essentially a matrix (potentially wrapped at the end of the terminal, resulting in a series of matrices), modular in nature. There are two ways to visualise the matrix, one where the columns are worker nodes and the rows are the node characteristics, and another where the matrix is transposed, i.e. the rows are the worker nodes and the columns are their characteristics. We'll be referring to the first form in our examples, which was also the original form used in old qtop.

The default characteristics are the node state, the names of the queues involved and the job per core allocation table.

This, however, is customisable: as long as there are characteristics of interest existing in the Resource Management System input file, the User should be able to describe the characteristic inside the config file and have a new vector spawn inside the matrix.

One dimension is used to array the worker nodes and another to expand the worker node characteristics (individual cores, node state, associated queues etc).

The default elements displayed are:

Worker node ID (virtual or real node name)

worker nodes ID

The options here are either the real full name of the worker nodes, or a numbering scheme. If the worker node numbering is atypical, e.g. from pc23 there's a jump to pc-b-145, the standard representation scheme will represent such gaps with non-existent worker nodes, which might not always be desirable. Hence, a remapping mechanism will be put in place, giving consistent numbering starting from zero, to all nodes belonging to the cluster.

Node state

Node state

The node state is denoted with the first letter of the state. For PBS, for example:

• j: job-exclusive
• b: busy
• o: offline
• d: down

The symbol "-" is used when the node is free.

Queue names

Queue name

The queue names sector shows the initials of the queue names loaded with jobs running on the specific worker node. The full names of the queues show up earlier in the Accounting Summary, color-coded in the same way. So if you have queues all starting with the same letter, e.g. grid3000M, grid2000M, grid5000M, the colouring will help you see which is which.

Job allocation table

Cores

The jobs are displayed via their owner, who is given a single-character ID. The mapping is shown on the User account information section. An underscore '_' denotes a free core. A hash symbol '#' means that the core in this position does not exist on this worker node. If, for example, a hundred-node cluster consists of 96 worker nodes of 32 cores and 4 worker nodes of 16 cores, the last 16 core position in those 4 worker nodes will be substituted with the hash symbol.

user accounts

• id is a symbol denoting a unique unix account
• the jobs of every user, with distinct mention for the running jobs vs. the total number of jobs (including the finished ones)
• the name of the unix account
• Grid certificate DN, or GECOS field

The former is useful in Grid-connected clusters. It shows the owner of the pool account at the current time.

Note: This can only be shown for root accounts, because of the dependence on edg-mkgridpool --list

The GECOS field either retrieves information from a cached file (described in the config file) or gets information in realtime from getent passwd

The watch mode is a step-up from the original implementation, where the user was able to run qtop with watch -d prepended, so as to be able to watch an almost real-time evolution of the system monitored. Alas, there was no way to retain qtop's coloring information with watch (not in older versions of watch, anyway). An in-house implementation of watch was hence inspired, and along with that came new features that original watch doesn't provide, giving interactivity to the user, in an ncurses-like fashion.

The options available in watch mode, as of version 0.8.9 are:

• navigation through a big matrix, as if more/less were used (more or less ;) with vim-like commands
• matrix transposition (switch between rows<->columns)
• node ID type switch (from proper full node name to bare numbering)
• coloring code switch (user id coloring/queue coloring)
• real-time filtering of nodes according to name substring or RegEx/node state/node number

The output of qtop is kept in /tmp/qtop_results_$USER (configurable in qtopconf.yaml), so that users can "replay" the state of their system from a particular point in time. By default, the output is kept for the last 24 hours, but this is also configurable in qtopconf.yaml.

To replay from a specific point in time, the User must invoke:

./qtop -R DATETIME [DURATION]

where DATETIME can be one of:

• yyyymmddTHHMMSS, e.g. 20161118T182300 (explicit form)
• HHMM, e.g. 1823 (current day is implied, or the previous day, depending)
• mmddTHHMM, e.g. 1118T1823 (current year is implied)

Colons, slashes and dashes are also supported, so the User can type, e.g. 18:23, or 11/18T18:23, instead.

A second value, DURATION, is optional and denotes the desired length of the playback. The notation for duration is X<unit>, where unit can either be h, m, s. Example:

./qtop -R 1823 1h

A default duration of 2m is used, if no value is given.

Important: What is displayed on screen during instant replay is what was displayed at that particular moment. If the User was navigating up and down while using filters, this is what is going to be displayed.

Customisation is achieved by means of editing a configuration file. Modifying most of the keys available in the configuration file can affect the behaviour of qtop in a live manner, if running in watch mode.

YAML was chosen as it is human-readable and should thus be easier for people to read and modify to their liking. That said, the configuration file is in a YAML-like format. It does not faithfully follow the format 100%, as it was written quickly. It was decided against using the standard PyYAML module because qtop should keep its runtime dependency surface minimal. Users often run qtop on early or bare HPC systems where extra Python packages, internet access, or administrator-managed Python stacks may not be available.

This has, unfortunately, resulted in some "quirks" in the YAMLish format now used, such as:

• having to end single-item lists in a comma, e.g. ['^wn-26',]
• two consecutive unindents not being allowed. There must always exist a once-unindented key-value after nested keys
• not being able to write arbitrarily deeply nested dictionary-like structures, without fear it's going to break the universe.

Anyone brave enough to dive into yaml_parser.py, or better still, rewrite the darn thing, is more than welcome to, by the way. It goes without saying that only a very basic implementation is required (what you can see used in the configuration file, really!) and none of the flashy features that YAML sports.

---
    savepath: /tmp/qtop_results_$USER
    schedulers:
      pbs:
        pbsnodes_file: %(savepath)s/pbsnodes_a%(pid)s.txt, pbsnodes -a
        qstatq_file: %(savepath)s/qstat_q%(pid)s.txt, qstat -q
        qstat_file: %(savepath)s/qstat%(pid)s.txt, qstat
      oar:
        oarnodes_s_file: %(savepath)s/oarnodes_s_Y%(pid)s.txt, oarnodes -s -Y
        oarnodes_y_file: %(savepath)s/oarnodes_Y%(pid)s.txt, oarnodes -Y
        oarstat_file: %(savepath)s/oarstat%(pid)s.txt, oarstat
      sge:
        sge_file: %(savepath)s/qstat%(pid)s.F.xml.stdout, qstat -F -xml -u '*'
      slurm:
        squeue_file: %(savepath)s/squeue%(pid)s.txt, squeue -h -o %%i|%%u|%%t|%%P|%%C|%%N
        sinfo_file: %(savepath)s/sinfo%(pid)s.txt, sinfo -N -h -o %%N|%%P|%%t|%%c
      demo:
        demo_file: %(savepath)s/demo%(pid)s.txt, echo 'Demo here'
---

There are two ways to run qtop, data-wise:

1. run it on a batch controller (Computing Element), or similar host, where qtop itself invokes the commands needed to gather the cluster information
2. run it wherever, after the cluster information has been stored locally for qtop to read

The second mode is handy for off-line debugging of qtop, running it in GitHub Codespaces, or using bare Linux containers on a laptop while away from the cluster.

In this area of the configuration file, two key elements are set for each scheduler:

• the needed shell commands to run on the Computing Element
• the filepath where the cluster information is to be saved

So, let's say your nick is Mike and you are administering an OAR cluster. If you just invoke:

./qtop -b oar

then qtop is going to invoke oarnodes -s -Y, oarnodes -Y, oarstat consecutively, and store it, respectively, in

/tmp/qtop_results_mike/oarnodes_s_Y.txt
/tmp/qtop_results_mike/oarnodes_Y.txt
/tmp/qtop_results_mike/oarstat.txt

If, instead, you invoke it as follows:

./qtop -b oar -s <path-to-cluster-information>

qtop will search for oarnodes_s_Y.txt, oarnodes_Y.txt and oarstat.txt in <path-to-cluster-information> (retrieved by you, earlier).

For Slurm, qtop expects saved squeue and sinfo command traces matching the configured commands. For example:

./qtop -b slurm -s <path-to-slurm-command-traces>

qtop will search for squeue.txt and sinfo.txt in <path-to-slurm-command-traces>.

qtop also has a scheduler-type discovery system, meaning it will try to guess which scheduler system is installed in your system. The keys below let the user decide which command it should be that uniquely characterises the scheduler. If, for example, qtop can successfully find qacct on the system where qtop is executed, it will decide SGE is installed.

---
    signature_commands:
      pbs: pbsnodes
      oar: oarnodes
      sge: qacct
      slurm: sinfo
      demo: echo
---

---
    faster_xml_parsing: False
---

As the input data coming from SGE systems is in XML format, the natural choice for parsing XML data in python is the lxml module. That, however, is not in the standard library, and thus requires an extra download. For users that are unable/unwilling to install extra modules in their systems, the standard library slower alternative is used instead.

---
    # Meaning of queue state abbreviations
    state_abbreviations:
      pbs:
        Q: queued_of_user
        R: running_of_user
        C: cancelled_of_user
        E: exiting_of_user
        W: waiting_of_user
      oar:
        E: Error
        F: Finishing
        S: Resuming
        H: Hold
        L: waiting_of_user
        W: queued_of_user
        R: running_of_user
        T: exiting_of_user
        S: cancelled_of_user
      sge:
         etc etc
      slurm:
        R: running_of_user
        RH: hold_of_user  ## requeued_hold_of_user

---

Since every scheduler has its own set of states, here is the place where you can inform qtop what states your particular scheduler uses. For now, these are not linked to any special behaviour, but this should change in the future.

---
    user_color_mappings:  # order should be from more generic-->more specific
    # - \w+: Gray_D  # enabling this cancels all saved accounts in colormap.py
    # - [\d]+: Gray_D
    # - [A-Za-z]+0\d{2}: Red_L
    # - [A-Za-z]+15\d: Gray_D
    # - \w*lhc\w*: Cyan_L
    # - \w+00\d: Blue
    # - \w*cms\w*: Red_L
     - \w*cms048: Blue
     - \w*cms193: Gray_L
     - \w*atl\w+: Red
     - snielsen: Blue
     - ekalesaki: Cyan_L
     - zzhang: Red_L
     - mmravlak: Red_L
     - dbobbili: Cyan_L
     - patls021: Cyan_L

    queue_color_mappings:
      - alice: Red_L
      - dteam\w+: Cyan_L

    nodestate_color_mappings:
     - au: BlackOnRed
     - d: Red_LOnGrayBG
---

Color mappings apply for now to three types of items:

• user ids
• queue name initials, in the queue name lines/columns
• node state

Color mappings can be described using regular expressions. The standard Python regular-expression module re is used for parsing. Order here matters, so the expressions on the top of the list get overwritten by expressions on the bottom.

Colors with backgrounds can also be used: for all the color combinations available, check color_to_code dictionary in colormap.py.

In there, you will also find a ton of ready-made colormaps (userid_pat_to_color_default dictionary), with a primary focus on user ids found in Large Hadron Collider related clusters (WLCG grid).

The command shown in the README, ./qtop, is the project wrapper script. Installed packages expose the same entry point as qtop. Run ./qtop --help to see the authoritative option list for your checkout.

• -b SYSTEM selects the scheduler backend. Common values are demo, pbs, sge, oar, and slurm.
• -s DIRECTORY reads scheduler output files from DIRECTORY instead of running scheduler commands live.
• -f FILE uses a custom qtop configuration file.
• -o OPTION overrides an option from qtopconf.yaml.

• -1 hides the accounting summary section.
• -2 hides the worker-node occupancy section.
• -3 hides the user accounts and pool mappings section.
• -F forces full worker node names.
• -G resolves user details with getent passwd.
• -T transposes the worker-node matrix.
• -l lets the matrix overflow horizontally, which is useful with less -RS.
• -m disables masking of early empty worker nodes.
• -r removes rows made only of absent core markers.
• -rr also removes rows made of free, unused cores.
• -c MODE sets color mode to ON, OFF, or AUTO.

• -w [SECS] enables watch mode. It refreshes every SECS seconds, or every 2 seconds when omitted.
• -R WHEN replays saved qtop output from a point in time.
• -E exports gathered cluster data to JSON.
• -O saves output without printing the rendered view to stdout.
• -B enables the web interface on port 8080.

• -v increases verbosity. Repeat it for more detail.
• -d prints debug messages to stdout in addition to the log file.
• -S strictly compares reported running jobs against jobs displayed by qtop.
• -L creates a sample archive for reports. Use -LL to also include qtop's config and code.
• -A anonymizes account and worker node names. The help output currently marks this as temporary and not ready for scheduler input files.
• -V prints the qtop version.
• -h shows command help.

• Start with ./qtop -b demo when testing terminal size, color, or keyboard navigation. It avoids accidental scheduler calls.
• On a live cluster, pass -b pbs, -b sge, -b oar, or -b slurm when auto-detection is not enough.
• Use -s when another process has already collected scheduler output files. This is helpful for debugging and for reproducing an issue away from the cluster login node.
• Combine -l with less -RS when the worker-node matrix is wider than the terminal.
• Use -L when preparing a bug report. Review the archive before sharing it, especially when scheduler output might include sensitive hostnames or account names.