fix: use base-2 mebibytes for memory requests based on LSF behavior - #9
fix: use base-2 mebibytes for memory requests based on LSF behavior#9acfoltzer wants to merge 1 commit into
Conversation
The LSF documentation is very confused and contradictory regarding whether memory requests and
reporting are in base-2 or base-10. Here is an adaptation of my notes from a private chat regarding
the behavior in practice on the St. Jude HPC:
---
LSF allows the following suffixes:
```KB or K (kilobytes)
MB or M (megabytes)
GB or G (gigabytes)
TB or T (terabytes)
PB or P (petabytes)
EB or E (exabytes)
ZB or Z (zettabytes)
```
Here's an example. 16 gigabytes should be 16000 megabytes or 16000000 kilobytes, right?
```
LSB_JOB_REPORT_MAIL=N bsub -oo lsf.out -eo lsf.err -R "rusage[mem=16000000KB/job]" sleep 20
```
```
Job <274348028>, User <afoltzer>, Project <default>, Status <RUN>, Queue <stand
ard>, Job Priority <50>, Command <sleep 20>, Share group c
harged </afoltzer>, Esub <hpcf>
Thu Nov 13 20:34:14: Submitted from host <noderome104>, CWD <$HOME>, Output Fil
e (overwrite) <lsf.out>, Error File (overwrite) <lsf.err>,
Requested Resources <rusage[mem=15625.00/job]>;
Thu Nov 13 20:34:20: Started 1 Task(s) on Host(s) <noderome209>, Allocated 1 Sl
ot(s) on Host(s) <noderome209>, Execution Home </home/afol
tzer>, Execution CWD </home/afoltzer>;
Thu Nov 13 20:34:28: Resource usage collected.
MEM: 7 Mbytes; SWAP: 0 Mbytes; NTHREAD: 4
PGID: 3985845; PIDs: 3985845 3985875 3985877
MEMLIMIT SWAPLIMIT
15.2 G 15.2 G
MEMORY USAGE:
MAX MEM: 8 Mbytes; AVG MEM: 4 Mbytes; MEM Efficiency: 0.05%
CPU USAGE:
CPU PEAK: 0.00 ; CPU PEAK DURATION: 0 second(s)
CPU AVERAGE EFFICIENCY: 0.00% ; CPU PEAK EFFICIENCY: 0.00%
SCHEDULING PARAMETERS:
r15s r1m r15m ut pg io ls it tmp swp mem
loadSched - 3.5 - - - - - - 1000M - -
loadStop - - - - - - - - - - -
waitpct cWs MBWps rgs01_load
loadSched 10.0 40.0 20000.0 130.0
loadStop - - - -
RESOURCE REQUIREMENT DETAILS:
Combined: select[(rhel8) && (type == any)] order[r15s:pg] rusage[mem=15625.00/
job] affinity[core(1)*1]
Effective: select[((rhel8) && (type == any))] order[r15s:pg] rusage[mem=15625.
00/job] affinity[core(1)*1]
```
Note in particular the mem and swap limit of `15.2 G` and the fact that the rusage string now reads
`mem=15625.00/job` (the default unit on _our_ cluster but not all LSF clusters is `MB`). So, whoops!
I misremembered: sometimes it looks like you've been given less than what you asked for, but that's
because it's reporting in base 2!
Now let's try 16777216 kibibytes, which is 16384 mebibytes or 16 gibibytes.
```
LSB_JOB_REPORT_MAIL=N bsub -oo lsf.out -eo lsf.err -R "rusage[mem=16777216KB/job]" sleep 20
```
There are 16777216 kibibytes in 16384 mebibytes or 16 gibibytes.
```
Job <274346639>, User <afoltzer>, Project <default>, Status <RUN>, Queue <stand
ard>, Job Priority <50>, Command <sleep 20>, Share group c
harged </afoltzer>, Esub <hpcf>
Thu Nov 13 20:30:10: Submitted from host <noderome104>, CWD <$HOME>, Output Fil
e (overwrite) <lsf.out>, Error File (overwrite) <lsf.err>,
Requested Resources <rusage[mem=16384.00/job]>;
Thu Nov 13 20:30:27: Started 1 Task(s) on Host(s) <noderome140>, Allocated 1 Sl
ot(s) on Host(s) <noderome140>;
MEMLIMIT SWAPLIMIT
16 G 16 G
MEMORY USAGE:
MEM Efficiency: 0.00%
CPU USAGE:
CPU PEAK: 0.00 ; CPU PEAK DURATION: 0 second(s)
CPU AVERAGE EFFICIENCY: 0.00% ; CPU PEAK EFFICIENCY: 0.00%
SCHEDULING PARAMETERS:
r15s r1m r15m ut pg io ls it tmp swp mem
loadSched - 3.5 - - - - - - 1000M - -
loadStop - - - - - - - - - - -
waitpct cWs MBWps rgs01_load
loadSched 10.0 40.0 20000.0 130.0
loadStop - - - -
RESOURCE REQUIREMENT DETAILS:
Combined: select[(rhel8) && (type == any)] order[r15s:pg] rusage[mem=16384.00/
job] affinity[core(1)*1]
Effective: select[((rhel8) && (type == any))] order[r15s:pg] rusage[mem=16384.
00/job] affinity[core(1)*1]
```
16 gibibytes is 16384 mebibytes, so if we give a base 2 memory constraint in rusage we do in fact
get a base 2 amount of memory reported by the system. Even though all the docs use the base 10 unit
names and suffixes.
|
Oddly, this does not appear to be the case when using |
|
Actually, I wonder if there is an issue in It does some additional checking and rewriting of the LSF parameters before passing the job to the scheduler. |
|
linking this PR for some extra weridness - stjudecloud/workflows#279
I saw differences between miniwdl and sprocket behavior regarding LSF memory that is not explained by base10->base2 conversion. I will point out that I've noticed that our LSF cluster is bad at spotting short spikes in memory usage. My understanding is that a running job can spike over it's "hard limit" without being noticed by LSF if the spike is short enough in duration. That may be at play here? |
|
Huh, I do agree that the behavior for I also looked through the The apparent difference with |
The LSF documentation is very confused and contradictory regarding whether memory requests and reporting are in base-2 or base-10. Here is an adaptation of my notes from a private chat regarding the behavior in practice on the St. Jude HPC:
LSF allows the following suffixes:
Here's an example. 16 gigabytes should be 16000 megabytes or 16000000 kilobytes, right?
Note in particular the mem and swap limit of
15.2 Gand the fact that the rusage string now readsmem=15625.00/job(the default unit on our cluster but not all LSF clusters isMB). So, whoops! I misremembered: sometimes it looks like you've been given less than what you asked for, but that's because it's reporting in base 2!Now let's try 16777216 kibibytes, which is 16384 mebibytes or 16 gibibytes.
There are 16777216 kibibytes in 16384 mebibytes or 16 gibibytes.
16 gibibytes is 16384 mebibytes, so if we give a base 2 memory constraint in rusage we do in fact get a base 2 amount of memory reported by the system. Even though all the docs use the base 10 unit names and suffixes.