Skip to content

fix: use base-2 mebibytes for memory requests based on LSF behavior - #9

Open
acfoltzer wants to merge 1 commit into
adthrasher:mainfrom
acfoltzer:acf/mebibytes
Open

fix: use base-2 mebibytes for memory requests based on LSF behavior#9
acfoltzer wants to merge 1 commit into
adthrasher:mainfrom
acfoltzer:acf/mebibytes

Conversation

@acfoltzer

Copy link
Copy Markdown

The LSF documentation is very confused and contradictory regarding whether memory requests and reporting are in base-2 or base-10. Here is an adaptation of my notes from a private chat regarding the behavior in practice on the St. Jude HPC:


LSF allows the following suffixes:

MB or M (megabytes)
GB or G (gigabytes)
TB or T (terabytes)
PB or P (petabytes)
EB or E (exabytes)
ZB or Z (zettabytes)

Here's an example. 16 gigabytes should be 16000 megabytes or 16000000 kilobytes, right?

LSB_JOB_REPORT_MAIL=N bsub -oo lsf.out -eo lsf.err -R "rusage[mem=16000000KB/job]" sleep 20
Job <274348028>, User <afoltzer>, Project <default>, Status <RUN>, Queue <stand
                     ard>, Job Priority <50>, Command <sleep 20>, Share group c
                     harged </afoltzer>, Esub <hpcf>
Thu Nov 13 20:34:14: Submitted from host <noderome104>, CWD <$HOME>, Output Fil
                     e (overwrite) <lsf.out>, Error File (overwrite) <lsf.err>,
                      Requested Resources <rusage[mem=15625.00/job]>;
Thu Nov 13 20:34:20: Started 1 Task(s) on Host(s) <noderome209>, Allocated 1 Sl
                     ot(s) on Host(s) <noderome209>, Execution Home </home/afol
                     tzer>, Execution CWD </home/afoltzer>;
Thu Nov 13 20:34:28: Resource usage collected.
                     MEM: 7 Mbytes;  SWAP: 0 Mbytes;  NTHREAD: 4
                     PGID: 3985845;  PIDs: 3985845 3985875 3985877

 MEMLIMIT SWAPLIMIT
   15.2 G    15.2 G

 MEMORY USAGE:
 MAX MEM: 8 Mbytes;  AVG MEM: 4 Mbytes; MEM Efficiency: 0.05%

 CPU USAGE:
 CPU PEAK: 0.00 ;  CPU PEAK DURATION: 0 second(s)
 CPU AVERAGE EFFICIENCY: 0.00% ;  CPU PEAK EFFICIENCY: 0.00%

 SCHEDULING PARAMETERS:
           r15s   r1m  r15m   ut      pg    io   ls    it    tmp    swp    mem
 loadSched   -    3.5    -     -       -     -    -     -  1000M     -      -
 loadStop    -     -     -     -       -     -    -     -     -      -      -

          waitpct     cWs   MBWps rgs01_load
 loadSched   10.0    40.0 20000.0      130.0
 loadStop      -       -       -          -

 RESOURCE REQUIREMENT DETAILS:
 Combined: select[(rhel8) && (type == any)] order[r15s:pg] rusage[mem=15625.00/
                     job] affinity[core(1)*1]
 Effective: select[((rhel8) && (type == any))] order[r15s:pg] rusage[mem=15625.
                     00/job] affinity[core(1)*1]

Note in particular the mem and swap limit of 15.2 G and the fact that the rusage string now reads mem=15625.00/job (the default unit on our cluster but not all LSF clusters is MB). So, whoops! I misremembered: sometimes it looks like you've been given less than what you asked for, but that's because it's reporting in base 2!

Now let's try 16777216 kibibytes, which is 16384 mebibytes or 16 gibibytes.

LSB_JOB_REPORT_MAIL=N bsub -oo lsf.out -eo lsf.err -R "rusage[mem=16777216KB/job]" sleep 20

There are 16777216 kibibytes in 16384 mebibytes or 16 gibibytes.

Job <274346639>, User <afoltzer>, Project <default>, Status <RUN>, Queue <stand
                     ard>, Job Priority <50>, Command <sleep 20>, Share group c
                     harged </afoltzer>, Esub <hpcf>
Thu Nov 13 20:30:10: Submitted from host <noderome104>, CWD <$HOME>, Output Fil
                     e (overwrite) <lsf.out>, Error File (overwrite) <lsf.err>,
                      Requested Resources <rusage[mem=16384.00/job]>;
Thu Nov 13 20:30:27: Started 1 Task(s) on Host(s) <noderome140>, Allocated 1 Sl
                     ot(s) on Host(s) <noderome140>;

 MEMLIMIT SWAPLIMIT
     16 G      16 G

 MEMORY USAGE:
 MEM Efficiency: 0.00%

 CPU USAGE:
 CPU PEAK: 0.00 ;  CPU PEAK DURATION: 0 second(s)
 CPU AVERAGE EFFICIENCY: 0.00% ;  CPU PEAK EFFICIENCY: 0.00%

 SCHEDULING PARAMETERS:
           r15s   r1m  r15m   ut      pg    io   ls    it    tmp    swp    mem
 loadSched   -    3.5    -     -       -     -    -     -  1000M     -      -
 loadStop    -     -     -     -       -     -    -     -     -      -      -

          waitpct     cWs   MBWps rgs01_load
 loadSched   10.0    40.0 20000.0      130.0
 loadStop      -       -       -          -

 RESOURCE REQUIREMENT DETAILS:
 Combined: select[(rhel8) && (type == any)] order[r15s:pg] rusage[mem=16384.00/
                     job] affinity[core(1)*1]
 Effective: select[((rhel8) && (type == any))] order[r15s:pg] rusage[mem=16384.
                     00/job] affinity[core(1)*1]

16 gibibytes is 16384 mebibytes, so if we give a base 2 memory constraint in rusage we do in fact get a base 2 amount of memory reported by the system. Even though all the docs use the base 10 unit names and suffixes.

The LSF documentation is very confused and contradictory regarding whether memory requests and
reporting are in base-2 or base-10. Here is an adaptation of my notes from a private chat regarding
the behavior in practice on the St. Jude HPC:

---

LSF allows the following suffixes:

```KB or K (kilobytes)
MB or M (megabytes)
GB or G (gigabytes)
TB or T (terabytes)
PB or P (petabytes)
EB or E (exabytes)
ZB or Z (zettabytes)
```

Here's an example. 16 gigabytes should be 16000 megabytes or 16000000 kilobytes, right?

```
LSB_JOB_REPORT_MAIL=N bsub -oo lsf.out -eo lsf.err -R "rusage[mem=16000000KB/job]" sleep 20
```

```
Job <274348028>, User <afoltzer>, Project <default>, Status <RUN>, Queue <stand
                     ard>, Job Priority <50>, Command <sleep 20>, Share group c
                     harged </afoltzer>, Esub <hpcf>
Thu Nov 13 20:34:14: Submitted from host <noderome104>, CWD <$HOME>, Output Fil
                     e (overwrite) <lsf.out>, Error File (overwrite) <lsf.err>,
                      Requested Resources <rusage[mem=15625.00/job]>;
Thu Nov 13 20:34:20: Started 1 Task(s) on Host(s) <noderome209>, Allocated 1 Sl
                     ot(s) on Host(s) <noderome209>, Execution Home </home/afol
                     tzer>, Execution CWD </home/afoltzer>;
Thu Nov 13 20:34:28: Resource usage collected.
                     MEM: 7 Mbytes;  SWAP: 0 Mbytes;  NTHREAD: 4
                     PGID: 3985845;  PIDs: 3985845 3985875 3985877

 MEMLIMIT SWAPLIMIT
   15.2 G    15.2 G

 MEMORY USAGE:
 MAX MEM: 8 Mbytes;  AVG MEM: 4 Mbytes; MEM Efficiency: 0.05%

 CPU USAGE:
 CPU PEAK: 0.00 ;  CPU PEAK DURATION: 0 second(s)
 CPU AVERAGE EFFICIENCY: 0.00% ;  CPU PEAK EFFICIENCY: 0.00%

 SCHEDULING PARAMETERS:
           r15s   r1m  r15m   ut      pg    io   ls    it    tmp    swp    mem
 loadSched   -    3.5    -     -       -     -    -     -  1000M     -      -
 loadStop    -     -     -     -       -     -    -     -     -      -      -

          waitpct     cWs   MBWps rgs01_load
 loadSched   10.0    40.0 20000.0      130.0
 loadStop      -       -       -          -

 RESOURCE REQUIREMENT DETAILS:
 Combined: select[(rhel8) && (type == any)] order[r15s:pg] rusage[mem=15625.00/
                     job] affinity[core(1)*1]
 Effective: select[((rhel8) && (type == any))] order[r15s:pg] rusage[mem=15625.
                     00/job] affinity[core(1)*1]
```

Note in particular the mem and swap limit of `15.2 G` and the fact that the rusage string now reads
`mem=15625.00/job` (the default unit on _our_ cluster but not all LSF clusters is `MB`). So, whoops!
I misremembered: sometimes it looks like you've been given less than what you asked for, but that's
because it's reporting in base 2!

Now let's try 16777216 kibibytes, which is 16384 mebibytes or 16 gibibytes.

```
LSB_JOB_REPORT_MAIL=N bsub -oo lsf.out -eo lsf.err -R "rusage[mem=16777216KB/job]" sleep 20
```

There are 16777216 kibibytes in 16384 mebibytes or 16 gibibytes.

```
Job <274346639>, User <afoltzer>, Project <default>, Status <RUN>, Queue <stand
                     ard>, Job Priority <50>, Command <sleep 20>, Share group c
                     harged </afoltzer>, Esub <hpcf>
Thu Nov 13 20:30:10: Submitted from host <noderome104>, CWD <$HOME>, Output Fil
                     e (overwrite) <lsf.out>, Error File (overwrite) <lsf.err>,
                      Requested Resources <rusage[mem=16384.00/job]>;
Thu Nov 13 20:30:27: Started 1 Task(s) on Host(s) <noderome140>, Allocated 1 Sl
                     ot(s) on Host(s) <noderome140>;

 MEMLIMIT SWAPLIMIT
     16 G      16 G

 MEMORY USAGE:
 MEM Efficiency: 0.00%

 CPU USAGE:
 CPU PEAK: 0.00 ;  CPU PEAK DURATION: 0 second(s)
 CPU AVERAGE EFFICIENCY: 0.00% ;  CPU PEAK EFFICIENCY: 0.00%

 SCHEDULING PARAMETERS:
           r15s   r1m  r15m   ut      pg    io   ls    it    tmp    swp    mem
 loadSched   -    3.5    -     -       -     -    -     -  1000M     -      -
 loadStop    -     -     -     -       -     -    -     -     -      -      -

          waitpct     cWs   MBWps rgs01_load
 loadSched   10.0    40.0 20000.0      130.0
 loadStop      -       -       -          -

 RESOURCE REQUIREMENT DETAILS:
 Combined: select[(rhel8) && (type == any)] order[r15s:pg] rusage[mem=16384.00/
                     job] affinity[core(1)*1]
 Effective: select[((rhel8) && (type == any))] order[r15s:pg] rusage[mem=16384.
                     00/job] affinity[core(1)*1]
```

16 gibibytes is 16384 mebibytes, so if we give a base 2 memory constraint in rusage we do in fact
get a base 2 amount of memory reported by the system. Even though all the docs use the base 10 unit
names and suffixes.
@adthrasher
adthrasher self-requested a review December 3, 2025 20:15
@adthrasher

Copy link
Copy Markdown
Owner

Oddly, this does not appear to be the case when using GB as the unit.

$ bsub -M "16GB" sleep 20
Job <276267301> is submitted to queue <compbio>.
$ bjobs -l 276267301

Job <276267301>, User <athrashe>, Project <CompBio>, Status <PEND>, Queue <comp
                     bio>, Command <sleep 20>, Esub <hpcf>
Thu Dec  4 08:06:26: Submitted from host <splprhpc11>, CWD <$HOME>, Requested R
                     esources <rusage[mem=16384]>;
 PENDING REASONS:
 New job is waiting for scheduling;


 MEMLIMIT SWAPLIMIT
     16 G      16 G

 SCHEDULING PARAMETERS:
           r15s   r1m  r15m   ut      pg    io   ls    it    tmp    swp    mem
 loadSched   -     -     -     -       -     -    -     -     -      -      -
 loadStop    -     -     -     -       -     -    -     -     -      -      -

              cWs   totWs    FOps    FCps rgs01_load
 loadSched   50.0   400.0  2000.0  2000.0      140.0
 loadStop      -       -       -       -          -

 RESOURCE REQUIREMENT DETAILS:
 Combined: -
 Effective: -

@adthrasher

Copy link
Copy Markdown
Owner

Actually, I wonder if there is an issue in /ifs001/lsf/lsf_prod/10.1/linux3.10-glibc2.17-x86_64/etc/hpcf_esub_funs.py. That script edits the submission request before passing it to LSF. I haven't looked through it in detail, but that's where the default memory specification comes from:

622     ### possibly print information to user
623     if( not user_requested_mem and not app_requested_mem ):
624         stderr.write("HPCF: WARNING! No Memory was requested!\n")
625         stderr.write("      A default memory request of %.2f GB has been placed for this job\n"%( hpcf_mem_req/1000.0 ))
626         stderr.write("      The job will be killed if   %.2f GB of memory is used\n"%( hpcf_mem_lim/1000.0 ))

It does some additional checking and rewriting of the LSF parameters before passing the job to the scheduler.

@a-frantz

a-frantz commented Dec 4, 2025

Copy link
Copy Markdown

linking this PR for some extra weridness - stjudecloud/workflows#279

That discrepancy however does not fully explain the scale of this change ( a +6 GB became +12) and I don't have an explanation for that. miniwdl suceeded with +6 and sprocket failed for some smaller bumps before succeeding at +12. I'm ok chocking that up to cluster weirdness instead of doing a full investigation 🤷‍♀️

I saw differences between miniwdl and sprocket behavior regarding LSF memory that is not explained by base10->base2 conversion.

I will point out that I've noticed that our LSF cluster is bad at spotting short spikes in memory usage. My understanding is that a running job can spike over it's "hard limit" without being noticed by LSF if the spike is short enough in duration. That may be at play here?

@acfoltzer

Copy link
Copy Markdown
Author

Huh, I do agree that the behavior for GB (and GB only?) appears to be consistent with that unit being treated as base-2, while others are base-10.

I also looked through the hpcf_esub_funs.py script, and it contains treatments of memory limits using both bases, but I don't think that would impact user-specified memory requests unless the user-specified requests exceeded cluster-wide or application-specific limits in such a way that the max() calls would give precedence to some of the HPCF-defined values. I also didn't see any special treatment of the GB suffix, so that remains puzzling.

The apparent difference with GB gives me less confidence that the change I'm proposing here is correct across different LSF implementations. Is there another cluster we might be able to check some of these sample jobs on to compare behavior?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants