Update omrsysinfo_cgroup_subsystem_iterator functions for cgroup v2 #6494

EricYangIBM · 2022-04-28T15:01:41Z

Update cgroup iterator functions to retrieve metrics from cgroup v2
files.

Issue: #1281
Signed-off-by: Eric Yang eric.yang@ibm.com

EricYangIBM · 2022-04-28T15:03:13Z

@babsingh iterator for v2 memory subsystem is done and working. Can you take an early look?

babsingh · 2022-04-28T15:16:55Z

Can you take an early look?

yes, I will do one by the end of day.

babsingh

first review pass.

port/unix/omrsysinfo.c

babsingh

Minor formatting suggestions. Pinged @0xdaryl on Slack for #6465 amd #6479 to be merged so that this PR can be rebased. More feedback on the functional behaviour to follow.

port/unix/omrsysinfo.c

babsingh

For verification, can you provide the Cgroup Information Section from a javacore for cgroup v1 and v2?

Examples in https://blog.openj9.org/2019/04/22/cgroup-metrics-now-available-in-javacore/.

port/unix/omrsysinfo.c

EricYangIBM · 2022-05-11T20:10:00Z

Cgroup v1

Host

2CICGRPINFO    subsystem : cpu
2CICGRPINFO    cgroup name : /user.slice
3CICGRPINFO        CPU Period : 100000 microseconds
3CICGRPINFO        CPU Quota : Not Set
3CICGRPINFO        CPU Shares : 1024
3CICGRPINFO        Period intervals elapsed count : 0
3CICGRPINFO        Throttled count : 0
3CICGRPINFO        Total throttle time : 0 nanoseconds
2CICGRPINFO    subsystem : memory
2CICGRPINFO    cgroup name : /user.slice/user-0.slice/session-1.scope
3CICGRPINFO        Memory Limit : Not Set
3CICGRPINFO        Memory + Swap Limit : Not Set
3CICGRPINFO        Memory Usage : 1309958144 bytes
3CICGRPINFO        Memory + Swap Usage : 1309958144 bytes
3CICGRPINFO        Memory Max Usage : 1309958144 bytes
3CICGRPINFO        Memory + Swap Max Usage : 1309958144 bytes
3CICGRPINFO        Memory limit exceeded count : 0
3CICGRPINFO        Memory + Swap limit exceeded count : 0
3CICGRPINFO        OOM Killer Disabled : 0
3CICGRPINFO        Under OOM : 0
2CICGRPINFO    subsystem : cpuset
2CICGRPINFO    cgroup name : /
3CICGRPINFO        CPU exclusive : 1
3CICGRPINFO        Mem exclusive : 1
3CICGRPINFO        CPUs : 0-7
3CICGRPINFO        Mems : 0

Container no limits

2CICGRPINFO    subsystem : cpu
2CICGRPINFO    cgroup name : /
3CICGRPINFO        CPU Period : 100000 microseconds
3CICGRPINFO        CPU Quota : Not Set
3CICGRPINFO        CPU Shares : 1024
3CICGRPINFO        Period intervals elapsed count : 0
3CICGRPINFO        Throttled count : 0
3CICGRPINFO        Total throttle time : 0 nanoseconds
2CICGRPINFO    subsystem : memory
2CICGRPINFO    cgroup name : /
3CICGRPINFO        Memory Limit : Not Set
3CICGRPINFO        Memory + Swap Limit : Not Set
3CICGRPINFO        Memory Usage : 35131392 bytes
3CICGRPINFO        Memory + Swap Usage : 35131392 bytes
3CICGRPINFO        Memory Max Usage : 35135488 bytes
3CICGRPINFO        Memory + Swap Max Usage : 35135488 bytes
3CICGRPINFO        Memory limit exceeded count : 0
3CICGRPINFO        Memory + Swap limit exceeded count : 0
3CICGRPINFO        OOM Killer Disabled : 0
3CICGRPINFO        Under OOM : 0
2CICGRPINFO    subsystem : cpuset
2CICGRPINFO    cgroup name : /
3CICGRPINFO        CPU exclusive : 0
3CICGRPINFO        Mem exclusive : 0
3CICGRPINFO        CPUs : 0-7
3CICGRPINFO        Mems : 0

Container with limits docker run -v /root/docker_hostdir:/root/hostdir -it -m 4GB --cpu-period=1000 --cpu-quota=10000 --cpuset-cpus=1,2,4-5 --rm openj9

2CICGRPINFO    subsystem : cpu
2CICGRPINFO    cgroup name : /
3CICGRPINFO        CPU Period : 1000 microseconds
3CICGRPINFO        CPU Quota : 10000 microseconds
3CICGRPINFO        CPU Shares : 1024
3CICGRPINFO        Period intervals elapsed count : 92
3CICGRPINFO        Throttled count : 0
3CICGRPINFO        Total throttle time : 0 nanoseconds
2CICGRPINFO    subsystem : memory
2CICGRPINFO    cgroup name : /
3CICGRPINFO        Memory Limit : 4294967296 bytes
3CICGRPINFO        Memory + Swap Limit : 8589934592 bytes
3CICGRPINFO        Memory Usage : 24539136 bytes
3CICGRPINFO        Memory + Swap Usage : 24539136 bytes
3CICGRPINFO        Memory Max Usage : 24543232 bytes
3CICGRPINFO        Memory + Swap Max Usage : 24543232 bytes
3CICGRPINFO        Memory limit exceeded count : 0
3CICGRPINFO        Memory + Swap limit exceeded count : 0
3CICGRPINFO        OOM Killer Disabled : 0
3CICGRPINFO        Under OOM : 0
2CICGRPINFO    subsystem : cpuset
2CICGRPINFO    cgroup name : /
3CICGRPINFO        CPU exclusive : 0
3CICGRPINFO        Mem exclusive : 0
3CICGRPINFO        CPUs : 1-2,4-5
3CICGRPINFO        Mems : 0

Cgroup v2

Host

2CICGRPINFO    subsystem : memory
2CICGRPINFO    cgroup name : /user.slice/user-0.slice/session-876.scope
3CICGRPINFO        Memory Limit : Not Set
3CICGRPINFO        Swap Limit : Not Set
3CICGRPINFO        Memory Usage : 1619886080 bytes
3CICGRPINFO        Swap Usage : 64802816 bytes
3CICGRPINFO        Approached memory limit count : 1246
3CICGRPINFO        Reached memory limit count : 0
3CICGRPINFO        Approached swap limit count : 0
3CICGRPINFO        Swap alloc failed count : 0

Container no limits

2CICGRPINFO    subsystem : memory
2CICGRPINFO    cgroup name : /
3CICGRPINFO        Memory Limit : Not Set
3CICGRPINFO        Swap Limit : Not Set
3CICGRPINFO        Memory Usage : 2601267200 bytes
3CICGRPINFO        Swap Usage : 0 bytes
3CICGRPINFO        Approached memory limit count : 0
3CICGRPINFO        Reached memory limit count : 0
3CICGRPINFO        Approached swap limit count : 0
3CICGRPINFO        Swap alloc failed count : 0
2CICGRPINFO    subsystem : cpuset
2CICGRPINFO    cgroup name : /
3CICGRPINFO        CPUs : Not Set
3CICGRPINFO        Mems : Not Set
3CICGRPINFO        Effective CPUs : 0-7
3CICGRPINFO        Effective Mems : 0
2CICGRPINFO    subsystem : cpu
2CICGRPINFO    cgroup name : /
3CICGRPINFO        CPU Quota : Not Set
3CICGRPINFO        CPU Period : 100000 microseconds
3CICGRPINFO        CPU Weight relative to procs in same cgroup : 100
3CICGRPINFO        Period intervals elapsed count : 0
3CICGRPINFO        Throttled count : 0
3CICGRPINFO        Total throttle time : 0 microseconds

Container with limits docker run -v /root/docker_hostdir:/root/hostdir -it -m 4GB --cpu-period=1000 --cpu-quota=10000 --cpuset-cpus=1,2,4-5 --rm openj9

2CICGRPINFO    subsystem : memory
2CICGRPINFO    cgroup name : /
3CICGRPINFO        Memory Limit : 4294967296 bytes
3CICGRPINFO        Swap Limit : 4294967296 bytes
3CICGRPINFO        Memory Usage : 22908928 bytes
3CICGRPINFO        Swap Usage : 0 bytes
3CICGRPINFO        Approached memory limit count : 0
3CICGRPINFO        Reached memory limit count : 0
3CICGRPINFO        Approached swap limit count : 0
3CICGRPINFO        Swap alloc failed count : 0
2CICGRPINFO    subsystem : cpuset
2CICGRPINFO    cgroup name : /
3CICGRPINFO        CPUs : 1-2,4-5
3CICGRPINFO        Mems : Not Set
3CICGRPINFO        Effective CPUs : 1-2,4-5
3CICGRPINFO        Effective Mems : 0
2CICGRPINFO    subsystem : cpu
2CICGRPINFO    cgroup name : /
3CICGRPINFO        CPU Quota : 10000 microseconds
3CICGRPINFO        CPU Period : 1000 microseconds
3CICGRPINFO        CPU Weight relative to procs in same cgroup : 100
3CICGRPINFO        Period intervals elapsed count : 103
3CICGRPINFO        Throttled count : 1
3CICGRPINFO        Total throttle time : 224 microseconds

port/unix/omrsysinfo.c

babsingh · 2022-05-11T20:31:20Z

re #6494 (comment):

For Cgroup v2 Host output, why cpu and cpuset subsystems are not shown?

EricYangIBM · 2022-05-11T20:34:25Z

Differences in v2 hierarchy. In v2 if a subsystem is not in cgroup.controllers it means the subsystem is not active / limits are not set for that cgroup

babsingh · 2022-05-11T20:35:49Z

OSX failures: https://github.com/eclipse/omr/pull/6494/checks?check_run_id=6395271683

These failures are related to the socket library, and probably caused due to network issues.

31: [  FAILED  ] PortSockTest.poll_functionality_basic (1001 ms)
31: /Users/runner/work/1/s/fvtest/porttest/omrsockTest.cpp:51: Failure
31:       Expected: privateOmrPortLibrary->sock_bind(privateOmrPortLibrary, *serverSocket, serverSockAddr)
31:       Which is: -506
31: To be equal to: 0
31: /Users/runner/work/1/s/fvtest/porttest/omrsockTest.cpp:1259: Failure
31:       Expected: privateOmrPortLibrary->sock_accept(privateOmrPortLibrary, serverSocket, &connectedServerSockAddr, &connectedServerSocket)
31:       Which is: -20
31: To be equal to: 0
31: [  FAILED  ] PortSockTest.poll_functionality_many_sockets (1 ms)
31: [----------] 19 tests from PortSockTest (1006 ms total)
31: 
31: [==========] 236 tests from 20 test cases ran. (91357 ms total)
31: [  PASSED  ] 234 tests.
31: [  FAILED  ] 2 tests, listed below:
31: [  FAILED  ] PortSockTest.poll_functionality_basic
31: [  FAILED  ] PortSockTest.poll_functionality_many_sockets

babsingh · 2022-05-12T12:41:36Z

jenkins build all

babsingh · 2022-05-12T13:36:44Z

Compilation error:

08:42:34  /home/jenkins/workspace/Build/port/unix/omrsysinfo.c:6835:4: error: ‘for’ loop initial declarations are only allowed in C99 mode
08:42:34      for (int32_t i = 0; i < state->fileMetricCounter; i++) {
08:42:34      ^

babsingh · 2022-05-12T14:51:37Z

jenkins build all

babsingh · 2022-05-12T15:03:02Z

jenkins build all

babsingh · 2022-05-12T15:05:04Z

@EricYangIBM OSX failures consistently happen. Can you confirm that the OSX failures reported in #6494 (comment) are not occurring due to this PR?

EricYangIBM · 2022-05-12T17:03:30Z

Rebasing didn't change anything but reverting the commit stopped the failures. All of the commit's changes are ifdefd for linux only so I don't know how this PR could cause mac failures

EricYangIBM · 2022-05-12T17:21:11Z

Seems like the most recent builds passed

babsingh · 2022-05-12T17:34:43Z

jenkins build all

babsingh · 2022-05-13T15:08:14Z

Restarting the zOS job since it timed out in finding a node.

babsingh · 2022-05-13T15:08:20Z

jenkins build zos

EricYangIBM · 2022-05-13T17:00:45Z

All zos nodes are offline

babsingh · 2022-05-13T18:19:32Z

All zos nodes are offline

@jdekonin @AdamBrousseau Are the zOS machines strategically turned off? or are we experiencing an outage? Is there a timeline when they will be online?

jdekonin · 2022-05-16T13:48:48Z

If there was an outage I wasn't advised or maybe I missed the email?? It would appear that both are online now though.

babsingh · 2022-05-16T15:51:34Z

jenkins build zos

dsouzai

Overall looks fine; just had a few questions.

port/unix/omrsysinfo.c

Update cgroup iterator functions to retrieve metrics from cgroup v2 files. Issue: eclipse-omr#1281 Signed-off-by: Eric Yang <eric.yang@ibm.com>

dsouzai · 2022-05-16T21:25:02Z

jenkins build all

mpirvu · 2022-05-17T14:06:44Z

Tests have passed.
@EricYangIBM is this the last PR that is needed to have cgroup v2 support in OpenJ9? Thanks

EricYangIBM · 2022-05-17T14:11:17Z

Yes, all that remains after this is related to testing (both v1 and v2)

mpirvu · 2022-05-17T14:13:31Z

Great work! Thanks!

github-actions bot added the comp:port label Apr 28, 2022

EricYangIBM force-pushed the iteratorCgroup branch from e02dc67 to 99958d9 Compare April 28, 2022 17:14

babsingh requested changes Apr 28, 2022

View reviewed changes

This was referenced Apr 28, 2022

Remove the --gtest_filter=${porttest_filter} option for porttest #6487

Merged

Expose limits imposed by cgroups via port library APIs #1281

Open

EricYangIBM force-pushed the iteratorCgroup branch 3 times, most recently from fada4a0 to 85e22f9 Compare May 4, 2022 17:46

EricYangIBM marked this pull request as ready for review May 4, 2022 17:48

EricYangIBM requested review from charliegracie, mstoodle, rwy7 and youngar as code owners May 4, 2022 17:48

EricYangIBM requested a review from babsingh May 5, 2022 14:13

babsingh self-assigned this May 5, 2022

EricYangIBM force-pushed the iteratorCgroup branch from 85e22f9 to 7b50477 Compare May 6, 2022 14:42

babsingh requested changes May 6, 2022

View reviewed changes

EricYangIBM force-pushed the iteratorCgroup branch from e904299 to 3bb6769 Compare May 9, 2022 15:12

babsingh requested changes May 11, 2022

View reviewed changes

port/unix/omrsysinfo.c Outdated Show resolved Hide resolved

port/unix/omrsysinfo.c Outdated Show resolved Hide resolved

port/unix/omrsysinfo.c Show resolved Hide resolved

EricYangIBM force-pushed the iteratorCgroup branch from d994db6 to cb26705 Compare May 11, 2022 19:56

babsingh reviewed May 11, 2022

View reviewed changes

port/unix/omrsysinfo.c Show resolved Hide resolved

EricYangIBM force-pushed the iteratorCgroup branch 2 times, most recently from fd0f8c5 to 4e84385 Compare May 12, 2022 17:00

babsingh approved these changes May 13, 2022

View reviewed changes

This was referenced May 13, 2022

OSX PR builds: OMR Socket Test Failures #6516

Open

Fix PortSysinfoTest failure when running as root #6513

Merged

dsouzai reviewed May 16, 2022

View reviewed changes

dsouzai self-assigned this May 16, 2022

EricYangIBM force-pushed the iteratorCgroup branch from 4e84385 to 110b845 Compare May 16, 2022 20:43

Update omrsysinfo_cgroup_subsystem_iterator functions for cgroup v2

2718485

Update cgroup iterator functions to retrieve metrics from cgroup v2 files. Issue: eclipse-omr#1281 Signed-off-by: Eric Yang <eric.yang@ibm.com>

EricYangIBM force-pushed the iteratorCgroup branch from 110b845 to 2718485 Compare May 16, 2022 20:44

dsouzai merged commit 5e51a12 into eclipse-omr:master May 17, 2022

EricYangIBM deleted the iteratorCgroup branch May 17, 2022 14:20

Update omrsysinfo_cgroup_subsystem_iterator functions for cgroup v2 #6494

Update omrsysinfo_cgroup_subsystem_iterator functions for cgroup v2 #6494

Uh oh!

Conversation

EricYangIBM commented Apr 28, 2022

Uh oh!

EricYangIBM commented Apr 28, 2022

Uh oh!

babsingh commented Apr 28, 2022

Uh oh!

babsingh left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

babsingh left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

babsingh left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

EricYangIBM commented May 11, 2022

Cgroup v1

Cgroup v2

Uh oh!

Uh oh!

babsingh commented May 11, 2022

Uh oh!

EricYangIBM commented May 11, 2022

Uh oh!

babsingh commented May 11, 2022

Uh oh!

babsingh commented May 12, 2022

Uh oh!

babsingh commented May 12, 2022

Uh oh!

babsingh commented May 12, 2022

Uh oh!

babsingh commented May 12, 2022

Uh oh!

babsingh commented May 12, 2022

Uh oh!

EricYangIBM commented May 12, 2022

Uh oh!

EricYangIBM commented May 12, 2022

Uh oh!

babsingh commented May 12, 2022

Uh oh!

babsingh commented May 13, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

babsingh commented May 13, 2022

Uh oh!

EricYangIBM commented May 13, 2022

Uh oh!

babsingh commented May 13, 2022

Uh oh!

jdekonin commented May 16, 2022

Uh oh!

babsingh commented May 16, 2022

Uh oh!

dsouzai left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

babsingh left a comment •

edited

Loading

babsingh commented May 13, 2022 •

edited

Loading