Repository navigation
Conversation
The build of the suite depended on `TARGET unit-test-libsinsp`. CMake evaluates that condition before it defines the target, so it never built the scap-file tests. Signed-off-by: Grzegorz Nosek <grzegorz.nosek@sysdig.com>
A later commit computes the parent instead of storing it on every thread. Code outside the thread must not touch the field, so the field becomes private. The behavior does not change. Signed-off-by: Grzegorz Nosek <grzegorz.nosek@sysdig.com>
The benchmarks measure these operations: - A walk up the ancestors (proc.aname, proc.aname[n]), also through dead ancestors. - The removal of a process that has children. - A clone or execve by a process whose ancestors died, and then a walk from that process. Each operation has its own instruction counter, so the counts do not include setup. Signed-off-by: Grzegorz Nosek <grzegorz.nosek@sysdig.com>
…he tree These tests record the current behavior. If the procexit event gives no reaper, the children of a dead process go to the nearest subreaper above it. If there is no subreaper, they go to init. The search does not go past the pid namespace of the process. An ancestor walk stops on a loop. Chains 300 deep walk to init. When their subreapers die from the bottom up, the chains reparent to init. This is true with or without the reapers from the kernel. A ppid loop of 300 from /proc also stops the walk. Signed-off-by: Grzegorz Nosek <grzegorz.nosek@sysdig.com>
The ancestor walks, the reaper search, and the parent filterchecks now find the parent through get_parent_thread(). A later change to how sinsp finds a parent then touches only this one function. The behavior does not change. Signed-off-by: Grzegorz Nosek <grzegorz.nosek@sysdig.com>
ASSERT_THREAD_CHILDREN now scans the thread table for the threads whose parent is the given thread. It no longer reads the child list of the parent, and it no longer tests the size of that list (`children_num`). For a thread that is not in the table, the test looks at the parent link of the thread instead. Signed-off-by: Grzegorz Nosek <grzegorz.nosek@sysdig.com>
All threads of a process have the same parent, so the thread group now stores the parent tid. The thread that creates the group sets it. A thread without a group uses its own value. Signed-off-by: Grzegorz Nosek <grzegorz.nosek@sysdig.com>
… when it is written Before, the parent of a process was a tid, and an ancestor walk looked it up on every hop. Now the group holds a pointer to the group of the parent, with the tid of the thread that forked. The writer resolves the link when it writes it. A reader takes that thread from the thread list of the group. If the thread is gone, the reader takes the first live thread of the group. Every link goes down in rank, so the links never form a cycle. The rank of a group is set before the group is published, and it never changes. The rank is the rank of the parent plus one, 0 without a parent, and a high value when the parent is unknown. A link to a group with the same or a higher rank is refused. Only bad data asks for such a link, for example a loop in the /proc ppids. The test needs only the two groups that the link joins. A refused link keeps the tid without a pointer, as an unknown parent does, and walks stop there. A debug log records each refused link. The /proc scan can reach a child before its parent. In that case, the scan creates the group of the parent first. A ppid loop can lead back to the child, so the scan first publishes the group of the child, with a rank that nothing can link to. The ptid of a thread is writable only until the thread joins a group. After that, the `ptid` field is read-only. A thread without a group has no parent to follow. Thus, the e2e walk test gives its threads a pid, and sinsp now refuses the loops of that test. Signed-off-by: Grzegorz Nosek <grzegorz.nosek@sysdig.com>
The links cannot form a loop, so the walk needs no second cursor. Debug builds assert that the chain ends. parent_loop_detected() can only return false now, so it is removed. Signed-off-by: Grzegorz Nosek <grzegorz.nosek@sysdig.com>
…ist of children Before, every thread kept a list of its children. When a thread died, sinsp rewrote its children one by one to the reaper. It also updated the list whenever a child went away. The lists are gone. Now a link, an exit, or the removal of a process each writes only that process. When the last thread of a group goes, the group leaves the group table and becomes a tombstone. A tombstone stays alive while a child links to it. Its reaper replaces its parent link, unless that link goes up in rank (for example, a tid reused after a lost exit). The reaper is the one from the kernel, or the one that the search finds. While the group lives, the children of a removed thread go to the first live thread of the group. A main thread stays in the table while its group lives, so sinsp marks the removed threads. The procexit parser keeps the reaper that the kernel sends. The kernel sends one only for a thread with children. The tests of assign_children_to_reaper are removed. A new test makes sure that a tombstone lives exactly as long as its children. Signed-off-by: Grzegorz Nosek <grzegorz.nosek@sysdig.com>
If the procexit event names no reaper, the tombstone keeps its parent link. A reader that reaches the tombstone searches above it for the nearest live subreaper or init, in its pid namespace. An exit no longer walks the tree, and the search writes nothing. A subreaper that is set after the death now counts. If sinsp refused the reaper from the kernel for a tombstone, the children of the tombstone get the reaper that the search finds. Signed-off-by: Grzegorz Nosek <grzegorz.nosek@sysdig.com>
…ng a reference Before, a read of a thread from the list locked a weak_ptr and released it at once. That cost two atomic read-modify-writes, which was most of the cost of resolving a parent. The thread table owns the threads, so an entry now keeps the raw pointer. The entry returns the pointer after it calls expired(). Signed-off-by: Grzegorz Nosek <grzegorz.nosek@sysdig.com>
Signed-off-by: Grzegorz Nosek <grzegorz.nosek@sysdig.com>
…execs A reader follows every tombstone above a process, and a tombstone lives while a child links to it. Clone and execve write the calling process anyway. Now they also link it straight to where its dead parents lead, and this frees the tombstones in between. Other events cost nothing. The search for an unknown reaper runs once, here, and sinsp links the result. The store goes down in rank. If the link of the process itself changed in the meantime, sinsp drops the store. Two tests use a debug-only hook to drive a relink into that window. Signed-off-by: Grzegorz Nosek <grzegorz.nosek@sysdig.com>
…d parents when it dies A tombstone with an unknown reaper keeps its parent link for the search. If that parent is dead too, the dying process links past it, as clone and execve do. This frees the tombstone of the parent. The change covers parents that die first. If a child dies first, its own next clone or execve links it past them. Signed-off-by: Grzegorz Nosek <grzegorz.nosek@sysdig.com>
Before, a process from /proc whose parent was not in the table got ptid 0. Now it keeps the tid of the parent. If the table has that parent by the next clone or execve of the process, that event links the process to it. The process ranks above any known ancestry, so the link goes down in rank. The exception is a parent whose ancestry is also unknown. sinsp refuses that link and does not try it again. A reused tid is linked as if it were the parent. A debug log records each late link. Signed-off-by: Grzegorz Nosek <grzegorz.nosek@sysdig.com>
At a procexit, the tool printed the children that it said sinsp moved to a new reaper. sinsp no longer moves children. Their links stay, and the dead group becomes a tombstone. The tool now shows what sinsp stores and does: - After a clone, an execve, or a procexit, the chain of parent links, one group per hop. Each hop shows the rank, the tid that the link names, and if the group is a tombstone. The thread that a reader gets comes last. - At a procexit, the reaper that the kernel sent, and the processes that link to the group of the thread. - After sinsp removes the thread on the next event, if the group was freed or stays as a tombstone, and the link of the tombstone. - The debug log lines about refused and late parent links, as they happen. Signed-off-by: Grzegorz Nosek <grzegorz.nosek@sysdig.com>
Contributor
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: gnosek The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #3133 +/- ##
==========================================
+ Coverage 77.46% 79.61% +2.14%
==========================================
Files 304 309 +5
Lines 34989 36273 +1284
Branches 5194 5212 +18
==========================================
+ Hits 27105 28878 +1773
+ Misses 7884 7395 -489
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Perf diff from master - unit testsHeap diff from master - unit testsHeap diff from master - scap fileBenchmarks diff from master |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
/kind feature
/kind cleanup
Any specific area of the project related to this PR?
/area libsinsp
/area tests
Does this PR require a change in the driver versions?
No.
What this PR does / why we need it:
TL;DR:
Rework how parent-child relations are stored in the thread table. Make the common case faster, some edge cases slower, but most importantly, enable the thread table to be used from multiple threads without cross-object atomic transactions (sequential individually atomic updates to single threadinfos are now enough). This does not make the thread table thread safe (far from it), but it makes the process significantly simpler (to implement, and more importantly, to understand).
The whole story from Claude:
This PR changes how libsinsp stores the process tree. Before, each thread kept a list of its
children, and each child kept the tid of its parent. When a process died, sinsp moved every
child to the reaper, one by one, and it changed both ends of each parent-child edge. This cost
O(N) for a process with N children. A write that must change two objects at the same time also
blocks a multithreaded thread table.
Now only the child stores the link, and nothing keeps a list of children:
thread_group_info). It is a pointerto the group of the parent, plus the tid of the thread that forked. sinsp resolves the link
when it writes it, so an ancestor walk does not look up a tid on every hop.
the rank of the parent plus one. A link can only go to a group with a lower rank, so the links
can never form a loop. The test needs only the two groups that the link joins, and it needs
no lock. A link that does not go down in rank is refused, and sinsp logs it at debug level.
Only bad or missing data asks for such a link, for example a loop in the ppids from
/proc,a tid reused after a lost exit, or a parent whose own ancestry is unknown.
while a child links to it). The reaper replaces the parent link of the tombstone. Sometimes
the procexit event names no reaper. Then a reader that reaches the tombstone searches for the
nearest live subreaper or init. The search stays in the same pid namespace.
process past its dead parents, which frees the tombstones in between.
/procwhose parent is not in the table keeps the tid of its parent insteadof 0. If the table has the parent by its next clone or execve, that event links it to the
parent.
The ancestor walk now follows the links with one cursor. Because the links cannot form a loop,
the walk needs no loop detection, and debug builds assert that the chain ends.
The first commit also fixes the CMake condition that kept the scap-file test suite from being
built, so these tests run again.
Benchmarks:
The base is commit 3, which adds the benchmark: master, the scap-suite fix, and the ptid
accessors. Both arms are Release builds with the same compiler flags. The
process_treebenchmark counts user-space instructions around each operation only. The table gives the median
of 3 interleaved rounds on one pinned core.
proc.aname[n], n = 1 to 8The walk through 15 dead ancestors is slower because master moved the children to the reaper at
each exit, which made the chain shorter. This PR keeps the tombstones until the next clone or
execve of the child. If the reaper is unknown, each read also searches for it again. Only a
process that runs no clone or execve after its ancestors die pays this cost, for example a
long-running daemon.
A clone or an execve costs 300 to 2000 more instructions. When ancestors are dead, part of that
is the reaper search and the release of the tombstones, which master paid at the exit. A cost of
300 to 550 instructions also occurs with no dead ancestors, on the first event of a new
inspector. It first appears in commit 8. A profile spreads it over unrelated functions, so it
looks like a change in the state of the allocator or the code layout, not a cost per event. A
repeated execve on the same tree differs by 7 instructions per event.
Replay of real captures through sinsp (a harness that is not part of this PR), in instructions
per event from a two-point slope. Each value reproduces to 0.1 instructions per event across
3 rounds. Both arms pass the same number of events through each filter.
The benchmarks outside the process tree (
plugin_field,utils) measure CPU time and change byless than the noise of the machine.
Which issue(s) this PR fixes:
Fixes #
Special notes for your reviewer:
The order of the commits follows the design: each commit builds and passes the unit tests. Commits
8 to 16 change one property at a time, so you can review them one by one.
The
ptidfield of the thread table is now read-only for a thread in a thread group. The reasonis that the parent link belongs to the group. A plugin that writes
ptidon such a thread getsSS_PLUGIN_FAILURE. A plugin can still writeptidon a new entry before it adds the entry tothe table.
sinsp logs each refused link and each late link at debug level, with the pid, the tid of the
parent, and for a refused link the two ranks. These log lines replace the removed warning for a
loop in the tree. They are for diagnosis, for example when a process has a short ancestry. They
are not metrics, because an operator cannot act on a count of them. Their causes are dropped
events, races in the
/procscan, and missing ancestry, and the drop counters already reportthe first one.
Behavioral differences
None of this should ever surface during a normal run, but the behavior in some edge cases (like missing the parent in the thread table) might be different.
Parent missing from the table. A process from /proc whose parent is not in the table got ptid 0 on master, and its ancestry stayed empty for good. Now it keeps the real ppid, so proc.ppid shows that value. A later clone or execve of the process links it to the parent if the table has the parent by then, and from then on the ancestry is complete. The same holds when a clone event names a parent that sinsp does not know. One risk is new: if the parent's pid is reused before the link happens, the process links to the unrelated new process with that pid, and its ancestry is wrong. On master it would have had none.
No reaper in the procexit event. On master, sinsp searched for the nearest subreaper or init when the process exited, and wrote the result into each child. Now the dead parent stays as a tombstone, and each read searches above it. The search stops at the same places as master: the nearest live subreaper, or init, within the same pid namespace. Two things differ. A subreaper that is set after the death now counts. And until the child's next clone or execve, the answer can change if subreapers above it appear or exit, because nothing stores it.
Reaper sent by the kernel. sinsp still uses it, unless the kernel's reaper ranks above the dying process. That can happen when a tid was reused after an exit that sinsp never saw. Master accepted such a reaper even when it created a loop in the tree. Now the link is refused, and readers search for a reaper instead.
Loops in the tree. On master, a loop could form, for example from a ppid loop in /proc or a bad reaper. The ancestor walk then detected it, stopped, set parent_loop_detected(), and logged the warning "Loop in parent thread state detected" once per process. Now a loop cannot form, because sinsp refuses the link that would close it. The walk stops at the process whose link was refused, which is not always where master stopped. The warning is gone. A DEBUG log line now records each refused link, and another records each late link.
Parent of a process, not of a thread. The parent is now stored once per thread group. If the thread that forked a child exits, readers get the first live thread of the same parentprocess, which is the thread that master also chose. Invalid threads (unknown pid, no thread group) now have no ancestry at all. On master, sinsp followed their own ptid.
Writing the ptid. A plugin write to the ptid field of a thread in a thread group now fails with SS_PLUGIN_FAILURE, and set_ptid() throws in C++. On master, the write changed the parent. A write to a new entry before it joins the
table still works.
reaper_tid field. The kernel sends
reaper_tid=0for childless processes. On master, if sinsp believes the process has no children, it leaves the reaper_tid untouched (stays at -1). In this branch, we don't do this check (it's no longer cheap) and always overwrite reaper_tid with what the kernel sent, so childless processes will see their reaper tid change from -1 to 0.Memory. A dead process with live children stays in memory as a tombstone, a dead thread group record, until the children link past it (on their next clone or execve) or exit. A tombstone is invisible through the thread table and the
group table.
Performance. The exit of a process with N children no longer touches the children, so it costs O(1) and follows pointers instead of looking up a tid at each hop. A walk through many dead ancestors gets slower until the child's next clone or execve, and much slower when the reaper is unknown, because each read searches again. This sounds bad but is an edge case of an edge case (we'd need a deep tree of dead parents, which doesn't really happen in the real world as you generally fork twice, not 20 times, to daemonize, and we'd need to have no reaper information from the driver)
API removals. The child list, the loop methods, and the DEFAULT_EXPIRED_CHILDREN_THRESHOLD macro are removed, m_ptid is private, and get_thread_list() has a new return type. These are code changes, not behavior, and the "action required" release notes list them.
Does this PR introduce a user-facing change?:
🤖 Generated with Claude Code
🧠 Reviewed and edited by a protein-based proxy