[Bug]: network_settings beacon leaks a pyroute2 NDB instance per loader refresh — unbounded thread growth and steadily rising minion CPU #70036

What happened?

salt/beacons/network_settings.py instantiates NDB() at module import time and never closes it:

try:
from pyroute2 import NDB
from pyroute2.ndb.compat import ipdb_interfaces_view

IP = NDB()          # <-- constructed at import, never closed
HAS_PYROUTE2 = True
HAS_NDB = True

except ImportError:
IP = None
HAS_NDB = False
HAS_PYROUTE2 = False
Salt's LazyLoader re-executes beacon modules whenever the beacon loader is rebuilt (grain refresh, saltutil.sync_beacons, highstate). Each re-execution constructs a new NDB() — each with its own thread, netlink socket, and in-memory SQLite database — and orphans the previous instance. Nothing retains a reference to the old object, so nothing ever closes it and its receiver thread runs for the life of the process.

The threads accumulate indefinitely. Every one of them processes every netlink neighbour event the kernel emits, and they contend on the GIL, so minion CPU rises steadily with uptime until the service is restarted.

This affects minions that do not configure the network_settings beacon at all. The leak happens at import, not at execution, so beacons.disable_beacon network_settings does not prevent it.

Impact
On our fleet this presented as minions "randomly" pegging a core with completely silent logs and no jobs running — the symptom is invisible at log_level: info because nothing in the leaked threads logs anything. It reads as an intermittent fault but is actually monotonic: CPU climbs with uptime, and a restart resets it, which makes it easy to misattribute to whatever was running at the time.

Evidence
Minion, Salt 3008.2, onedir, Python 3.14, uptime 79 hours, no network_settings beacon configured:

/var/cache/salt/minion/proc/ empty (no jobs in flight), yet the main minion process held a sustained 58.4% CPU over a 10s sample.

Thread count tracks uptime in hours — 78 threads at 79 hours uptime, i.e. approximately one leaked instance per hour, matching the hourly highstate/mine schedule. Thread IDs increase monotonically (4703, 5010, 35376, 67472 … 2268897), confirming progressive accumulation rather than a pool.

py-spy dump — dozens of threads all parked in the same place:

Thread 1041251 (idle)
execute (pyroute2/ndb/schema.py:451)
load_netlink (pyroute2/ndb/schema.py:899)
load_ndmsg (pyroute2/ndb/objects/neighbour.py:36)
receiver (pyroute2/ndb/task_manager.py:133)
_run (asyncio/events.py:94)
_run_once (asyncio/base_events.py:2056)
run_forever (asyncio/base_events.py:677)
run_until_complete (asyncio/base_events.py:706)
run (asyncio/runners.py:128)
run (asyncio/runners.py:205)
main (pyroute2/ndb/task_manager.py:197)
run (threading.py:1024)
_bootstrap_inner (threading.py:1082)
_bootstrap (threading.py:1044)
strace -c -f over 15s — the CPU is lock contention, not work:

% time seconds usecs/call calls errors syscall

96.27 275.698710 1268 217320 107610 futex
2.59 7.414526 17909 414 epoll_pwait
1.08 3.087155 1543577 2 clock_nanosleep
0.05 0.132764 1097 121 recvfrom
voluntary_ctxt_switches: 14573219. The process sits in S state throughout despite the high CPU — consistent with GIL churn across 78 threads rather than compute. RSS was ~411MB, largely the accumulated in-memory SQLite databases.

Master connections were healthy ESTAB on 4505/4506 throughout — this is not a transport problem.

Steps to reproduce
Install a 3008.2 minion on Linux with pyroute2 available (the onedir bundles it).
Do not configure the network_settings beacon.
Schedule anything that triggers a beacon-loader refresh on an interval (an hourly state.apply is sufficient).
Record the main minion process thread count:
ls /proc/$(systemctl show -p MainPID --value salt-minion)/task | wc -l
Re-check after several hours. Thread count grows by roughly one per refresh and never falls; process CPU rises in step.

Expected behavior
Beacon modules should not create long-lived OS resources at import time, and a beacon that is not configured should create none at all. Repeated loader refreshes should not accumulate threads, sockets, or databases.

Additional context
Introduced when pyroute2.IPDB was replaced by pyroute2.NDB (#62218, shipped in 3006.0), so 3006.x and 3007.x are likely affected in the same way — we have only confirmed 3008.2.

Proposed fix: keep the imports at module scope so virtual() behaviour is unchanged, and defer NDB() / IPDB() construction to first use inside beacon(). That alone eliminates the leak entirely for the (common) case where the beacon is not configured.

Note that lazy initialisation into a plain module-level global is not sufficient on its own when the beacon is configured: because the loader re-executes the module, the global is reset each refresh and a new NDB is still constructed. That case needs either a process-wide cache that survives re-import, or explicit teardown of the previous instance. We are running the former locally as a _beacons/ override and it holds thread count flat.

Secondary bug in the same block. The legacy IPDB fallback clears IP on ImportError:

try:
from pyroute2 import IPDB

if IP is None:
    IP = IPDB()
    HAS_PYROUTE2 = True

except ImportError:
IP = None # <-- wipes a working NDB handle
If pyroute2 ships without IPDB (it is slated for removal) but NDB imported successfully, this sets IP = None while leaving HAS_PYROUTE2 = True. virtual() then returns the virtualname and beacon() fails on a None handle. The except branch should not discard an NDB instance that was already obtained.

(Note. Claude.ai assisted for diagnostics, patch, and github issue details. I'd spend weeks beating my head on it with no luck.)

Type of salt install

Official deb

Major version

3008.x

What supported OS are you seeing the problem on? Can select multiple. (If bug appears on an unsupported OS, please open a GitHub Discussion instead)

debian-13, ubuntu-24.04

salt --versions-report output

Salt Version:
               Salt: 3008.2

Python Version:
             Python: 3.14.6 (main, Jun 11 2026, 02:19:05) [GCC 11.2.0]

Dependency Versions:
               cffi: 2.0.0
           cherrypy: 18.10.0
       cryptography: 48.0.0
           dateutil: 2.9.0.post0
          docker-py: Not Installed
              gitdb: 4.0.12
          gitpython: 3.1.50
             Jinja2: 3.1.6
            libgit2: 1.9.4
       looseversion: 1.3.0
           M2Crypto: Not Installed
               Mako: Not Installed
            msgpack: 1.1.2
       msgpack-pure: Not Installed
       mysql-python: Not Installed
          packaging: 24.0
          pycparser: 3.00
           pycrypto: Not Installed
       pycryptodome: 3.23.0
             pygit2: 1.19.3
       python-gnupg: 0.5.6
             PyYAML: 6.0.3
              PyZMQ: 27.1.0
             relenv: 0.22.14
              smmap: 5.0.2
            timelib: 0.3.0
            Tornado: 6.5.7
                ZMQ: 4.3.5

Salt Extensions:
 saltext.kubernetes: 2.1.0
      saltext.mysql: 1.1.0
    saltext.proxmox: 2.0.1

Salt Package Information:
       Package Type: onedir

System Versions:
               dist: ubuntu 24.04.4 noble
             locale: utf-8
            machine: x86_64
            release: 6.8.0-136-generic
             system: Linux
            version: Ubuntu 24.04.4 noble