Skip to content

Paginate the initial list with limit/continue#1309

Open
casassg wants to merge 3 commits into
nolar:mainfrom
casassg:gerardc/paginate-initial-list
Open

Paginate the initial list with limit/continue#1309
casassg wants to merge 3 commits into
nolar:mainfrom
casassg:gerardc/paginate-initial-list

Conversation

@casassg

@casassg casassg commented Jun 1, 2026

Copy link
Copy Markdown

Add an opt-in settings.watching.batch_size to fetch the initial listing (the list preceding every watch) in limit/continue pages instead of one large request, capping the cold-start peak-memory spike for high-cardinality cluster-wide watches (e.g. Pods/Jobs).

  • list_objs() paginates via limit/continue when batch_size is set, aggregating items across pages.
  • Keeps the snapshot resourceVersion (consistent across chunks) to start the watch from the fully-listed state.
  • Fully backward compatible: batch_size defaults to None → single unpaginated request, unchanged URL/params.
  • tests/k8s/test_list_objs.py (kmock-based) covers the default single request, multi-page aggregation + query params, empty-continue stop, and kind/apiVersion defaulting across pages.

Why

list_objs() issues one unpaginated GET and parses the entire collection into memory at once before the watch begins. For operators/observers watching high-cardinality built-in resources cluster-wide, the cold-start list materializes every object simultaneously — a peak-memory spike scaling with total object count.

We hit this with a Prefect Kubernetes worker, which embeds a kopf observer watching all Pods/Jobs cluster-wide (prefect_kubernetes/observer.py): after a restart, the cold-start list of thousands of objects pushed the container past its memory limit and it crash-looped (OOMKilled). The Kubernetes API already supports chunked retrieval for exactly this (Retrieving large results in chunks), and client-go's reflector paginates its list by default — kopf just wasn't using it.

Example usage

@kopf.on.startup()
def configure(settings: kopf.OperatorSettings, **_):
    settings.watching.batch_size = 500   # page size; None (default) = single request

@casassg casassg requested a review from nolar as a code owner June 1, 2026 22:34
@casassg casassg marked this pull request as draft June 1, 2026 22:37
@casassg casassg force-pushed the gerardc/paginate-initial-list branch from 160cd7e to 8bc8527 Compare June 2, 2026 16:28
casassg added 3 commits June 2, 2026 11:39
The initial listing that precedes every watch fetches the whole
collection in a single request and parses it into memory at once. For
operators/observers that watch high-cardinality built-in resources
(e.g. Pods or Jobs) cluster-wide, this produces a large peak memory
spike on cold start, proportional to the total number of objects in
the cluster.

Add an opt-in `settings.watching.batch_size` that fetches the initial
list in chunks using the Kubernetes API's `limit`/`continue`
pagination, capping the peak memory footprint of the bootstrap listing.
The snapshot resourceVersion (consistent across all chunks) is used to
start the watch from the fully listed state.

Defaults to `None`, preserving the current single-request behaviour.

Signed-off-by: Gerard Casas Saez <gerardc@squareup.com>
Signed-off-by: Gerard Casas Saez <gerardc@squareup.com>
Signed-off-by: Gerard Casas Saez <gerardc@squareup.com>
@casassg casassg force-pushed the gerardc/paginate-initial-list branch from 8bc8527 to 8cf2f6a Compare June 2, 2026 16:40
@casassg casassg changed the title feat: paginate the initial list with limit/continue Paginate the initial list with limit/continue Jun 2, 2026
@casassg casassg marked this pull request as ready for review June 2, 2026 16:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant