Add hook for formatting Kubernetes Event messages by agoose77 · Pull Request #910 · jupyterhub/kubespawner

agoose77 · 2026-03-03T12:57:38Z

TL;DR

Note

No LLMs were used in the authoring of this PR.

2i2c is currently working on a user-story to improve the kubespawner progress messages, as part of an initiative to improve the spawn-progress page.

This PR does several things:

Adds a custom decorate_progress_message that overrides the pretty-printing of event messages.
Adds a kubespawner.events module for richer built-in formatting of log messages.
Adds a kubespawner.events.RuleEventFormatter and other types for defining event formatting rules.

See the before and after:

Before

After

Goal

The goal is modest: to improve the human readability of spawn messages, and to allow further customisation.

Example Decoration Hook

Basic hook

def decorate_progress_message(spawner, event, text):
    return { 
        "message": f"custom-message-{text}",
        "html_message": f"<span>{text}</span>"
    }
c.KubeSpawner.decorate_progress_message = decorate_progress_message

Use the rules to define custom renderers

c.RuleEventFormatter.rules = {
    "01-container-image-events": {
        "match": {
            "reportingComponent": r"kubelet",
            "fieldPath": r"spec\.(?P<container>initContainers|containers)\{([^}]+)\}",
            "reason": r"(?P<action>Pulling|Pulled)",
        },
        "template": "{action} {image} image for the {container} container",
    },
    "02-container-lifecycle-events": {
        "match": {
            "reportingComponent": r"kubelet",
            "fieldPath": r"spec\.(?P<container>initContainers|containers)\{([^}]+)\}",
            "reason": r"(?P<action>Started|Killing|Created|Stopped)",
        },
        "template": "{action} the {container} container",
    },
    "03-pod-resource-events": {
        "match": {
            "reportingComponent": r"kubelet",
            "reason": r"OutOf(?P<resource>memory|cpu|ephemeral-storage|pods)",
        },
        "template": "The node selected to run your server ran out of {resource}",
    },
    "04-scheduler-node-found-events": {
        "match": {
            "reportingComponent": r".*-user-scheduler",
            "reason": r"Scheduled",
            "message": r".*?assigned \S+ to (?P<node>\S+)",
        },
        "template": "A node ({node}) has been found to run your server",
    },
    "05-scheduler-no-nodes-events": {
        "match": {
            "reportingComponent": r".*-user-scheduler",
            "reason": r"FailedScheduling",
        },
        "template": "No existing nodes are currently able to run your server",
    },
    "06-cluster-autoscaler-events": {
        "match": {
            "reportingComponent": r"cluster-autoscaler",
            "reason": r"TriggeredScaleUp",
        },
        "template": "Launching new nodes by scaling up the cluster",
    },
    "07-node-affinity-events": {
        "match": {
            "reportingComponent": r"kubelet",
            "message": r"Predicate NodeAffinity failed.*",
            "reason": r"NodeAffinity",
        },
        "template": "It was not possible to find or launch any nodes to run your server. This is likely due to a configuration problem with the infrastructure or the JupyterHub",
    },
    "08-gke-scheduler-node-found-events": {
        "match": {
            "reportingComponent": r"gke\.io/optimize-utilization-scheduler",
            "reason": r"Scheduled",
            "message": r".*?assigned \S+ to (?P<node>\S+)",
        },
        "template": "A node ({node}) has been found to run your server",
    },
    "09-gke-scheduler-no-nodes-events": {
        "match": {
            "reportingComponent": r"gke\.io/optimize-utilization-scheduler",
            "reason": r"FailedScheduling",
        },
        "template": "No existing nodes are currently able to run your server",
    },
    "10-taint-eviction-events": {
        "match": {
            "reportingComponent": r"taint-eviction-controller",
            "reason": r"gke\.io/optimize-utilization-scheduler",
            "message": r"Cancelling deletion of Pod.*",
        },
        "template": "Cancelling deletion of your server. This normally happens when a scale-up has just taken place.",
    },
}

Design Details

UI

Timestamps are formatted to regular isoformat-like %Y-%m-%DTHH:MM:SSZ to keep fixed width
Timestamps and message types are pretty formatted as button-pills
Messages are simplified where possible

Constraints

I targeted Python 3.7 given pyproject.toml, meaning no match, :=, or removeprefix.

Questions

Is Kubespawner making too many assumptions if we bake-in the expectation of Bootstrap?
Could we consider adding a start-time timestamp so that times can simply be given as "minutes since spawn" rather than UTC times?

manics · 2026-03-03T16:20:00Z

This does effectively vendor some cluster-provider specifics (like the GCP scheduler). I think that's OK? But if we are vehemently against that, we can just pull those parts out.

Whatever we decide we need to be consistent in future. If we add GCP specific code we need to accept code for other clouds, including from third parties who use platforms that we can't test ourselves.

jnywong

I enjoy this feature ❤️ I like that the format_event_hook was easy to configure for basic formatting. It took me a while to understand what was going on with the default formatting, but I think there will alway be room for improvements there.

In general, my main comment is to complete the test suite to regenerate how the sample-events were created through the message.py, since I think that represents the bulk of the work in this PR. The default formatting may undergo further development, but like you say I think we want to add in some basic regression testing and update that if needed in future.

In answer to your questions:

Is Kubespawner making too many assumptions if we bake-in the expectation of Bootstrap?

We know Bootstrap ships with JupyterHub, so I think this is a safe assumption for now. I don't think we need to worry about this for BinderHub?

Could we consider adding a start-time timestamp so that times can simply be given as "minutes since spawn" rather than UTC times?

I think this is a nice-to-have. Most users hopefully shouldn't have to dwell on the spawn progress screen, but if they do, then they will likely screenshot their spawn failure to send to an admin. Keeping the timestamps consistent with server side logs with raw k8s events should be useful for sysadmins for troubleshooting.

Should I rework the built-in formatter to generate HTML at every stage — would it be useful to have e.g. image names/tags, and node names in button-tags?

At this stage, I would prefer not to have the default formatter be too flashy.

Is there motivation to move the default message format into its own configurable, rather than requiring users to create their own hook?

Yes, out of all of these questions I think this would be one to focus on. Most people will probably want to configure an extension to the default formatter.

jnywong · 2026-03-05T17:30:30Z

This does effectively vendor some cluster-provider specifics (like the GCP scheduler). I think that's OK? But if we are vehemently against that, we can just pull those parts out.

Whatever we decide we need to be consistent in future. If we add GCP specific code we need to accept code for other clouds, including from third parties who use platforms that we can't test ourselves.

I am okay with that – I don't think we need to assume full responsibility for testing code on platforms we don't have access to, but we should ensure contributors who would like this functionality to include full test suites for that. I think the scope of changes in this PR are pretty cosmetic, so there doesn't seem to be huge scope for someone to introduce anything too crazy on a third party platform.

There is a small question about how to structure this as the corpus of messages to reformat scales, but I think we can cross that bridge when we get to it?

yuvipanda

Thank you for working on this, @agoose77! I left a comment about changing the implementation to be a lot more declarative than it is now, which should hopefully make both maintenance and extension much easier.

agoose77 · 2026-03-06T11:38:34Z

@yuvipanda I've reworked the PR to leave the functional hook that completely bypasses event formatting and remove the default formatter callable.

I think the test failure is just a flaky test?

I've then added a rule system with no defaults. This was a two-fold decision:

It keeps kubespawner leaner (no cloud specific functionality)
It resolve the problem of making this overrideable.¹

As such, I intend to put the specific implementation of these rules in the z2hj chart instead. I'm happy to revert that decision, but I am not off-the-top-of-my-head aware of a nice way to do it other than defining the default value as a dict and implementing the extra functionality in kubespawner itself, where users could clobber the default names and set them to None (plus None handling logic to remove these).

My knowledge of traitlets is that one can't refer to the default value of a trait. This means that to allow users to override this, e.g. z2jh would literally have to import the HasTraits-derived parent class and extract the default to compose them. ↩

agoose77 · 2026-03-06T15:36:09Z

+    await spawner.stop()
+
+
+@pytest.mark.parametrize("rules_as_dict", [True, False])


TODO: test validation logic

yuvipanda · 2026-03-06T19:03:28Z

I haven't looked at the implementation yet, but thank you for reworking it! I think the rules system should be here, not in z2jh. You can make it extensible by having a default list in here, and then just allowing extra_event_formatter_rules or similar that are appended. This is essential functionality that will benefit everyone using kubespawner, and I'm not concerned about a few extra rules that are cloud provider specific, especially as they also mostly come from the open source autoscaler project. We aren't doing anything specific to any cloud provider here, but supporting things from the autoscaler which has plugins for cloud specific functionality.

This reverts commit 510c01c.

This makes it even easier to functionally override formatted messages, and brings us in-line with existing hooks

Co-authored-by: Jenny Wong <jnywong.pro@gmail.com>

for more information, see https://pre-commit.ci

agoose77 added 4 commits March 3, 2026 13:22

feat: add format_event_hook hook

e9100fb

test: add test for progress formatter

00c24bf

feat: add built-in formatting for common k8s events

119dd5c

feat: add CLI

d4d1e0e

agoose77 force-pushed the feat-add-hook branch from b6165ef to 65a0c71 Compare March 3, 2026 13:26

agoose77 commented Mar 3, 2026

View reviewed changes

Comment thread kubespawner/spawner.py Outdated

agoose77 force-pushed the feat-add-hook branch from 5559607 to c375648 Compare March 3, 2026 13:30

agoose77 mentioned this pull request Mar 3, 2026

[Compute usage quotas] Plan and author upstream PR for kubespawner message formatting 2i2c-org/infrastructure#7718

Closed

1 task

agoose77 commented Mar 3, 2026

View reviewed changes

Comment thread kubespawner/messages.py Outdated

agoose77 marked this pull request as ready for review March 3, 2026 14:00

jnywong mentioned this pull request Mar 3, 2026

[Compute usage quotas] Review upstream kubespawner PR 2i2c-org/infrastructure#7823

Open

2 tasks

agoose77 added 4 commits March 3, 2026 15:26

feat: add rich-repr

67fea13

fix: handle timestamps carefully

7188e8d

fix: update type anotations to use Optional

3f2fcc3

refactor: upper-snake default formatters

22d1dd1

agoose77 force-pushed the feat-add-hook branch 3 times, most recently from 4cf984c to ed9f488 Compare March 3, 2026 16:00

agoose77 force-pushed the feat-add-hook branch 3 times, most recently from dc81d08 to 8e4f471 Compare March 3, 2026 17:28

jnywong reviewed Mar 5, 2026

View reviewed changes

Comment thread kubespawner/spawner.py

Comment thread kubespawner/spawner.py Outdated

Comment thread tests/test_spawner.py Outdated

Comment thread kubespawner/messages.py Outdated

Comment thread kubespawner/messages.py Outdated

Comment thread kubespawner/messages.py Outdated

yuvipanda requested changes Mar 5, 2026

View reviewed changes

Comment thread kubespawner/messages.py Outdated

agoose77 force-pushed the feat-add-hook branch from 0cc0c57 to 6be4627 Compare March 6, 2026 11:19

agoose77 commented Mar 6, 2026

View reviewed changes

agoose77 force-pushed the feat-add-hook branch from 7aef1bb to fac0e1a Compare March 10, 2026 17:06

agoose77 added 12 commits April 23, 2026 18:05

feat: add default rules

a1c4567

test: restore tests

46f27f3

This reverts commit 510c01c.

feat!: rename hook

afcbd06

This makes it even easier to functionally override formatted messages, and brings us in-line with existing hooks

fix: use raw strings for readability

5235741

tests: re-build regression tests

0f5858d

test: add tests for extends

9999655

fix: is None → is not None

2514020

fix: user or default scheduler

ca07cc3

fix: simplify scheduler regex

3eeaa50

fix: allow prefixed schedulers

cbaa26a

test: fix assertion

3a7f1de

refactor: rename messages to events

93b3d5b

agoose77 force-pushed the feat-add-hook branch from 9217cdc to 53058dd Compare April 23, 2026 17:22

agoose77 added 3 commits April 23, 2026 18:35

refactor: pull out event matching logic

a8c4bfa

refactor: lift more into events

30fed4c

refactor: add EventFormatter configurable

22f9c6e

agoose77 force-pushed the feat-add-hook branch from 53058dd to 9aede42 Compare April 23, 2026 17:36

agoose77 and others added 10 commits April 23, 2026 18:43

refactor: move pretty decoration into separate hook

2bbe73b

docs: add docs on events

2523abf

refactor: remove unused import

d9dff9f

Apply suggestions from code review

d4d8002

Co-authored-by: Jenny Wong <jnywong.pro@gmail.com>

docs: add better docs for template arg

60f37f5

test: fix format

0c46cc0

test: add test for error in templating

2283050

fix: pass through fallback rule ID

8888f4f

feat: add rules for pull failures

3ab5ad7

feat: add regro test

fda0f9d

agoose77 force-pushed the feat-add-hook branch from 9aede42 to fda0f9d Compare April 23, 2026 17:43

[pre-commit.ci] auto fixes from pre-commit.com hooks

4c2e1b7

for more information, see https://pre-commit.ci

jnywong mentioned this pull request Apr 28, 2026

Optionally mute k8s event progress messages #916

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add hook for formatting Kubernetes Event messages#910

Add hook for formatting Kubernetes Event messages#910
agoose77 wants to merge 51 commits into
jupyterhub:mainfrom
agoose77:feat-add-hook

agoose77 commented Mar 3, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

manics commented Mar 3, 2026

Uh oh!

jnywong left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jnywong commented Mar 5, 2026

Uh oh!

yuvipanda left a comment

Uh oh!

Uh oh!

agoose77 commented Mar 6, 2026 •

edited

Loading

Uh oh!

agoose77 Mar 6, 2026

Uh oh!

yuvipanda commented Mar 6, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		await spawner.stop()


		@pytest.mark.parametrize("rules_as_dict", [True, False])

Conversation

agoose77 commented Mar 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TL;DR

Goal

Example Decoration Hook

Design Details

UI

Constraints

Questions

Uh oh!

Uh oh!

Uh oh!

manics commented Mar 3, 2026

Uh oh!

jnywong left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jnywong commented Mar 5, 2026

Uh oh!

yuvipanda left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

agoose77 commented Mar 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Footnotes

Uh oh!

agoose77 Mar 6, 2026

Choose a reason for hiding this comment

Uh oh!

yuvipanda commented Mar 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

agoose77 commented Mar 3, 2026 •

edited

Loading

jnywong left a comment •

edited

Loading

agoose77 commented Mar 6, 2026 •

edited

Loading

yuvipanda commented Mar 6, 2026 •

edited

Loading