Skip to content

Plugin based kittest inspector#8136

Draft
lucasmerlin wants to merge 37 commits into
lucas/kittest-pluginfrom
lucas/kittest-inspector-plugin
Draft

Plugin based kittest inspector#8136
lucasmerlin wants to merge 37 commits into
lucas/kittest-pluginfrom
lucas/kittest-inspector-plugin

Conversation

@lucasmerlin

Copy link
Copy Markdown
Collaborator

Alternative implementation of the inspector from #8119, based on the plugin api from #8135

@github-actions

github-actions Bot commented Apr 27, 2026

Copy link
Copy Markdown

Preview available at https://egui-pr-preview.github.io/pr/8136-lucaskittest-inspector-plugin
Note that it might take a couple seconds for the update to show up after the preview_build workflow has completed.

View snapshot changes at kitdiff

@lucasmerlin lucasmerlin force-pushed the lucas/kittest-inspector-plugin branch 2 times, most recently from eab06d9 to 8350402 Compare April 27, 2026 11:40
@lucasmerlin lucasmerlin force-pushed the lucas/kittest-plugin branch from 6ac83f8 to 5e7c70b Compare May 21, 2026 08:50
@lucasmerlin lucasmerlin force-pushed the lucas/kittest-inspector-plugin branch 3 times, most recently from 2634e78 to f4d60f0 Compare May 22, 2026 09:41
emilk and others added 21 commits May 22, 2026 11:57
The `zip(a, b)` variant produces clearer code imho.

Downside: added dependency on `itertools`
* Part of #8180

So far, you've been able to move any `egui::Window` by dragging anywhere
on it. This makes sense on touch screens with thick fingers, but less so
on non-touch-screens.

With this PR, you can now control it with a new enum `WindowDrag`
## Related
* #5851
* #7988

## What
We want to make it easier to understand the lineage of a `Id` (which is
the parent `Id`, and the parent of that, etc?)

As a first step of that, we want to clarify the different between a
globally unique `Id`, and an `IdSalt`

I also introduced the `AsId` and `AsIdSalt` traits, which are
implemented of anything that implements `Hash` and `Debug`. The `Debug`
half of that is unused here, but will later be used in `Debug` builds to
produce a proper tree.
* Closes #8029

---------

Co-authored-by: lucasmerlin <hi@lucasmerlin.me>
## What

Adds a way for apps to push an RGBA bitmap as the OS cursor — the
missing companion to `Context::set_cursor_icon`. The integration
translates it into a real `winit::CustomCursor`, so the cursor is drawn
by the compositor and can extend past the egui window edge like any
native cursor.

## Why

Apps with custom-shaped windows (Winamp-style skins, themed launchers,
kiosk apps) currently have no clean way to display a custom cursor:
- `CursorIcon` is limited to the standard system enum.
- Painting the cursor sprite via `egui::Painter` works inside the canvas
but gets clipped at the window edge — the bottom/right of the cursor
disappears the moment the pointer is near the boundary, and there's no
way to render onto the desktop area exposed by a
transparent/region-shaped window.

`winit` 0.30+ already supports `CustomCursor::from_rgba` +
`ActiveEventLoop::create_custom_cursor`, but `egui-winit` doesn't
surface it. This PR exposes it through egui.

### Visual demonstration

Driving use case: a Winamp WSZ skin player
([all3f0r1/oneamp](https://github.com/all3f0r1/oneamp)) with a
transparent + region-shaped window where the skin ships its own `.cur`
files. The bottom-right corner of the playlist exposes the resize cursor
— notice how it gets clipped at the window edge in the painter-based
approach.

| Before (cursor painted via `egui::Painter`) | After (cursor pushed via
`set_cursor_image`) |
| --- | --- |
| ![cursor clipped at the bottom-right of the playlist
window](https://raw.githubusercontent.com/all3f0r1/egui/pr-assets/cursor-clipping-before.png)
| ![cursor extends cleanly past the window edge onto the
desktop](https://raw.githubusercontent.com/all3f0r1/egui/pr-assets/cursor-clipping-after.png)
|

## API

```rust
// new in egui::data::output
pub struct CustomCursorImage {
    pub rgba: std::sync::Arc<[u8]>,
    pub size: [u16; 2],     // matches winit's u16 to avoid lossy casts
    pub hotspot: [u16; 2],
}

// new field on PlatformOutput (skipped from serde — ephemeral)
pub cursor_image: Option<CustomCursorImage>,

// new method on Context
ctx.set_cursor_image(Some(image)); // overrides cursor_icon for this frame
ctx.set_cursor_image(None);        // revert to cursor_icon
```

`Arc<[u8]>` is intentional: the integration dedupes by `Arc::as_ptr`, so
reusing the same Arc across frames means the bitmap is only uploaded to
the OS once per skin, not once per frame.

## Integration changes

- `egui_winit::State::handle_platform_output_with_event_loop(window,
Option<&ActiveEventLoop>, ...)` is a new method that threads the active
event loop so it can call `event_loop.create_custom_cursor(...)`.
- The legacy `handle_platform_output(window, ...)` delegates with `None`
and silently drops `cursor_image`. **No existing callers break.**
- The icon and bitmap paths are unified in a private `apply_cursor`. The
no-flicker dedupe of the old `set_cursor_icon` is preserved on both
paths.
- If `CustomCursor::from_rgba` rejects the bitmap (bad dimensions,
hotspot OOB, etc.), we log a warning and fall back to the icon path.
- eframe's wgpu + glow integrations thread `&ActiveEventLoop` through
`run_ui_and_paint` (glow already had it; wgpu needed one extra
parameter) and call the new method.
- Immediate viewports keep the old path because they're invoked from a
`Context` callback that doesn't have an event loop reference. Custom
cursors are a no-op in immediate viewports — acceptable since they're a
niche path.

## Fallback semantics

| backend / context              | what happens                  |
|--------------------------------|-------------------------------|
| eframe wgpu/glow main viewport | bitmap displayed via OS       |
| eframe immediate viewport      | falls back to `cursor_icon`   |
| eframe web                     | falls back to `cursor_icon`   |
| custom integrations not opted in | falls back to `cursor_icon` |
| `from_rgba` returns `BadImage` | warning + falls back to icon  |

## Verification

- `cargo fmt --all -- --check` ✅
- `cargo clippy -p egui -p egui-winit -p eframe --all-targets
--all-features -- -D warnings` ✅
- `cargo doc --lib --no-deps -p egui -p egui-winit -p eframe
--all-features` ✅
- `cargo check -p egui --no-default-features --features serde` ✅
(validates the `serde(skip)` on `cursor_image`)
- Interactive validation on Linux/Wayland with the OneAmp WSZ skin
player — see screenshots above.

I haven't run the full snapshot test suite (`scripts/check.sh`) because
we're on Linux and the snapshots are macOS-rendered — happy to run it if
you'd like.

## Notes

Drafted per the contributing guide ("open a draft PR, you may get
helpful feedback early"). Open to design feedback on:
1. Whether `CustomCursorImage` should live in `egui::viewport` rather
than `egui::data::output`.
2. Whether the legacy `handle_platform_output` should grow `event_loop`
directly (breaking) instead of getting a sibling method (non-breaking,
what I did).
3. Whether to also wire it through eframe-web (probably not —
`wasm-bindgen-cursor` would need its own path).

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Closes #7254

You can now drag-to-close a panel. Also drag-to-expand panels.

This is a breaking change: the animated panel functions now take a
`open: &mut bool` instead of `open: bool`.

This is only enabled for resizable panels
The three methods for showing a `Panel` are now:

* `panel.show`: always show the panel.
* `panel.show_collapsible`: show or hide the panel, with a slide
animation in between.
* `Panel::show_switched`: animate between two different panels: a
thin/collapsed one and a thick/expanded one.
Double-clicking the edge of a resizable panel will now toggle it
collapsed/expanded
Current behavior fails when translating file uris that contain windows
UNC paths. This commit attempts to fix that behavior by looking at the
hostname attribute of the uri and changing behavior if the hostname is
present.

* Closes <#8161>
* [x] I have followed the instructions in the PR template
This changes the monitor selection used when restoring a persisted
window position.

Currently, `egui-winit` picks a monitor by checking whether the saved
window position fits inside a loose monitor range. This can choose the
wrong monitor when a saved window rectangle slightly overlaps another
monitor.

My failure case was on Windows with two monitors:
- primary monitor on the right
- secondary monitor on the left
- window maximized on the primary monitor
- persisted outer position was slightly negative, e.g. `x = -8`, because
of the invisible window border

That position matched both monitor ranges, so the restored maximized
window could
open on the secondary monitor instead of the primary one.

This PR picks the monitor with the largest overlap with the saved window
rectangle instead.

Related note: probably the best solution would be to save the normal
window position when maximized, so that when unmaximizing, the window
would get restored to the previous state. It's mentioned in this comment
#3494 (comment). I
tried doing that in
[fix-windows-maximized-restore-placement](https://github.com/YelovSK/egui/tree/fix-windows-maximized-restore-placement),
and it works, but it requires adding windows-sys as a dependency to call
a relevant winapi, so that's probably not the right solution. Winit
doesn't seem to provide an API that would return this information.

* [x] I have followed the instructions in the PR template

---------

Co-authored-by: Emil Ernerfeldt <emil.ernerfeldt@gmail.com>
`ab_glyph` would output coverage values, but `vello` outputs RGBA. So
the old name was a misnomer.

I also suspect our default values are wrong, but I need to investigate
that more properly in a separate PR.
* Closes #8055
* Closes #8056

This changes the behavior of `Panel` to NEVER overflow
`Panel::max_size`, nor the available space in the parent UI.

If you do overflow it, the content will be silently clipped.
* Follows #8199

This makes the animation of the collapsing panel a bit smoother, by
taking into account the spacing between the header and the body.
New `egui_inspection` crate ships:
- `protocol` (default): wire types + length-prefixed msgpack framing for the
  inspector ↔ egui-peer connection. Transport-neutral (stdio / unix socket / TCP).
- `plugin`: `InspectionPlugin`, an `egui::Plugin` that dials a unix socket from
  `EGUI_INSPECTION_SOCKET`, streams frames + accesskit tree updates, and applies
  inbound `InspectorCommand`s back into the running `egui::Context`.

eframe gains an `inspection` feature that auto-attaches the plugin during native
startup (glow + wgpu integrations) when the env var is set. Connection failures
log via `log::warn!` and do not abort startup.

Lives in its own crate (rather than `egui_kittest`) so eframe can pull the
protocol in without picking up the test harness, and so external tools can
depend on it directly.
@lucasmerlin lucasmerlin force-pushed the lucas/kittest-inspector-plugin branch from f4d60f0 to 4a5daa6 Compare May 26, 2026 09:41
Replace std::os::unix::net::UnixStream in the InspectionPlugin with the
interprocess crate's local_socket::Stream, so the transport works on Windows
(named pipe) as well as unix/macOS (unix domain socket).

- New transport module (transport feature) with socket_name() and
  generate_socket_target() — one shared, platform-split place to build/allocate
  local-socket names, used by both ends of the connection.
- Drop the cfg(unix) gates on the plugin module; gate on the plugin feature only.
- attach() now takes a socket name string and connects via interprocess; the
  stream is split with Stream::split() instead of UnixStream::try_clone().
@lucasmerlin lucasmerlin force-pushed the lucas/kittest-inspector-plugin branch from 4a5daa6 to 026a859 Compare May 26, 2026 12:27
…l feature

- FrameScreenshot now carries PNG bytes instead of raw RGBA (PROTOCOL_VERSION 1→2);
  add a shared `encode_png` helper behind a new `png` feature so the live plugin and the
  kittest harness encode frames identically.
- Make the protocol module unconditional: drop the `protocol` feature flag and the
  optional serde/serde_bytes/rmp-serde deps it gated.
- plugin.rs: re-stamp screenshot-bearing frames with the current step (so inspectors
  waiting for step > prev don't reject them) and pump a tail-side repaint while awaiting
  the GPU readback.
@lucasmerlin lucasmerlin force-pushed the lucas/kittest-inspector-plugin branch from 026a859 to 39978b7 Compare May 26, 2026 14:21
Add transport::connect (dial + split) and a sync transport::Listener
(bind + accept) so both ends of the inspection connection build streams
identically without depending on interprocess directly. Plugin now dials
via transport::connect. These back the kittest harness moving onto the
same local socket as the live plugin.
Replaces the separate on_accesskit_update hook with a `&TreeUpdate`
parameter on `after_step` — one hook per step, tree delivered inline.
`step_no_side_effects` now returns the TreeUpdate so plugins driving
the harness from within their own hook (where nested dispatches are
suppressed) can still see it.

Also adds `#[track_caller]` to the internal `_step` / `_step_no_side_effects`
/ `_try_run` so `Location::caller()` inside `step()` walks up to the
user's original call site when reached via `run()`.
Drop the redundant `plugin_` prefix on the dispatch helper (it's already a
method on `Harness`). Make `TestResult` `Copy` so it can be passed by value
inside the dispatch closure, removing the need for the `fail_ref` re-borrow
helper. Also drop the empty-plugins early-return microopt in `dispatch` —
the `mem::take`/`extend` dance is cheap on empty vecs.
The wrapping Option only existed so Drop could consume SnapshotResults
before plugins fire. mem::take swaps in a fresh default instead, dropping
the original — same outcome with no Option-juggling at every call site
(no more .as_mut().expect(...), and take_snapshot_results stops needing
.replace()).

The default SnapshotResults has handled = true so dropping the placeholder
is a no-op.
New `InspectorPlugin` (gated behind `inspector` feature) launches a
`kittest_inspector` child process and streams the harness's frames + accesskit
tree updates to it over framed MessagePack on stdin/stdout. The inspector
drives the harness by sending `InspectorCommand`s back; supported commands
include `Step` / `Run` / `Play` / `Pause` (deterministic stepping),
`Handle { events }` (event injection), `Resize`, and `Screenshot`.

Auto-attaches when the `KITTEST_INSPECTOR` env var is truthy — the inspector
binary path can be overridden via `KITTEST_INSPECTOR_PATH`. Uses the new
`egui_inspection::protocol` types and starts every connection with a
`HarnessMessage::Hello { peer_kind: Kittest, capabilities: KITTEST }` so the
inspector can render the right controls.

Also re-exports `egui_inspection` as `egui_kittest::inspector_api` for crates
that only depend on kittest.
…tion::encode_png

- Encode harness frames with egui_inspection::encode_png instead of sending raw RGBA, so
  the inspector socket carries compressed bytes.
- Features: pull egui_inspection/png + image/png for the encoder; drop the now-removed
  egui_inspection `protocol` feature.
The harness inspector now speaks the wire protocol over the same
interprocess local socket as the live egui_inspection plugin, in two
modes:
- connect: EGUI_INSPECTION_SOCKET set -> dial the listening socket
  (e.g. the kittest MCP bridge).
- spawn: KITTEST_INSPECTOR truthy, no socket -> bind a socket, spawn
  the kittest_inspector binary pointed at it, accept.

env_enabled() now also auto-enables when the socket var is set. Pulls
egui_inspection/transport into the inspector_api feature.
@lucasmerlin lucasmerlin force-pushed the lucas/kittest-inspector-plugin branch from 39978b7 to e3bdddf Compare May 26, 2026 14:34
@lucasmerlin lucasmerlin added this to the 0.35.0 milestone May 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants