Editor-Ready UI

This is a Focus Area tracking issue

Before we can start work on the Bevy Editor, we need a solid UI implementation. Bevy UI already has nice "flexbox" layout, and we already have a first stab at buttons and interaction events. But Bevy UI still needs a lot more experimentation if we're going to find the "right" patterns and paradigms. Editor-Ready UI has the following requirements:

Embraces the Bevy architecture: Bevy ECS, Bevy Scenes, Bevy Assets, Bevy Events
A Canvas-style API for drawing widgets with shapes and anti-aliased curves
Define a consistent way to implement widgets
A core set of widgets: buttons, inputs, resizable panels, etc
Theme-ability
"Interaction" and "focus" events
Translation-friendly. We can't be anglo-centric here

Active Crates / Repos

No active crates or repos. Feel free to make one. Link to it in this issue and I'll add it here!

Sub Issues

No active issues discussing subtopics for this focus area. If you would like to discuss a particular topic, look for a pre-existing issue in this repo. If you can't find one, feel free to make one! Link to it in this issue and I'll add it to the index.

Documents

Collection of thoughts from Discord discussions: https://hackmd.io/lclwOxi3TmCTi8n5WRJWvw?view

I wanted to share a few quick thoughts on UI systems in games. I'm a gamedev professionally, and I read this list of requirements, and had a bit of a knee-jerk reaction. These are good features from a basic point of view, but sort of miss the boat on what I would consider are the harder questions around UI:

Structure: Immediate Mode vs. Retained Mode
Performance: How fast is a Noop render (where nothing in the UI changes)?
Styling: CSS-like, or in-code?
Flexibility: How easy is it to build any UI, vs. just 'widgets'

I'm happy to elaborate on any of these, but primarily I'd recommend trying to learn from where Unity is at the moment. They started with an immediate mode UI engine for both the Editor and the Runtime. Later, because of performance and flexibility, they adopted a retained mode UI for the Runtime. Now, as of two years ago, they've finally been building a (really fantastic) unified UI system for both the Editor and Runtime, open source.

This is a great talk on the new design (UIElements/UIToolkit): https://www.youtube.com/watch?v=zeCdVmfGUN0

I'd highly recommend taking as much learning as you can from their path. Frankly, both previous systems were a total mess. They seemed great in the moment, but just collapsed under any sort of complexity.

What makes UI challenging is that it's very hard to build incrementally, you have to plan a lot of these things in from the start. I don't think Bevy UI should be nearly as complicated as the latest Unity UI, but it's really worth learning, and not repeating their mistakes.

And also, if the intention for Bevy is to start off as a hobbyist engine, incremental is perfectly fine. I would just expect to have to rewrite the UI a few times as you want to attract larger projects.

Some early distillation of thoughts is happening here:
https://hackmd.io/lclwOxi3TmCTi8n5WRJWvw?view

At the moment anyone is free to add their particular domain experience. This is stopgap - suggestions are that Cart takes ownership of the hackmd / it's used as ingest for a github wiki/markdown page

Would it be worth also looking as SwiftUI as an example of prior art?

Would it be worth also looking as SwiftUI as an example of prior art?

Add your write-up of it to the doc if you’d like 👍

Can we add "screen-reader friendly" and "multiple input types" (keyboard, mouse) as hard requirements for any UI framework / UI primitives?

At the risk of being very ignorant, is screen-reader friendliness a high priority for a visual editor? I have a feeling visually impaired folks will have difficulty getting use out of the editor. Making a visual scene editor friendly to visually impaired users sounds like an entire research project. Mouse + keyboard navigation is a clear win for accessibility so I think I'd want to focus efforts there, given our limited resources.

I'm certainly not saying we shouldn't make ourselves screen reader friendly, as that will absolutely be useful in other apps, I'm just questioning if it should be an Editor-UI priority. I'm curious what the use cases would be for screen readers + visual editors.

Please tell me if I'm wrong here.

Yeah no worries, it was a genuine question. I'm not sure how well screen readers are supported in other editors, I am coming from a context of developing on the open web and it is really important in that context. I imagine native software is a lot more complicated.

The extent to which it is important comes down to how visual the editor is. If there are lots of text inputs (e.g. a way to change transform values of an entity numerically rather than just using click + drag) then I think screen reader support would be something to consider. If at least in the short-term we are looking at a more purely graphical interface then it is less important.

Also for the record, I don't use screen readers so I don't actually know what users would prefer in this case. Maybe folks would rather just edit the RON files directly. Let's not worry about it too much. :-)

Have been looking at SwiftUI and one of the nice things about the declarative approach is that it does mean your data is well structured for adding things like that in the future.

The general concepts they use are really elegant and I suspect would translate well to Rust, although it may be an "all or nothing" commitment to go that route.

Main principle is defining a tree of very lightweight units composed together to create consistent and predictable results with surprisingly little boilerplate and small number of primitive/leaf components. Data dependencies are all declared up front so the framework knows what to update as & when the underlying data changes and there's an "environment" concept so that data can be passed down the hierarchy without needing to be passed through every hop.

I quite like the bounds calculation process too, where parent gives its children size "suggestions", they respond with their desired size and parent determines ultimate positioning (but has to respect child's desired size)

Worth watching the WWDC '19 & '20 talks on it all for ideas.

@tektrip-biggles

@tektrip-biggles: FYI, @raphlinus has done a bunch of great research in this whole topic, as well as in comparison to existing systems like SwiftUI:

https://raphlinus.github.io/rust/druid/2019/10/31/rust-2020.html

A few thoughts:

Screen Reading: If the goal is for Bevy UI to support in-game UI as well as the editor, that means the UI will need to support gamepad-only and keyboard-only navigation of UI from the beginning. So in a lot of ways, you're already halfway there to basic screen-reading support. The bigger question is one of UX design (ie. does the Editor require the mouse?), rather than implementation.

FRP: Modern FRP-based approaches work fantastically for the web. I think they're really strong when you have a flat UI that is structured as a hierarchy. Game UI can be much more complex. In-Game UI often isn't a flat plane, nor a tree. There might be a myriad of text cards, health-bars, holographic displays, etc. Depending on the game, it may be hard to treat this as a single tree.

Additionally, there are entire projects currently figuring out FRP in Rust (Yew, etc). It's been a massive undertaking, and most require lots of macros, templates, generics, etc. And that's without having to build a renderer. So I worry about complexity, scope here.

What I'd favor is a general, high performance UI rendering engine, integrated with the ECS. An FRP crate could be built on top of it, but wouldn't be explicitly baked into the solution. That would allow UI-heavy and 2D games to use FRP as needed, but not require jamming all other use cases inside of it.

I think Godot can serve as a nice project to look at, while developing the BevyUI

Just wanna highlight again that editor-ready ui crate is already exists - it is fyrox-ui (link). Not everything should be ecs based, a solution is good if does its job in reasonable time. Building your own ui library is cool in terms of satisfying programmers curiosity, but counterproductive in terms of development time. Objections?

This is a dump of my personal "UI is usable" checklist:

Essential

Border rendering support (Ui Node Borders #7795)
Built-in scrolling support (WIP: Support scrolling (Overflow::Scroll) in bevy_ui #8104)
Better text rendering support (Migrate to cosmic-text #8808)
Better event handling (https://github.com/aevyrie/bevy_eventlistener)
A declarative widget system that handles dynamically re-rendering and state management
Built in widgets:
- Button (this can possibly just be good click event support)
- Text Input (https://github.com/StaffEngineer/velo/tree/main/crates/bevy_cosmic_edit)
- Drop-down select

Nice to have

Rounded corners (UI Node Border Radius and Shadows #8973)

@mrDIMAS

Just wanna highlight again that editor-ready ui crate is already exists - it is fyrox-ui (link). Not everything should be ecs based, a solution is good if does its job in reasonable time.

@mrDIMAS, do you have any examples of using it with Bevy engine? I'm using bevy-egui currently for my projects, but don't mind exploring other options.

FYI, I don't think that firox-ui has any keyboard navigation.

Related to this UI project thingy and regarding these points from the top post:

Define a consistent way to implement widgets

A core set of widgets: buttons, inputs, resizable panels, etc

I'd like to just drop some of my experience using Bevy UI so far and how I've built a little widget system on top of it. I'm curious whether anyone else has done something similar (surely someone has?) and whether this is an approach that could be built into Bevy (or maybe used in the editor or something). Or maybe this is not desirable for some reason? I haven't explored this approach a lot so perhaps there's some disadvantages I'm not seeing yet.

I think that a big issue with Bevy UI at the moment is that it's very verbose to define UI and it's cumbersome to compartmentalize. It's also not intuitive how a "widget" (or element or whatever you want to call it) can have control over where to place its internals (i.e. its children UI elements).

After experimenting a bit, I ended up writing these couple of traits:

/// Types that can spawn new entities.
pub(crate) trait Spawner<'w, 's> {
    /// Spawns a new entity with the given bundle of components.
    ///
    /// Returns an [`EntityCommands`] to enable further commands on the entity.
    fn spawn<B: Bundle>(&mut self, bundle: B) -> EntityCommands<'w, 's, '_>;
}

impl<'w, 's> Spawner<'w, 's> for Commands<'w, 's> {
    fn spawn<B: Bundle>(&mut self, bundle: B) -> EntityCommands<'w, 's, '_> {
        self.spawn(bundle)
    }
}

impl<'w, 's> Spawner<'w, 's> for ChildBuilder<'w, 's, '_> {
    fn spawn<B: Bundle>(&mut self, bundle: B) -> EntityCommands<'w, 's, '_> {
        self.spawn(bundle)
    }
}

/// A trait for widgets that lets them be spawned neatly.
pub(crate) trait Widget<'w, 's> {
    /// Spawns the widget with the given spawner.
    fn spawn<S: Spawner<'w, 's>>(self, spawner: &mut S) -> EntityCommands<'w, 's, '_>;
}

The Spawner trait is just unifying the spawn methods of Commands and ChildBuilder. I'm honestly confused why this isn't built into Bevy already but I can define it myself pretty easily anyway.

The Widget trait then allows any implementor to be spawned under a given Spawner. For example, you could define a simple menu like this (some used functions returning NodeBundle's left out):

/// The menu widget consists of a container with a title.
pub(crate) struct MenuWidget<F: FnOnce(&mut ChildBuilder)> {
    /// The title is displayed at the top of the menu.
    title: &'static str,
    /// The content of the menu is filled by this closure.
    content: F,
}

impl<F: FnOnce(&mut ChildBuilder)> MenuWidget<F> {
    /// Creates a new menu with the given title and content.
    pub(crate) fn new(title: &'static str, content: F) -> Self {
        MenuWidget { title, content }
    }
}

impl<'w, 's, F> Widget<'w, 's> for MenuWidget<F>
where
    F: FnOnce(&mut ChildBuilder),
{
    fn spawn<S: Spawner<'w, 's>>(self, spawner: &mut S) -> EntityCommands<'w, 's, '_> {
        let mut root = spawner.spawn(menu_body());
        root.with_children(|root| {
            root.spawn(menu_background()).with_children(|background| {
                background.spawn(
                    TextBundle::from_section(
                        self.title,
                        TextStyle {
                            font_size: 72.0,
                            color: Color::WHITE,
                            ..default()
                        },
                    )
                    .with_style(Style {
                        margin: UiRect::all(Val::Px(48.0)),
                        ..default()
                    }),
                );

                (self.content)(background)
            });
        });
        root
    }
}

Note how this allows the MenuWidget to decide the location of its inner content (it's nested into the menu's background, not just as a direct child of the menu). The definition of the MenuWidget is still a little verbose, but now it's packed away in a neat self-contained package. By then also defining a ButtonWidget and a CheckboxWidget, I can define UI that uses all the widgets quite neatly. For instance, a very simple settings menu could look like this:

fn setup_settings_menu(mut cmds: Commands, settings: Res<Settings>) {
    info!("Setting up settings menu...");

    MenuWidget::new("Settings", |menu| {
        CheckboxWidget::new("Enable vsync:", settings.vsync, Action::ToggleVsync).spawn(menu);

        CheckboxWidget::new(
            "Enable development mode:",
            settings.dev_mode == DevMode::On,
            Action::ToggleDevMode,
        )
        .spawn(menu);

        ButtonWidget::new("Back", |world| {
            world
                .resource_mut::<NextState<AppState>>()
                .set(AppState::Menu(MenuState::Main));
        })
        .spawn(menu);
    })
    .spawn(&mut cmds)
    // Insert a marker component to easily despawn the menu later.
    .insert(SettingsMenu);
}

I find the above quite neat as each widget is self-contained. I only have to give the widget it's required inputs and it'll take care of the rest.

In the above code, buttons take a label and an action (a fn(&mut World)) that they can use to do something once pressed. Checkboxes take a label, an initial state (bool) and an Action enum that is a list of user input actions. But the specific inputs aren't important and are mostly implementation details of how I've built these widgets for myself right now.

The point is that each widget is in control of what inputs it gets and what it uses those inputs for in order to construct the UI. It definitely still requires some additional systems to make these widgets do things, but I thought I'd still share this approach in case it is interesting to anyone.

Closing as unhelpful at this point. I still agree with its conclusions, but this isn't helping us organize work at this stage.

Navigation Menu

Repository navigation

Editor-Ready UI #254

Active Crates / Repos

Sub Issues

Documents

120 remaining items

Essential

Nice to have

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Editor-Ready UI #254

Description

Active Crates / Repos

Sub Issues

Documents

Activity

Kleptine commented on Aug 20, 2020

Kleptine commented on Aug 20, 2020

thefuntastic commented on Aug 21, 2020

tektrip-biggles commented on Aug 21, 2020

erlend-sh commented on Aug 22, 2020

stefee commented on Aug 22, 2020

cart commented on Aug 22, 2020

stefee commented on Aug 22, 2020

stefee commented on Aug 22, 2020

tektrip-biggles commented on Aug 22, 2020

regexident commented on Aug 23, 2020

Kleptine commented on Aug 23, 2020

ShalokShalom commented on Aug 24, 2020

120 remaining items

mrDIMAS commented on Dec 1, 2022

nicoburns commented on Jun 2, 2023

Essential

Nice to have

rlidwka commented on Jun 5, 2023

Barugon commented on Jun 5, 2023

SorteKanin commented on Aug 20, 2023

alice-i-cecile commented on Jun 7, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions