Rendering#

Programming#

The Filament renderer is organized in three layers. At the base is the externally developed Filament library itself, which provides the physically-based rendering pipeline and graphics API backends. The public interface to the renderer is the mjrf C API, declared in mjrfilament.h. Unlike the classic mjr API, which redraws an mjvScene every frame, mjrf is retained-mode and asynchronous: applications create long-lived objects, mutate them incrementally, and submit batches of render and pixel-readback requests. On top of the C API, the C++ support layer in src/render/filament/support (also bound in Python under mujoco.mjrf) automates creating and updating mjrf scene objects from mjModel, mjData, and mjvOption.

Below is a simplified C++ outline (adapted from RenderFilament() in render.cc) illustrating how these components fit together for offscreen rendering:

// create Filament context and scene
mjrfContextConfig context_cfg;
mjrf_defaultContextConfig(&context_cfg);
auto ctx = mujoco::CreateContext(context_cfg);

mjrfSceneParams scene_params;
mjrf_defaultSceneParams(&scene_params);
auto scene = mujoco::CreateScene(ctx.get(), scene_params);
mjrf_configureSceneFromModel(scene.get(), m);

// create retained model components and decoration helper
auto model_resources   = std::make_unique<ModelResources>(m, ctx.get());
auto model_lights      = std::make_unique<ModelLights>(scene.get(), model_resources.get());
auto model_renderables = std::make_unique<ModelRenderables>(scene.get(), model_resources.get());
auto model_decorations = std::make_unique<ModelDecorations>(ctx.get(), scene.get(), m);

// create offscreen render target
mjrfRenderTargetConfig target_cfg;
mjrf_defaultRenderTargetConfig(&target_cfg);
target_cfg.width        = width;
target_cfg.height       = height;
target_cfg.color_format = mjPIXEL_FORMAT_RGB8;
target_cfg.depth_format = mjPIXEL_FORMAT_DEPTH32F;
auto render_target      = mujoco::CreateRenderTarget(ctx.get(), target_cfg);

// update retained lights, renderables, and transient decorations from mjData
mjrRect viewport = {0, 0, width, height};
model_renderables->SetOptions(opt);
model_lights->Update(d);
model_renderables->Update(d);
model_decorations->Update(d, &opt, nullptr, &cam, viewport);

// submit render and pixel-readback requests and wait for completion
mjrfRenderRequest request;
mjrf_defaultRenderRequest(&request);
request.scene               = scene.get();
request.camera              = mjv_camera2GLCamera(m, d, &cam);
request.viewport            = viewport;
request.target              = render_target.get();
request.headlight    = m->vis.headlight.active;
request.headlight_intensity = model_lights->GetHeadlightIntensity();

mjrfReadRequest read_request;
mjrf_defaultReadRequest(&read_request);
read_request.target    = render_target.get();
read_request.output    = rgb.data();
read_request.num_bytes = width * height * 3;

mjrfFrameHandle frame = mjrf_render(ctx.get(), &request, 1, &read_request, 1);
mjrf_waitForFrame(ctx.get(), frame);

// remove transient decorations before destroying the scene
model_decorations->Clear();

Authoring scenes#

The classic renderer implements a traditional Phong lighting model: surface color is a sum of ambient, diffuse, specular and emissive terms, and light intensities are dimensionless values tuned by eye. Modern physically-based rendering (PBR) engines describe surfaces with empirically measurable properties and lights with photometric units. Because material and lighting parameters are physical, assets retain their appearance across scenes and lighting conditions, and effects like image-based lighting and realistic metals become possible. For an introduction to the underlying theory, see Filament’s PBR documentation.

The Filament renderer supports both models. In general, we recommend that scenes are authored with modern PBR (e.g. metallic-roughness) methods to produce the best results. However, we assume that many existing models are authored for the classic renderer and so we default to using Filament’s PBR-based approximation of Phong shading. This fallback mode has been tuned to produce reasonable results, but the final image can vary significantly compared to the classic renderer. The two decisions, how to shade materials and how to light the scene, are made independently and are described below.

Rendering options#

The Filament renderer provides a set of options for controlling how the final image is rendered. While some of these options can be set directly on the scene elements as described above (e.g. setting the PBR properties of the material or the intensities of the lights), some features are controlled by custom model elements (e.g. numeric or text) whose names begin with filament. As the feature set stabilizes, important flags will be promoted to first-class MJCF attributes.

<custom>
  <numeric name="filament.clear_color" data="0.5 0.6 0.7 1"/>
</custom>

Most flags correspond directly to Filament’s rendering options and are read by mjrf_configureSceneFromModel. Boolean flags are numerics taking values 0 or 1. Enum flags (e.g. filament.ao.quality) take string values that correspond to the underlying enum’s name identifier. Flags not specified in the model use the defaults listed below. All flag names begin with filament.

Using OpenGL#

MuJoCo uses OpenGL 1.5 in the compatibility profile with the ARB_framebuffer_object and ARB_vertex_buffer_object extensions. OpenGL symbols are loaded via GLAD the first time the mjr_makeContext function is called. This means that the MuJoCo library itself does not have an explicit dependency on OpenGL and can be used on systems without OpenGL support, as long as mjr_ functions are not called.

Applications that use MuJoCo’s built-in rendering functionalities are responsible for linking against an appropriate OpenGL context creation library and for ensuring that there is an OpenGL context that is made current on the running thread. On Windows and macOS, there is a canonical OpenGL library provided by the operating system. On Linux, MuJoCo currently supports GLX for rendering to an X11 window, OSMesa for headless software rendering, and EGL for hardware accelerated headless rendering.

Context and GPU resources#

The first step in the rendering process is to create the model-specific GPU context mjrContext. This is done by first clearing the data structure with the function mjr_defaultContext, and then calling the function mjr_makeContext. This was already illustrated earlier; the relevant code is:

mjModel* m;
mjrContext con;

// clear mjrContext only once before first use
mjr_defaultContext(&con);

// create window with OpenGL context, make it current
GLFWwindow* window = glfwCreateWindow(1200, 900, "Demo", NULL, NULL);
glfwMakeContextCurrent(window);

// ... load MuJoCo model

// make model-specific mjrContext
mjr_makeContext(m, &con, mjFONTSCALE_100);

// ... load another MuJoCo model

// make mjrContext for new model (old context freed automatically)
mjr_makeContext(m, &con, mjFONTSCALE_100);

// free context when done
mjr_freeContext(&con);

How is mjrContext related to an OpenGL context? An OpenGL context is what enables the application to talk to the video driver and send rendering commands. It must exist and must be current in the calling thread before mjr_makeContext is called. GLFW and related libraries provide the necessary functions as shown above.

mjrContext is specific to MuJoCo. After creation, it contains references (called “names” in OpenGL) to all the resources that were uploaded to the GPU by mjr_makeContext. These include model-specific resources such as meshes and textures, as well as generic resources such as font bitmaps for the specified font scale, framebuffer objects for shadow mapping and offscreen rendering, and associated renderbuffers. It also contains OpenGL-related options copied from mjModel.vis, capabilities of the default window framebuffer that are discovered automatically, and the currently active buffer for rendering; see buffers below. Note that even though MuJoCo uses fixed-function OpenGL, it avoids immediate mode rendering and instead uploads all resources to the GPU upfront. This makes it as efficient as a modern shader, and possibly more efficient, because fixed-function OpenGL is now implemented via internal shaders that have been written by the video driver developers and tuned extensively.

Most of the fields of mjrContext remain constant after the call to mjr_makeContext. The only exception is mjrContext.currentBuffer which changes whenever the active buffer changes. Some of the GPU resources may also change because the user can upload modified resources with the functions mjr_uploadTexture, mjr_uploadMesh, mjr_uploadHField. This can be used to achieve dynamic effects such as inserting a video feed into the rendering, or modulating a terrain map. Such modifications affect the resources residing on the GPU, but their OpenGL names are reused; thus, the change is not actually visible in mjrContext.

The user should never make changes to mjrContext directly. MuJoCo’s renderer assumes that only it can manage mjrContext. In fact this kind of object would normally be opaque and its internal structure would not be exposed to the user. We are exposing it because MuJoCo has an open design and also because users may want to interleave their own OpenGL code with MuJoCo’s renderer, in which case they may need read access to some fields of mjrContext. For example in VR applications the user needs to blit from MuJoCo’s offscreen buffer to a texture provided by a VR SDK.

When a different MuJoCo model is loaded, mjr_makeContext must be called again. There is also the function mjr_freeContext which frees the GPU resources while preserving the initialization and capabilities flags. This function should be called when the application is about to exit. It is called automatically from within mjr_makeContext, so you do not need to call it directly when a different model is loaded, although it is not an error to do so. The function mjr_defaultContext must be called once before rendering starts, to clear the memory allocated for the data structure mjrContext. If you call it after calling mjr_makeContext, it will wipe out any record that GPU resources were allocated without freeing those resources, so don’t do that.

Buffers for rendering#

In addition to the default window framebuffer, OpenGL can support unlimited framebuffer objects (FBOs) for custom rendering. In MuJoCo we provide systematic support for two framebuffers: the default window framebuffer, and one offscreen framebuffer. They are referred to by the constants in the enum type mjtFramebuffer, namely mjFB_WINDOW and mjFB_OFFSCREEN. At any time, one of these two buffers is active for the purposes of MuJoCo rendering, meaning that all subsequent commands are directed to it. There are two additional framebuffer objects referenced in mjrContext, needed for shadow mapping and resolving multi-sample buffers, but these are used internally and the user should not attempt to access them directly.

The active buffer is set with the function mjr_setBuffer. This sets the value of mjrContext.activeBuffer and configures the OpenGL state accordingly. When mjr_makeContext is called, internally it calls mjr_setBuffer with argument mjFB_WINDOW, so that rendering starts in the window buffer by default. If the specified buffer does not exist, mjr_setBuffer automatically defaults to the other buffer (note that when using headless rendering on Linux, there may be no window framebuffer).

From the perspective of OpenGL, there are important differences between the window framebuffer and offscreen framebuffer, and these differences affect how the MuJoCo user interacts with the renderer. The window framebuffer is created and managed by the operating system and not by OpenGL. As a result, properties such as resolution, double-buffering, quad-buffered stereo, multi-samples, v-sync are set outside OpenGL; this is done by GLFW calls in our code samples. All OpenGL can do is detect these properties; we do this in mjr_makeContext and record the results in the various window capabilities fields of mjrContext. This is why such properties are not part of the MuJoCo model; they are session/software-specific and not model-specific. In contrast, the offscreen framebuffer is managed entirely by OpenGL, and so we can create that buffer with whatever properties we want, namely with the resolution and multi-sample properties specified in mjModel.vis.

The user can directly access the pixels in the two buffers. This is done with the functions mjr_readPixels, mjr_drawPixels and mjr_blitBuffer. Read/draw transfer pixels from/to the active buffer to/from the CPU. Blit transfers pixels between the two buffers on the GPU and is therefore much faster. The direction is from the active buffer to the buffer that is not active. Note that mjr_blitBuffer has source and destination viewports that can have different size, allowing the image to be scaled in the process.

Drawing pixels#

The main rendering function is mjr_render. Its arguments are a rectangular viewport for rendering, the mjvScene which was populated by the abstract visualizer, and the mjrContext which was created by mjr_makeContext. The viewport can be the entire active buffer, or part of it for custom effects. A viewport corresponding to the entire buffer can be obtained with the function mjr_maxViewport. Note that while the offscreen buffer size does not change, the window buffer size changes whenever the user resizes or maximizes the window. Therefore user code should not assume fixed viewport size. In the code sample simulate.cc we use a callback which is triggered whenever the window size changes, while in basic.cc we simply check the window size every time we render. On certain scaled displays (notably on MacOS) the window size and framebuffer size can be different. So if you are getting the size with GLFW functions, use glfwGetFramebufferSize rather than glfwGetWindowSize. On the other hand, mouse coordinates are returned by the operating system in window rather than framebuffer units; thus the mouse interaction functions should use glfwGetWindowSize to obtain the window height needed to normalize the mouse displacement data.

mjr_render renders all mjvGeoms from the list mjvScene.geom. The abstract visualization options mjvOption are no longer relevant here; they are used by mjv_updateScene to determine which geoms to add, and as far as mjr_render is concerned these options are already baked-in. There is however another set of rendering options that are embedded in mjvScene, and these affect the OpenGL rendering process. The array mjvScene.flags contains flags indexed by the enum type mjtRndFlag and include options for enabling and disabling wireframe mode, shadows, reflections, skyboxes and fog. Shadows and reflections involve additional rendering passes. MuJoCo’s renderer is very efficient, but depending on the model complexity and available GPU, it may be necessary to disable one or both of these effects in some cases.

The parameter mjvScene.stereo determines the stereo mode. The possible values are given by the enum type mjtStereo and are as follows:

mjSTEREO_NONE

Stereo rendering is disabled. The average of the two OpenGL cameras in mjvScene is used. Note that the renderer always expects both cameras to be properly defined, even if stereo is not used.

mjSTEREO_QUADBUFFERED

This mode works only when the active buffer is the window, and the window supports quad-buffered OpenGL. This requires a professional video card. The code sample simulate.cc attempts to open such a window. In this mode MuJoCo’s renderer uses the GL_BACK_LEFT and GL_BACK_RIGHT buffers to render the two views (as determined by the two OpenGL cameras in mjvScene) when the window is double-buffered, and GL_FRONT_LEFT and GL_FRONT_RIGHT otherwise. If the window does not support quad-buffered OpenGL or the active buffer is the offscreen buffer, the renderer reverts to the side-by-side mode described next.

mjSTEREO_SIDEBYSIDE

This stereo mode does not require special hardware and is always available. The viewport given to mjr_render is split in two equal rectangles side-by-side. The left view is rendered on the left side and the right view on the right side. In principle users can cross their eyes and see stereo on a regular monitor, but the goal here is to show it in a stereoscopic device. Most head-mounted displays support this stereo mode.

In addition to the main mjr_render function, we provide several functions for “decorating” the image. These are 2D rendering functions and include mjr_overlay, mjr_text, mjr_rectangle, and mjr_figure. The user can draw additional decorations with their own OpenGL code. This should be done after mjr_render, because mjr_render clears the viewport.

We also provide the functions mjr_finish and mjr_getError for explicit synchronization with the GPU and for OpenGL error checking. They simply call glFinish and glGetError internally. This together with the basic 2d drawing functions above is meant to provide enough functionality so that most users will not need to write OpenGL code. Of course we cannot achieve this in all cases, short of providing wrappers for all of OpenGL.