High-Performance Native-First Media Engine
Zero-Copy Support (GPU/CPU Handles) β’ Hardware Offloading β’ Sans-IO Cores β’ Permissive (MIT / Apache-2.0)
Status: early development (
0.x), building toward 1.0 with the community. Not recommended for production yet. Pre-1.0: public APIs, crate layout, and backends may change often (no stability guarantee) β contributions and real-hardware testing are what move each backend closer to 1.0. Details: docs/spec/status.md.
Rust media stack: high-level pipelines built from first-class low-level APIs β OS/GPU codec sessions, GpuBufferHandle, and sans-io mux/demux/bitstream cores. Prefer Zero-Copy paths (GPU handles or shared CPU buffers); name CPU readback, cross-API copies, and SW fallbacks when they exist.
Design: docs/spec/vision.md.
Covers device capture (camera, mic, screen, window, desktop audio), encode/decode, containers, and FFI / WASM bindings across Windows, Web, Linux, other. Layout: sans-io cores, facade crates with OS backends as #[cfg]-gated modules (e.g. mediaway-device contains windows/linux/web). C ABI: a single mediaway-ffi facade β docs/spec/c-ffi.md Β· docs/spec/crate-packaging.md.
The Windows H.264 path has an enabled end-to-end verification slice:
- Rust API: H.264 encode β fragmented MP4 mux/demux β CPU decode, including a
trim/splice/re-encode round trip (
crates/mediaway/tests/trim_and_splice_windows.rs). - C ABI: the same encode β MP4 mux/demux β decode flow through
mediaway-ffi, including decoded pixel-content assertions (crates/mediaway-ffi/tests/decode_smoke.rs).
These tests skip only when the host has no usable Windows Media Foundation backend. The slice is verified; broader production readiness, API stability, and cross-platform hardware coverage remain in progress.
Repository: github.com/nyxways/mediaway
Questions / support: please open a GitHub Issue (English). Do not rely on private email for general inquiries.
Mediaway resolves the fundamental trade-offs and fragmentation in modern multimedia engineering:
- OS & Hardware Offloading (Small Binaries & Peak Efficiency): Rather than bundling heavy C/Rust software codecs that inflate binary sizes to tens of megabytes and consume high CPU power, Mediaway delegates heavy encoding/decoding workloads to native OS APIs (WMF, VideoToolbox, VA-API, WebCodecs) and dedicated GPU hardware engines (NVENC, QuickSync, AMF). This keeps binary footprints minimal while enabling low-power 4K/8K real-time execution.
- Zero-Copy Engine & Framework Interop (Seamless DX): Designed for zero-copy integration with graphics frameworks and game engines (
wgpu,Three.js/ WebGPU,Unity,Godot). Passes native GPU surface handles (GpuBufferHandle/ D3D11, D3D12, Vulkan, Metal pointers) directly across capture, processing, encoding, and rendering without CPU readback stalls. - Permissive & GPL-Free (MIT OR Apache-2.0): Shipped under MIT OR Apache-2.0 with zero LGPL/GPL dependencies (no linking to FFmpeg/
libav*), eliminating licensing risks for commercial and proprietary applications. - Unified & Composable Stack: Integrates device capture (camera, mic, screen, window), hardware video/audio/image codecs, and container muxing/demuxing into a single cohesive pipeline.
- Uncompromised Freedom + High-Level Ergonomics: Provides clean end-to-end abstractions (
mediaway) while keeping raw GPU handles, sans-io state machines, and bitstream timebase controls 100% public and unblocked.
All examples are in
examples/, one capability per file, grouped by sector βcontainer/(mux/demux only),encode//decode/(one codec direction, no container),device/(one capture source, no encode),pipeline/(composed end-to-end flows). Run any of them with:cargo run --example <name>
Register tracks, push packets, collect bytes, demux β no OS codec, no unsafe.
The muxer is typed via a typestate (Open β Live); the demuxer is
fully streaming (feed bytes, poll packets).
use mediaway_common::{Bytes, CodecKind, Packet, Rational, StreamInfo, VideoGeometry};
use mediaway_container::mp4;
let mut muxer = mp4::Muxer::new();
let v_track = muxer.add_track(StreamInfo::Video {
id: 0, codec: CodecKind::H264,
time_base: Rational::new(1, 30),
geometry: VideoGeometry { width: 1920, height: 1080 },
extra_data: Bytes::new(),
})?;
let mut muxer = muxer.begin(); // Open -> Live
muxer.push_packet(&Packet {
stream_id: v_track, pts: 0, dts: 0, duration: 1,
is_keyframe: true, is_discard: false,
payload: Bytes::new(),
})?;
muxer.flush();
let mut mp4_bytes: Vec<u8> = Vec::new();
muxer.poll_bytes(&mut mp4_bytes);
let mut demux = mp4::Demuxer::new();
demux.push_bytes(&mp4_bytes);
while let Some(pkt) = demux.poll_packet() { /* β¦ */ }Full runnable example: examples/container/mux_demux_mp4.rs
mediaway composes the low-level VideoEncoder trait + mp4::Muxer for
you: platform::AutoEncoder::open selects the best available path (Zero-Copy GPU β
CPU upload β β¦), and EncodeSession handles the poll-loop β mux wiring so callers
just push frames.
use mediaway_encoder::auto::AutoVideoEncodeConfig;
use mediaway::{platform, EncodeSession};
let config = AutoVideoEncodeConfig {
bitrate_bps: 8_000_000,
// gpu_device: Some(GpuDeviceHandle::DirectX11(handle)), // for Zero-Copy
..AutoVideoEncodeConfig::new(CodecKind::H264, 1920, 1080, Rational::new(1, 30))
// backend defaults to Auto, max_path_class to CpuUpload (ZC -> CPU upload, no readback/SW)
};
let encoder = platform::AutoEncoder::open(&config)?;
let mut session = EncodeSession::open(encoder)?;
session.write_frame(&nv12_frame)?;
let mp4_bytes = session.finish()?; // flush + mux flush + poll_bytesThe low-level path (VideoEncoder trait + manual poll_packet/mux loop) stays fully
public and usable without this crate β see examples/encode/encode_h264.rs
(encoder in isolation) and examples/container/mux_demux_mp4.rs.
Full runnable example: examples/pipeline/encode_to_mp4.rs
platform::ScreenCapture / platform::Microphone / platform::AutoEncoder are typed
against facade traits (&mut dyn VideoCapture / &mut dyn AudioCapture / &mut dyn VideoEncoder); platform-specific construction is the only OS-specific code, and it
lives entirely in mediaway::platform.
use mediaway_common::Rational;
use mediaway_device::desktop::DesktopVideoCaptureConfig;
use mediaway_device::{Select, AudioCaptureConfig};
use mediaway::{platform, EncodeSession};
let tb = Rational::new(1, 30);
let mut cap = platform::ScreenCapture::open(&DesktopVideoCaptureConfig::screen(Select::Default, tb))?;
let mut mic = platform::Microphone::open(&AudioCaptureConfig::microphone(Rational::new(1, 48_000)))?;
let encoder = platform::AutoEncoder::open(&enc_config)?;
let mut session = EncodeSession::open(encoder)?;
loop {
if let Some(frame) = cap.poll_frame()? {
session.write_frame(&frame)?; // handles poll_packet -> mux internally
cap.release_frame()?;
}
// β¦ encode `mic.poll_frame()` PCM to AAC and mux as a second track β
// full version below composes that against a shared `mp4::Muxer` directly,
// since `EncodeSession` stays video-only/single-track by design (ADR-0014).
}Full runnable example, with real second-track AAC audio muxed alongside video:
examples/pipeline/screen_record.rs. For each
capture source on its own (no encode), see examples/device/.
A real non-linear edit built from the low-level VideoDecoder/VideoEncoder traits +
mediaway-container mux/demux β no new DecodeSession/EditTimeline abstraction. Encodes
two synthetic clips, decodes each back, drops the first/last frame of each (trim),
concatenates what's left with renumbered contiguous timestamps (splice), and re-encodes.
use mediaway::platform;
let encoder = platform::AutoEncoder::open(&enc_config)?;
let decoder = platform::AutoDecoder::open(&dec_config)?; // dispatches per-OS under the hood
// β¦ encode two clips, mux, demux, decode, trim by index/PTS, chain + renumber, re-encodeFull runnable example: examples/pipeline/trim_and_splice.rs.
For decode on its own, see examples/decode/decode_h264.rs. Detail:
docs/ai/wiki/pipeline/trim-and-splice.md.
Rust is the primary, always-first-class native API. Bindings for the supported
host languages are real, verified code under bindings/ β not
aspirational sketches. Each language links only the capabilities it needs
(container, encoder, device, β¦), so app binaries don't pull in unused codecs.
| Language / Host | Interop Path | Status | Target Ergonomics & Use Cases | Examples |
|---|---|---|---|---|
| Rust | Native crates | β primary | Primary API (100% Zero-Copy, Sans-IO, traits) | examples/ |
| C | Direct C ABI | β verified | ABI contract baseline, zero-overhead FFI | bindings/c/examples/ |
| C++ | Thin RAII C ABI | β verified | Native desktop apps, custom engines, rendering pipelines | bindings/cpp/examples/ |
| C# (.NET) | P/Invoke | β verified | Windows desktop, WPF/WinUI, Unity native plugins | bindings/csharp/ |
| Python | ctypes / cffi |
β verified | Data processing pipelines, ML model input/output streams | bindings/python/examples/ |
| Node.js (TS) | Native Addon / N-API (koffi FFI today) | β verified | Node.js server-side video processing and CLI tools | bindings/nodejs/examples/ |
| Browser (TS) | WASM + WebCodecs | β verified | Browser-native high-performance video apps (no C FFI) | bindings/browser/ |
| Unity / Godot | Native GPU Plugin | Planned | Seamless Zero-Copy GPU texture sharing & camera/screen record | β |
Status legend: β
verified = real binding source built and run against the native
libraries; π design = README brief + example code only (nothing compiles/ships
yet). Per-language detail: bindings/README.md and each
folder's own README.
| Mark | Meaning |
|---|---|
| β | Hardware / real-session-verified β real hardware or a real live session was actually exercised (not just "tests for the claimed scope" β implementation + unit tests alone are π, not β ) |
| β‘ | Zero-Copy path β no payload memcpy (GPU handle or shared CPU buffer); implies β
β still needs real hardware verification, not just that the Zero-Copy code path exists |
| π | Implemented, compiles β not (yet) verified against real hardware / a real session |
| π οΈ | Planned |
| β | Attempted and genuinely blocked β no upstream API to build on, a hard version/license conflict, or a real query returned "unsupported." Not a "ran out of time" π οΈ. |
| π» | Not exercisable yet β license/patent blocked, no target hardware, out of scope, or no device/daemon/session available to run otherwise-tested code against |
Cell = encode/decode. One mark means both; A/B means encode / decode.
OS codec APIs (WMF, WebCodecs, VA-API, β¦) fed with CPU buffers.
| Codec | Windows | Web | Linux | Apple | Android |
|---|---|---|---|---|---|
| H.264 / AVC | β / β | β / β | π / π | π / π | π / π |
| HEVC / H.265 | β / β | β / β | π / π | π / π | π» |
| AV1 | π οΈ / β | β | β / π | β / π | π» |
| VP9 | β / β | β | π / π | β / π | π» |
| ProRes | π» | π» | π» | π / π | π» |
| AAC | π / β | π | π οΈ | π / π | π» |
| Opus | β / β | π | π οΈ | π / π | π» |
Same OS APIs with GPU surfaces (GpuBufferHandle, DXGI, β¦). Video only.
| Codec | Windows | Web | Linux | Apple | Android |
|---|---|---|---|---|---|
| H.264 / AVC | π / π | β | π / π | π / π | π» |
| HEVC / H.265 | π / π | π / π οΈ | π οΈ | π / π | π» |
| AV1 | π / π | π / π οΈ | π οΈ | β / π | π» |
| VP9 | π / π | π / π οΈ | π οΈ | β / π | π» |
| ProRes | π» | π» | π» | π / π | π» |
Graphics interop (D3D11, Vulkan, Metal, β¦) β which API your textures use. Video only.
Adapters: mediaway wgpu module.
| Codec | D3D11 | D3D12 | Vulkan | Metal |
|---|---|---|---|---|
| H.264 / AVC | π / π | β / π οΈ | β / β | π / π |
| HEVC / H.265 | π / π | β / π | β / β | π / π |
| AV1 | π / π | β / π | β / β | β / π |
| VP9 | π / π | π» | π» | β / π |
| ProRes | π» | π» | π» | π / π |
Vendor SDKs (NVENC, AMF, QuickSync, β¦) β separate from OS + graphics interop. Video only.
See mediaway-encoder for NVIDIA/AMD/Intel backends.
| Codec | NVIDIA | AMD | Intel | Qualcomm |
|---|---|---|---|---|
| H.264 / AVC | β | π | β | π» |
| HEVC / H.265 | β | π | β | π» |
| AV1 | β | π | β | π» |
| VP9 | π» | π» | π» | π» |
Pure Rust sans-io software codecs β no C codec FFI (OpenH264, libvpx, β¦). Opt-in only; never a silent HW fallback. Detail: mediaway-sw (H.264 decode, AV1/Opus encode, PCM, audio processing).
| Codec | Status |
|---|---|
| H.264 / AVC | π |
| HEVC / H.265 | π» |
| AV1 | π |
| VP9 | π» |
| AAC | π» |
| Opus | β |
| PCM / raw | π |
Freestanding mux/demux cores plus the mediaway-container facade (wraps all eight as Mediaway-typed StreamInfo/Packet modules: mp4, webm, wav, adts, mp3, ogg, flv, ts). Same marks as Β§ Codec support β not every π-mux implements the shared Mux trait; see mediaway-container for which.
| Format | Mux | Demux |
|---|---|---|
| MP4 / fMP4 | β | β |
| WebM | β | β |
| WAV / RIFF (PCM) | π | π |
| ADTS (raw AAC) | β | β |
| MP3 (MPEG Layer III) | π | β |
| Ogg | π | π |
| FLV | β | β |
| MPEG-TS | π | β |
What mediaway-device backends target (camera, mic, screen, window). Same marks as Β§ Codec support. β‘ = Zero-Copy out (GPU GpuBufferHandle or shared CPU PCM without payload copy); cell = CPU capture / GPU surface where both apply (π / β‘).
| Source | Windows | Web | Linux | Apple | Android |
|---|---|---|---|---|---|
| Camera (video) | β | π | π | π | π |
| Microphone | β | π | π | π | π |
| Screen / display | β | π | π | π | π |
| Window | β‘ | π | π | π | β |
| Desktop audio (system loopback / per-process) | β | β | β | β | β |
Window β‘ holds for an uncropped window, or a capture region at the origin. A region anywhere else costs one GPU copy per frame (GpuCopy, not β‘), and only WGC accepts a region at all β the other backends refuse it with Unsupported. Desktop audio is WASAPI-only today; per-process loopback needs Windows 10 2004+.
| Crate | Role |
|---|---|
mediaway-common |
Shared types (Rational, formats, GpuBufferHandle, packets/frames) |
iso-bmff |
MP4 / ISOBMFF mux + demux core (fMP4, ClearKey CENC) |
iso-cenc |
ClearKey CENC sample crypto (AES-128-CTR) |
ebml-webm |
EBML / WebM mux + demux core |
riff-wave-core |
WAV / RIFF PCM mux + demux core |
adts-core |
ADTS (raw AAC) mux + demux core |
mpeg-audio |
MP3 (MPEG Layer III) mux + demux core |
ogg-core |
Ogg page/packet mux + demux core |
flv-core |
FLV tag mux + demux core |
mpeg-ts-core |
MPEG-2 Transport Stream mux + demux core |
rtp-core |
RTP payloadization for H.264/HEVC (RFC 3550/6184/7798) |
rtmp |
RTMP publish-client handshake + chunk stream + AMF0 command mux |
mediaway-container |
Container facade: shared traits + typed mp4/webm/wav/adts/mp3/ogg/flv/ts, plus a replay ring of encoded packets |
mediaway-encoder |
Encode traits + auto selection; Windows WMF / NVENC / QuickSync / Vulkan / WebCodecs / VA-API backends |
mediaway-decoder |
Decode traits; Windows WMF (HW, DX11 Zero-Copy) / Vulkan / WebCodecs backends |
mediaway-device |
Capture + playback traits; Windows DXGI/WGC/WASAPI, Linux portal+PipeWire+V4L2, Web getUserMedia/getDisplayMedia |
mediaway |
Convenience pipeline (EncodeSession + platform auto-dispatch + wgpu interop) |
mediaway-sw |
Pure Rust sans-io software codecs (H.264 decode, AV1/Opus encode, PCM, audio processing) |
vpl-sys |
oneVPL FFI bindings (runtime-loaded; no build-time link) |
iso-bmff-wasm |
WASM bindings for iso-bmff (browser) |
mediaway-ffi |
Single C ABI facade (container / device / pipeline) |
mediaway-test-media |
Rust-generated test fixtures (local cache only) |
mediaway-avcli |
AV CLI (mux; not affiliated with FFmpeg) |
mediaway-avprobe |
Media probe CLI (not affiliated with FFmpeg) |
OS backends live as #[cfg]-gated modules inside the facade crates
(mediaway-encoder, mediaway-decoder, mediaway-device).
# Toolchain (rust-toolchain.toml pins stable)
rustup show
# Hooks
cargo install lefthook cargo-deny
lefthook install
# CI (GitHub Actions) runs the same gates on push/PR to main β see docs/conventions/hooks.md
# Optional
cargo install cargo-nextest gitleaks # or scoop/brew for gitleaks- CONTRIBUTING.md β how to contribute
- docs/contributing/pull-requests.md β PR checklist (doc sync, quality gates, β¦)
- docs/contributing/for-agents.md β for AI assistants helping contributors
- docs/contributing/ β getting started, docs map, PRs
- docs/spec/ β product vision and design
- docs/conventions/ β commits, hooks, style, license, testing
- docs/roadmap.md β platform order and crate roadmap index
- Codec support tables β this README (Β§ Codec support)
- Container support table β this README (Β§ Container support)
- Device table β this README (Β§ Device)
- Per crate:
crates/<name>/README.md,crates/<name>/docs/roadmap.md,crates/<name>/adr/
- License: MIT OR Apache-2.0 β LICENSE-MIT, LICENSE-APACHE.
- Cargo graph: no GPL/LGPL (etc.) deps; no linking
libav* / FFmpeg libraries in shipped crates. See docs/spec/vision.md Β§ License & dependency boundary. - System
ffmpeg/ffprobeonPATHare optional test/dev oracles only (ADR-0002).