Skip to content
nyxwaysPublic

About

High-performance native-first media engine: Zero-Copy paths (GPU/CPU handles), hardware offloading, sans-io mux/demux cores, permissive MIT OR Apache-2.0

Topics

Resources

Contributing

Security policy

Stars

7 stars

Watchers

1 watching

Forks

Latest commit

Β 

History

261 Commits

Folders and files

NameName
Last commit message
Last commit date
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

Mediaway Logo

Mediaway

High-Performance Native-First Media Engine

Zero-Copy Support (GPU/CPU Handles) β€’ Hardware Offloading β€’ Sans-IO Cores β€’ Permissive (MIT / Apache-2.0)

CI Docs License Status


Status: early development (0.x), building toward 1.0 with the community. Not recommended for production yet. Pre-1.0: public APIs, crate layout, and backends may change often (no stability guarantee) β€” contributions and real-hardware testing are what move each backend closer to 1.0. Details: docs/spec/status.md.

Rust media stack: high-level pipelines built from first-class low-level APIs β€” OS/GPU codec sessions, GpuBufferHandle, and sans-io mux/demux/bitstream cores. Prefer Zero-Copy paths (GPU handles or shared CPU buffers); name CPU readback, cross-API copies, and SW fallbacks when they exist.

Design: docs/spec/vision.md.

Covers device capture (camera, mic, screen, window, desktop audio), encode/decode, containers, and FFI / WASM bindings across Windows, Web, Linux, other. Layout: sans-io cores, facade crates with OS backends as #[cfg]-gated modules (e.g. mediaway-device contains windows/linux/web). C ABI: a single mediaway-ffi facade β€” docs/spec/c-ffi.md Β· docs/spec/crate-packaging.md.

Verified Windows slice

The Windows H.264 path has an enabled end-to-end verification slice:

  • Rust API: H.264 encode β†’ fragmented MP4 mux/demux β†’ CPU decode, including a trim/splice/re-encode round trip (crates/mediaway/tests/trim_and_splice_windows.rs).
  • C ABI: the same encode β†’ MP4 mux/demux β†’ decode flow through mediaway-ffi, including decoded pixel-content assertions (crates/mediaway-ffi/tests/decode_smoke.rs).

These tests skip only when the host has no usable Windows Media Foundation backend. The slice is verified; broader production readiness, API stability, and cross-platform hardware coverage remain in progress.

Repository: github.com/nyxways/mediaway

Questions / support: please open a GitHub Issue (English). Do not rely on private email for general inquiries.

Why Mediaway?

Mediaway resolves the fundamental trade-offs and fragmentation in modern multimedia engineering:

  • OS & Hardware Offloading (Small Binaries & Peak Efficiency): Rather than bundling heavy C/Rust software codecs that inflate binary sizes to tens of megabytes and consume high CPU power, Mediaway delegates heavy encoding/decoding workloads to native OS APIs (WMF, VideoToolbox, VA-API, WebCodecs) and dedicated GPU hardware engines (NVENC, QuickSync, AMF). This keeps binary footprints minimal while enabling low-power 4K/8K real-time execution.
  • Zero-Copy Engine & Framework Interop (Seamless DX): Designed for zero-copy integration with graphics frameworks and game engines (wgpu, Three.js / WebGPU, Unity, Godot). Passes native GPU surface handles (GpuBufferHandle / D3D11, D3D12, Vulkan, Metal pointers) directly across capture, processing, encoding, and rendering without CPU readback stalls.
  • Permissive & GPL-Free (MIT OR Apache-2.0): Shipped under MIT OR Apache-2.0 with zero LGPL/GPL dependencies (no linking to FFmpeg/libav*), eliminating licensing risks for commercial and proprietary applications.
  • Unified & Composable Stack: Integrates device capture (camera, mic, screen, window), hardware video/audio/image codecs, and container muxing/demuxing into a single cohesive pipeline.
  • Uncompromised Freedom + High-Level Ergonomics: Provides clean end-to-end abstractions (mediaway) while keeping raw GPU handles, sans-io state machines, and bitstream timebase controls 100% public and unblocked.

Quick start

All examples are in examples/, one capability per file, grouped by sector β€” container/ (mux/demux only), encode/ / decode/ (one codec direction, no container), device/ (one capture source, no encode), pipeline/ (composed end-to-end flows). Run any of them with:

cargo run --example <name>

Mux + demux roundtrip β€” all platforms

Register tracks, push packets, collect bytes, demux β€” no OS codec, no unsafe. The muxer is typed via a typestate (Open β†’ Live); the demuxer is fully streaming (feed bytes, poll packets).

use mediaway_common::{Bytes, CodecKind, Packet, Rational, StreamInfo, VideoGeometry};
use mediaway_container::mp4;

let mut muxer = mp4::Muxer::new();
let v_track = muxer.add_track(StreamInfo::Video {
    id: 0, codec: CodecKind::H264,
    time_base: Rational::new(1, 30),
    geometry: VideoGeometry { width: 1920, height: 1080 },
    extra_data: Bytes::new(),
})?;

let mut muxer = muxer.begin(); // Open -> Live
muxer.push_packet(&Packet {
    stream_id: v_track, pts: 0, dts: 0, duration: 1,
    is_keyframe: true, is_discard: false,
    payload: Bytes::new(),
})?;
muxer.flush();

let mut mp4_bytes: Vec<u8> = Vec::new();
muxer.poll_bytes(&mut mp4_bytes);

let mut demux = mp4::Demuxer::new();
demux.push_bytes(&mp4_bytes);
while let Some(pkt) = demux.poll_packet() { /* … */ }

Full runnable example: examples/container/mux_demux_mp4.rs


Auto encode to MP4 β€” high-level (Windows WMF backend shown)

mediaway composes the low-level VideoEncoder trait + mp4::Muxer for you: platform::AutoEncoder::open selects the best available path (Zero-Copy GPU β†’ CPU upload β†’ …), and EncodeSession handles the poll-loop β†’ mux wiring so callers just push frames.

use mediaway_encoder::auto::AutoVideoEncodeConfig;
use mediaway::{platform, EncodeSession};

let config = AutoVideoEncodeConfig {
    bitrate_bps: 8_000_000,
    // gpu_device: Some(GpuDeviceHandle::DirectX11(handle)), // for Zero-Copy
    ..AutoVideoEncodeConfig::new(CodecKind::H264, 1920, 1080, Rational::new(1, 30))
    // backend defaults to Auto, max_path_class to CpuUpload (ZC -> CPU upload, no readback/SW)
};

let encoder = platform::AutoEncoder::open(&config)?;
let mut session = EncodeSession::open(encoder)?;

session.write_frame(&nv12_frame)?;
let mp4_bytes = session.finish()?; // flush + mux flush + poll_bytes

The low-level path (VideoEncoder trait + manual poll_packet/mux loop) stays fully public and usable without this crate β€” see examples/encode/encode_h264.rs (encoder in isolation) and examples/container/mux_demux_mp4.rs.

Full runnable example: examples/pipeline/encode_to_mp4.rs


Screen record pipeline β€” Windows (DXGI + WMF), video + audio

platform::ScreenCapture / platform::Microphone / platform::AutoEncoder are typed against facade traits (&mut dyn VideoCapture / &mut dyn AudioCapture / &mut dyn VideoEncoder); platform-specific construction is the only OS-specific code, and it lives entirely in mediaway::platform.

use mediaway_common::Rational;
use mediaway_device::desktop::DesktopVideoCaptureConfig;
use mediaway_device::{Select, AudioCaptureConfig};
use mediaway::{platform, EncodeSession};

let tb = Rational::new(1, 30);
let mut cap = platform::ScreenCapture::open(&DesktopVideoCaptureConfig::screen(Select::Default, tb))?;
let mut mic = platform::Microphone::open(&AudioCaptureConfig::microphone(Rational::new(1, 48_000)))?;
let encoder = platform::AutoEncoder::open(&enc_config)?;
let mut session = EncodeSession::open(encoder)?;

loop {
    if let Some(frame) = cap.poll_frame()? {
        session.write_frame(&frame)?; // handles poll_packet -> mux internally
        cap.release_frame()?;
    }
    // … encode `mic.poll_frame()` PCM to AAC and mux as a second track β€”
    // full version below composes that against a shared `mp4::Muxer` directly,
    // since `EncodeSession` stays video-only/single-track by design (ADR-0014).
}

Full runnable example, with real second-track AAC audio muxed alongside video: examples/pipeline/screen_record.rs. For each capture source on its own (no encode), see examples/device/.


Decode β†’ trim β†’ splice β†’ re-encode β€” cross-platform (Windows + Linux)

A real non-linear edit built from the low-level VideoDecoder/VideoEncoder traits + mediaway-container mux/demux β€” no new DecodeSession/EditTimeline abstraction. Encodes two synthetic clips, decodes each back, drops the first/last frame of each (trim), concatenates what's left with renumbered contiguous timestamps (splice), and re-encodes.

use mediaway::platform;

let encoder = platform::AutoEncoder::open(&enc_config)?;
let decoder = platform::AutoDecoder::open(&dec_config)?; // dispatches per-OS under the hood
// … encode two clips, mux, demux, decode, trim by index/PTS, chain + renumber, re-encode

Full runnable example: examples/pipeline/trim_and_splice.rs. For decode on its own, see examples/decode/decode_h264.rs. Detail: docs/ai/wiki/pipeline/trim-and-splice.md.


Language support & FFI

Rust is the primary, always-first-class native API. Bindings for the supported host languages are real, verified code under bindings/ β€” not aspirational sketches. Each language links only the capabilities it needs (container, encoder, device, …), so app binaries don't pull in unused codecs.

Language / Host Interop Path Status Target Ergonomics & Use Cases Examples
Rust Native crates βœ… primary Primary API (100% Zero-Copy, Sans-IO, traits) examples/
C Direct C ABI βœ… verified ABI contract baseline, zero-overhead FFI bindings/c/examples/
C++ Thin RAII C ABI βœ… verified Native desktop apps, custom engines, rendering pipelines bindings/cpp/examples/
C# (.NET) P/Invoke βœ… verified Windows desktop, WPF/WinUI, Unity native plugins bindings/csharp/
Python ctypes / cffi βœ… verified Data processing pipelines, ML model input/output streams bindings/python/examples/
Node.js (TS) Native Addon / N-API (koffi FFI today) βœ… verified Node.js server-side video processing and CLI tools bindings/nodejs/examples/
Browser (TS) WASM + WebCodecs βœ… verified Browser-native high-performance video apps (no C FFI) bindings/browser/
Unity / Godot Native GPU Plugin Planned Seamless Zero-Copy GPU texture sharing & camera/screen record β€”

Status legend: βœ… verified = real binding source built and run against the native libraries; πŸ“ design = README brief + example code only (nothing compiles/ships yet). Per-language detail: bindings/README.md and each folder's own README.


Codec support

Mark Meaning
βœ… Hardware / real-session-verified β€” real hardware or a real live session was actually exercised (not just "tests for the claimed scope" β€” implementation + unit tests alone are πŸ†—, not βœ…)
⚑ Zero-Copy path β€” no payload memcpy (GPU handle or shared CPU buffer); implies βœ… β€” still needs real hardware verification, not just that the Zero-Copy code path exists
πŸ†— Implemented, compiles β€” not (yet) verified against real hardware / a real session
πŸ› οΈ Planned
❌ Attempted and genuinely blocked β€” no upstream API to build on, a hard version/license conflict, or a real query returned "unsupported." Not a "ran out of time" πŸ› οΈ.
πŸ‘» Not exercisable yet β€” license/patent blocked, no target hardware, out of scope, or no device/daemon/session available to run otherwise-tested code against

Cell = encode/decode. One mark means both; A/B means encode / decode.

OS Β· CPU

OS codec APIs (WMF, WebCodecs, VA-API, …) fed with CPU buffers.

Codec Windows Web Linux Apple Android
H.264 / AVC βœ… / βœ… βœ… / βœ… πŸ†— / πŸ†— πŸ†— / πŸ†— πŸ†— / πŸ†—
HEVC / H.265 βœ… / βœ… ❌ / βœ… πŸ†— / πŸ†— πŸ†— / πŸ†— πŸ‘»
AV1 πŸ› οΈ / βœ… βœ… ❌ / πŸ†— ❌ / πŸ†— πŸ‘»
VP9 βœ… / βœ… βœ… πŸ†— / πŸ†— ❌ / πŸ†— πŸ‘»
ProRes πŸ‘» πŸ‘» πŸ‘» πŸ†— / πŸ†— πŸ‘»
AAC πŸ†— / βœ… πŸ†— πŸ› οΈ πŸ†— / πŸ†— πŸ‘»
Opus βœ… / βœ… πŸ†— πŸ› οΈ πŸ†— / πŸ†— πŸ‘»

OS Β· GPU

Same OS APIs with GPU surfaces (GpuBufferHandle, DXGI, …). Video only.

Codec Windows Web Linux Apple Android
H.264 / AVC πŸ†— / πŸ†— βœ… πŸ†— / πŸ†— πŸ†— / πŸ†— πŸ‘»
HEVC / H.265 πŸ†— / πŸ†— πŸ†— / πŸ› οΈ πŸ› οΈ πŸ†— / πŸ†— πŸ‘»
AV1 πŸ†— / πŸ†— πŸ†— / πŸ› οΈ πŸ› οΈ ❌ / πŸ†— πŸ‘»
VP9 πŸ†— / πŸ†— πŸ†— / πŸ› οΈ πŸ› οΈ ❌ / πŸ†— πŸ‘»
ProRes πŸ‘» πŸ‘» πŸ‘» πŸ†— / πŸ†— πŸ‘»

GPU β€” by API

Graphics interop (D3D11, Vulkan, Metal, …) β€” which API your textures use. Video only.

Adapters: mediaway wgpu module.

Codec D3D11 D3D12 Vulkan Metal
H.264 / AVC πŸ†— / πŸ†— βœ… / πŸ› οΈ βœ… / βœ… πŸ†— / πŸ†—
HEVC / H.265 πŸ†— / πŸ†— βœ… / πŸ†— βœ… / βœ… πŸ†— / πŸ†—
AV1 πŸ†— / πŸ†— βœ… / πŸ†— ❌ / βœ… ❌ / πŸ†—
VP9 πŸ†— / πŸ†— πŸ‘» πŸ‘» ❌ / πŸ†—
ProRes πŸ‘» πŸ‘» πŸ‘» πŸ†— / πŸ†—

GPU β€” by vendor

Vendor SDKs (NVENC, AMF, QuickSync, …) β€” separate from OS + graphics interop. Video only. See mediaway-encoder for NVIDIA/AMD/Intel backends.

Codec NVIDIA AMD Intel Qualcomm
H.264 / AVC βœ… πŸ†— βœ… πŸ‘»
HEVC / H.265 βœ… πŸ†— βœ… πŸ‘»
AV1 βœ… πŸ†— ❌ πŸ‘»
VP9 πŸ‘» πŸ‘» πŸ‘» πŸ‘»

CPU / SW

Pure Rust sans-io software codecs β€” no C codec FFI (OpenH264, libvpx, …). Opt-in only; never a silent HW fallback. Detail: mediaway-sw (H.264 decode, AV1/Opus encode, PCM, audio processing).

Codec Status
H.264 / AVC πŸ†—
HEVC / H.265 πŸ‘»
AV1 πŸ†—
VP9 πŸ‘»
AAC πŸ‘»
Opus βœ…
PCM / raw πŸ†—

Container support

Freestanding mux/demux cores plus the mediaway-container facade (wraps all eight as Mediaway-typed StreamInfo/Packet modules: mp4, webm, wav, adts, mp3, ogg, flv, ts). Same marks as Β§ Codec support β€” not every πŸ†—-mux implements the shared Mux trait; see mediaway-container for which.

Format Mux Demux
MP4 / fMP4 βœ… βœ…
WebM βœ… βœ…
WAV / RIFF (PCM) πŸ†— πŸ†—
ADTS (raw AAC) βœ… βœ…
MP3 (MPEG Layer III) πŸ†— βœ…
Ogg πŸ†— πŸ†—
FLV βœ… βœ…
MPEG-TS πŸ†— βœ…

Device

What mediaway-device backends target (camera, mic, screen, window). Same marks as Β§ Codec support. ⚑ = Zero-Copy out (GPU GpuBufferHandle or shared CPU PCM without payload copy); cell = CPU capture / GPU surface where both apply (πŸ†— / ⚑).

Source Windows Web Linux Apple Android
Camera (video) βœ… πŸ†— πŸ†— πŸ†— πŸ†—
Microphone βœ… πŸ†— πŸ†— πŸ†— πŸ†—
Screen / display βœ… πŸ†— πŸ†— πŸ†— πŸ†—
Window ⚑ πŸ†— πŸ†— πŸ†— ❌
Desktop audio (system loopback / per-process) βœ… β€” β€” β€” β€”

Window ⚑ holds for an uncropped window, or a capture region at the origin. A region anywhere else costs one GPU copy per frame (GpuCopy, not ⚑), and only WGC accepts a region at all β€” the other backends refuse it with Unsupported. Desktop audio is WASAPI-only today; per-process loopback needs Windows 10 2004+.

Crates

Crate Role
mediaway-common Shared types (Rational, formats, GpuBufferHandle, packets/frames)
iso-bmff MP4 / ISOBMFF mux + demux core (fMP4, ClearKey CENC)
iso-cenc ClearKey CENC sample crypto (AES-128-CTR)
ebml-webm EBML / WebM mux + demux core
riff-wave-core WAV / RIFF PCM mux + demux core
adts-core ADTS (raw AAC) mux + demux core
mpeg-audio MP3 (MPEG Layer III) mux + demux core
ogg-core Ogg page/packet mux + demux core
flv-core FLV tag mux + demux core
mpeg-ts-core MPEG-2 Transport Stream mux + demux core
rtp-core RTP payloadization for H.264/HEVC (RFC 3550/6184/7798)
rtmp RTMP publish-client handshake + chunk stream + AMF0 command mux
mediaway-container Container facade: shared traits + typed mp4/webm/wav/adts/mp3/ogg/flv/ts, plus a replay ring of encoded packets
mediaway-encoder Encode traits + auto selection; Windows WMF / NVENC / QuickSync / Vulkan / WebCodecs / VA-API backends
mediaway-decoder Decode traits; Windows WMF (HW, DX11 Zero-Copy) / Vulkan / WebCodecs backends
mediaway-device Capture + playback traits; Windows DXGI/WGC/WASAPI, Linux portal+PipeWire+V4L2, Web getUserMedia/getDisplayMedia
mediaway Convenience pipeline (EncodeSession + platform auto-dispatch + wgpu interop)
mediaway-sw Pure Rust sans-io software codecs (H.264 decode, AV1/Opus encode, PCM, audio processing)
vpl-sys oneVPL FFI bindings (runtime-loaded; no build-time link)
iso-bmff-wasm WASM bindings for iso-bmff (browser)
mediaway-ffi Single C ABI facade (container / device / pipeline)
mediaway-test-media Rust-generated test fixtures (local cache only)
mediaway-avcli AV CLI (mux; not affiliated with FFmpeg)
mediaway-avprobe Media probe CLI (not affiliated with FFmpeg)

OS backends live as #[cfg]-gated modules inside the facade crates (mediaway-encoder, mediaway-decoder, mediaway-device).

Dev setup

# Toolchain (rust-toolchain.toml pins stable)
rustup show

# Hooks
cargo install lefthook cargo-deny
lefthook install

# CI (GitHub Actions) runs the same gates on push/PR to main β€” see docs/conventions/hooks.md

# Optional
cargo install cargo-nextest gitleaks   # or scoop/brew for gitleaks

Docs

License & dependencies

  • License: MIT OR Apache-2.0 β€” LICENSE-MIT, LICENSE-APACHE.
  • Cargo graph: no GPL/LGPL (etc.) deps; no linking libav* / FFmpeg libraries in shipped crates. See docs/spec/vision.md Β§ License & dependency boundary.
  • System ffmpeg / ffprobe on PATH are optional test/dev oracles only (ADR-0002).

About

High-performance native-first media engine: Zero-Copy paths (GPU/CPU handles), hardware offloading, sans-io mux/demux cores, permissive MIT OR Apache-2.0

Topics

Resources

Contributing

Security policy

Stars

7 stars

Watchers

1 watching

Forks

Releases

Packages

Used by

Contributors

Languages