Kinect v2 WebRTC Streamer (Python)

This is a minimal Python WebRTC server to broadcast a video feed to a browser. It currently supports a webcam via OpenCV and includes a placeholder for libfreenect2 (Kinect v2) integration.

Once we confirm environment and dependencies, we can swap the source to the Kinect feed and support selecting color/ir/depth.

Quick start

1. Create a virtualenv and install deps

   • python -m venv .venv
   • source .venv/bin/activate
   • pip install -U pip
   • pip install -e ./kinect2-webrtc

2. Run the server (kinect source by default)

   • python -m kinect2_webrtc.server

3. Open the client

   • Visit http://localhost:8080 in a browser and click "Start" to receive the stream.

Client integration

This server uses a minimal, stateless WebRTC signaling flow. There are no peer IDs or rooms. Each client posts an SDP offer and gets an SDP answer back for a one‑way video stream from the server.

• Signaling endpoint: POST /offer
• Request body (JSON): { "sdp": "<offer sdp>", "type": "offer" }
• Response body (JSON): { "sdp": "<answer sdp>", "type": "answer" }
• Media: one video track from the server; client receives only (no upstream media required).

Browser example (vanilla JS)

<video id="video" autoplay playsinline controls></video>
<script>
  async function connect() {
    const pc = new RTCPeerConnection();
    const video = document.getElementById('video');
    pc.ontrack = (e) => { video.srcObject = e.streams[0]; };

    // Ensure a video m-line exists in the offer
    const t = pc.addTransceiver('video', { direction: 'recvonly' });

    // Prefer H.264 for better Safari compatibility when available
    if (t.setCodecPreferences && RTCRtpSender.getCapabilities) {
      const caps = RTCRtpSender.getCapabilities('video');
      if (caps && caps.codecs) {
        const h264 = caps.codecs.filter(c => c.mimeType === 'video/H264');
        const rest = caps.codecs.filter(c => c.mimeType !== 'video/H264');
        if (h264.length) {
          try { t.setCodecPreferences(h264.concat(rest)); } catch {}
        }
      }
    }

    await pc.setLocalDescription(await pc.createOffer());
    const res = await fetch('http://<server-host>:8080/offer', {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ sdp: pc.localDescription.sdp, type: pc.localDescription.type })
    });
    const answer = await res.json();
    await pc.setRemoteDescription(answer);
  }
  connect();
  // Call pc.close() when done.

Key points for client implementors

• No peer ID: The server creates a new RTCPeerConnection per request to /offer; no ID is required.
• Multiple viewers: Call /offer from each client. The server sends the same outbound track to all peers.
• ICE/NAT: Only host ICE candidates are used (no STUN/TURN configured). Works best on localhost or within the same LAN. For WAN/cross‑NAT, add STUN/TURN to both sides (see Notes below).
• CORS: The example assumes same‑origin. If hosting your client on a different origin, proxy /offer through your server or enable CORS on the aiohttp app.
• Codecs: Depth is sent as 8‑bit grayscale packed into RGB; browsers negotiate a common codec (H.264 is preferred by the sample client for Safari).

Optional: Python receiver (aiortc)

If you need a headless receiver, you can implement the same offer/answer flow with aiortc:

import asyncio, json, aiohttp
from aiortc import RTCPeerConnection

async def main():
    pc = RTCPeerConnection()
    pc.addTransceiver('video', direction='recvonly')

    @pc.on('track')
    def on_track(track):
        print('Receiving track:', track.kind)
        # Consume frames or attach a recorder here

    offer = await pc.createOffer()
    await pc.setLocalDescription(offer)

    async with aiohttp.ClientSession() as s:
        async with s.post('http://<server-host>:8080/offer', json={
            'sdp': pc.localDescription.sdp,
            'type': pc.localDescription.type,
        }) as resp:
            answer = await resp.json()
    await pc.setRemoteDescription(answer)

    await asyncio.sleep(60)
    await pc.close()

asyncio.run(main())

Selecting streams

• --source: one of [opencv, kinect].
• --stream: one of [color, depth, ir]. IR is TBD.
• For kinect depth grayscale mapping, use --min-depth and --max-depth (meters).

Kinect integration plan

• Implemented: tiny C/C++ shim exposing a C API over libfreenect2, compiled as a shared library with CMake, loaded from Python via ctypes.
• Color and depth (grayscale 8-bit) are supported. IR can be added similarly.

Build the native shim (macOS example)

1. Ensure libfreenect2 is built (this repository) and Kinect v2 works with ./bin/Protonect.

2. Build the shim:

   • cd kinect2-webrtc/native
   • mkdir -p build && cd build
   • cmake -DCMAKE_BUILD_TYPE=Release ..
   • cmake --build . --config Release

   If CMake cannot find libfreenect2 automatically, set LIBFREENECT2_ROOT env var to your libfreenect2 root (it will look in ${LIBFREENECT2_ROOT}/include and ${LIBFREENECT2_ROOT}/build/lib).

3. Run the server with Kinect frames

   • python -m kinect2_webrtc.server --source kinect --stream color --kinect-lib kinect2-webrtc/native/build/libkinect2shim.dylib
   • or depth grayscale:
   • python -m kinect2_webrtc.server --source kinect --stream depth --min-depth 0.5 --max-depth 4.5 --kinect-lib kinect2-webrtc/native/build/libkinect2shim.dylib

Notes

• This uses aiortc and aiohttp for signaling; PyAV for frames. If installation fails in your environment, we can vendor wheels or build from source with system ffmpeg libraries.
• HEVC note: browsers’ HEVC-over-WebRTC support varies and aiortc does not yet negotiate HEVC. We currently send depth as 8‑bit grayscale packed into RGB and encode using a widely-supported codec (e.g., H.264). If HEVC monochrome is required, we can explore a GStreamer-based sender or a custom RTP payloader path compatible with your target browser (Safari).
• Networking: To accept LAN clients, run with --host 0.0.0.0 and connect to http://<server-ip>:8080. For Internet-facing scenarios, add STUN/TURN servers on both ends and serve over HTTPS (reverse proxy recommended). This repo does not include TURN by default.