Qiangjian Xi

⚙️ One library, real DSP speed. There’s a gap in the developer ecosystem. Teams doing DSP/ML often migrate to Python, because that’s where most tooling lives. But Python wasn’t built for high-throughput I/O — under load, it spawns new interpreters/processes, creating duplicated memory, slow cold starts, and more containers. Compute stays fast, but orchestration cost explodes. Meanwhile, Node.js/TS scale I/O well, but were never meant for serious compute. Pure-JS DSP hits performance walls. Even worse, serializing data between Redis ↔ DSP/ML adds latency and forces batching — slowing real-time pipelines. So devs are stuck: Python → strong math, weak concurrency Node → strong networking, weak compute dspx closes that gap — TypeScript DX + native C++/SIMD (AVX2/SSE3/SSE2/NEON) under the hood. 🧩 How dspx works FIR / FFT / Conv1D run in optimized C++/SIMD Memory-safe circular buffers → O(1) throughput TS handles Kafka / Redis / WebSockets Redis persistence < 0.5 ms Batched logging avoids I/O stalls At tiny batch sizes, N-API overhead can make naive JS look competitive. At realistic scale, batched native pipelines win. ⚡ Benchmarks Dell OptiPlex 3000 Micro (i5-12600T · AVX2 · Node 22) FFT: 2.6× faster than fft.js, ~9× faster than tfjs-node FIR (51-tap): ~4× faster than fili / naive JS Conv1D (128-kernel): ~3.5× faster than tfjs-node Moving Avg (O(1)): 559× more throughput/sec than naive JS Redis save/load: sub-ms Logging: <3% overhead 📊 Full tables + charts in carousel 💬 Observations SIMD FMA + contiguous memory → big FIR/Conv wins O(1) moving-average design → massive throughput gain Sub-ms Redis + low I/O overhead → real-time persistence in pure Node 🧠 Open Source dspx is Apache 2.0, free for commercial/academic use. Looking for contributors interested in: ARM NEON / Apple M / Graviton tuning Audio / sensor / biomedical DSP Visualization + benchmark tooling 📦 npm i dspx 🔗 https://lnkd.in/e-tWxAgu 🧠 Real-time DSP for Node, TypeScript & Redis 💼 Note I’m exploring opportunities in full-stack, real-time systems, performance engineering, and DSP. If your team works in this space, I’d love to connect. My next post will cover why sub-ms Redis latency isn’t just technical — it’s economic. Lower serialization + compute overhead → lower infra + energy cost. 🔖 Tags & Mentions #NodeJS #TypeScript #DSP #PerformanceEngineering #EdgeComputing #OpenSource #SIMD #Redis #Cplusplus #RealTime NodeJS Developer TensorFlow Google Microsoft JavaScript Developer Amazon Web Services (AWS) Vercel

5

1 Comment

What does comfortable, confident, safe autonomy feel like? Ride through San Francisco with NVIDIA founder and CEO Jensen Huang and NVIDIA VP of Automotive Xinzhou Wu as they discuss the technology powering the next generation of autonomous vehicles. ▶️ Watch now: https://bit.ly/4rz72M1

70

[ The Rubin Paradox: A 31% Drop in FP16 Efficiency?] NVIDIA's Rubin architecture undoubtedly pushes the raw performance envelope: * 2.75x increase for FP16 FLOPS * 5x increase for FP8 FLOPS However, this comes at a steep cost: * Power consumption has surged to 2,300W. * This is a massive increase, especially given that TDP remained constant at 700W from the H100 to the B100. The result? A critical shift in performance per Watt. The FLOPS/W metrics have significantly declined. In fact, FP16 FLOPS/W is down by a substantial 31% compared to the Blackwell (B100). It's clear why the GTC narrative steered toward Tokens/sec/W and Tokens/sec/User rather than absolute performance. This focus on inference efficiency over pure computational density is a telltale sign. This leads to a compelling question: Is planar process scaling finally at its limits? Perhaps this explains the push for new breakthroughs like: * Advanced Packaging - Die Stacking (Feynman) * Disaggregated Architectures (+Groq) ________________  H100   B100  Rubin GPU TDP[W]  700    700   2300 FP16[TFLOPS]     989  1750   4000 FP8[TFLOPS]     1979  3500   17500 #HBM4 #DieStacking #Feynman #DisaggregatedArchitecture #Dataflow #Groq #PowerEfficiency #TDP

6

Serving LLMs on TPUs with SGLang ! I've been exploring the JAX ecosystem for inference recently and came across SGL-JAX, a high-performance, JAX-based inference engine for LLMs, specifically optimized for Google TPUs. Some highlights: 🌳 The "Radix Tree" KV Cache: SGL-JAX implements a Radix Tree (similar to PagedAttention) to manage memory. This enables efficient prefix sharing for multi-turn chat or agentic workloads where the system prompt remains constant. ⚡ JAX-Native FlashAttention: It integrates a high-performance FlashAttention kernel directly into the JAX compute graph. This is critical for faster, more memory-efficient attention usage, particularly when dealing with long sequence lengths on TPUs. 🧩 Native Tensor Parallelism: It handles the sharding logic natively using JAX distributed primitives. It supports distributing large models (like the Qwen 3 MoE series) across multiple TPU devices without needing complex, manual mesh definitions. ⏳ Continuous Batching: It dynamically schedules incoming requests to maximize TPU core saturation. You aren't waiting for a fixed batch size to fill up while tail latency spikes. 🔧 Drop-in Compatibility: It exposes an OpenAI-compatible API standard. You can literally point your existing LangChain or LlamaIndex setup at the new endpoint and switch hardware backends seamlessly. For more info about the architecture and how it works, check out the repo below! #JAX #MachineLearning #TPU #LLMOps #GoogleCloud #AIEngineering

46

1 Comment

In this week’s AI infra chat, we sit down with Seiji Eicher, Distributed LLM Inference Engineer at Anyscale to unpack how large MoE models are actually served in practice, what it looks like to work deep in the LLM inference stack & more 🕐 Timestamps: 𝟬𝟬:𝟬𝟬 Why MoE comes up in modern AI infra 𝟬𝟬:𝟮𝟯 What Mixture of Experts actually changes inside a model 𝟬𝟭:𝟮𝟰 Why expert parallelism exists (and what becomes inefficient without it) 𝟬𝟮:𝟱𝟳 When MoE helps in practice 𝟬𝟰:𝟮𝟯 How Ray Serve fits into real inference stacks 𝟬𝟳:𝟬𝟳 What a Distributed LLM Inference Engineer works on day to day 𝟭𝟭:𝟭𝟵 Where the space is headed and how to learn more If you want to go deeper, Seiji gave a Ray Summit talk that dives into production-grade serving of MoE models, including KV-cache optimization, parallelism strategies, and multi-node orchestration with Ray Serve and vLLM. Check out the talk linked in the comments 👇

60

3 Comments