Moving Beyond the Broadcast

SCS Researchers Reimagine How Audiences Experience Live Sporting Events

The Breakdown:

• LiveSplats lets audiences experience sports events in 3D.
• Fans can choose their own camera angles for an immersive experience instead of watching on a flatscreen. 
• The system uses advanced rendering techniques to recreate scenes with high visual fidelity. 

* * *

Imagine standing court side during a championship basketball game, hearing the squeak of sneakers echo across the hardwood as players weave past each other. As the ball arcs toward the hoop, you change your perspective to get a close-up of it rattling through the rim as time expires.

Researchers at the Carnegie Mellon University School of Computer Science are working to make that kind of immersive sports experience possible. New technology, called LiveSplats, combines virtual reality technology and AI-powered rendering to allow fans to follow scenes from any angle, shift perspectives in real time and move through the game itself –– despite being miles from the arena. 

And for fans that miss out on tickets for their favorite team or can’t attend premier sporting events like the Olympics, FIFA World Cup or a Formula One race, this technology transports them there, offering new possibilities for global access.

“We experience the world in 3D, but when we stream events, we are forced to watch them on a 2D flat screen with no freedom to choose our viewpoints,” said Saswat Mallick, a Robotics Institute (RI) master’s student and member of the research team. “LiveSplats turns passive viewing into an interactive experience, where audiences are no longer just spectators but are participants in the scene. Traditional video simply can’t provide that level of immersion.”

At the core of LiveSplats is Gaussian splatting, a 3D rendering technique that reconstructs scenes using millions of tiny, cloud-like elements instead of traditional geometric models. Each of these elements stores information about position, color and transparency, and forms a complete 3D representation of a scene. Gaussian splatting allows the same scene to be rendered from many different viewpoints without being rebuilt each time. The method enables fast, flexible visualization, which is an important requirement for live, rapidly changing environments.

But even with Gaussian splatting, reconstructing live events at scale remains challenging. The LiveSplats system solves this problem by splitting scenes into smaller components and separating elements such as player motion from more static background features like the court, stands or scoreboard. By breaking the problem into manageable pieces, the system can process different parts in parallel, improving both efficiency and visual fidelity.

“In sports, capturing player motion is central to the experience,” said Fernando De la Torre, an RI research professor and leader of the Human Sensing Lab, where the research was done. “Isolating and reconstructing human movement independently allows the system to better represent the quick interactions and subtle details that make these events so exciting to watch.”

A key challenge in building these systems is the lack of ground truth — real-world training data for complex, multiview sports scenes with precise, known information about what’s actually happening in the scene. To address this issue, the researchers created a synthetic benchmark dataset, called Basketball Synthetic BenchmarK for Enhanced Telepresence (BASKET), that includes many different camera views and detailed information like player positions, depth and movement. Some scenes are full games captured from virtual cameras, while others are simpler setups designed to test specific challenges like lighting changes or fast movement. Using this dataset gives researchers a controlled way to evaluate how well their methods work before trying them on real games.

“There are still hurdles to overcome before systems like LiveSplats can be widely deployed. Visual challenges like reflections, the motion of a ball or glare on a court surface can all affect realism,” said Francisco Vicente, a senior project scientist in the RI and a research team member. “But even so, this work reflects a broader shift driven by advances in augmented and virtual reality hardware, more capable AI systems, and faster rendering techniques that enable actual immersive media.”

Beyond sports, LiveSplats’ potential applications extend to concerts, graduations and other large-scale live events. As the underlying technology matures, the research team envisions a future where multiperspective, 3D presence becomes a standard part of digital media, reshaping what it means to attend an event.

Along with Mallick, Vicente and De la Torre, the research team includes Junkai Huang, an RI master’s student; Alejandro Amat, a research associate in the Human Sensing Lab; Marc Ruiz Olle, a technical artist in the Human Sensing Lab; Albert Mosella-Montoro, a senior project scientist in the RI; and Bernhard Kerbl, a former visiting researcher in the Human Sensing Lab.  

To learn more about LiveSplats, visit the project website. 

For More Information: Aaron Aupperlee | 412-268-9068 | aaupperlee@cmu.edu