This project is based on 3DGS, GS-IR and UnityGaussianSplatting.

3D reconstruction, originating from computer graphics, aims to recover informations of 3Dscenes from 2D images or sensor data and holds significant value in many fields. Among current 3D scene reconstruction methods, 3D Gaussian Splatting (3DGS) has emerged as a novel scene representation and rendering approach by leveraging learnable 3D Gaussian distributions and thereby achieves high-quality real-time rendering with remarkable computational efficiency.

Additionally, relighting technology, which aims to modify illumination conditions while preserving scene geometry, has important value of enhancing realism and immersion in related fields as well. Building upon existing techniques, this paper implements and refines a complete pipeline for 3DGS-based scene reconstruction and relighting.

• Firstly, the pipeline reconstructs 3D scenes and performs inverse rendering using 2D image data based on 3DGS and its inverse rendering extension GS-IR(3D Gaussian Splatting for Inverse Rendering), recovering scene appearance and furthermore physical material properties.
• Subsequently, this paper introduces RUGS (Relightable Unity Gaussian Splatting)—a new real-time relighting and rendering framework developed in Unity, based on the open-source 3DGS renderer UnityGaussianSplatting. After converting and processing the extended point cloud data, RUGS utilizes a custom rendering pipeline and compute shaders to achieve image-based lighting (IBL) for fast and realistic real-time rendering.
• Additionally, RUGS supports hybrid rendering with virtual objects and light sources, and provides curve-based scene editing features to further enhance practicality and demonstrate extensibility.

Through these open-source frameworks and methodologies, this paper demonstrates a comprehensive 3D scene reconstruction and relighting pipeline, along with high-quality real-time rendering results. With RUGS’s interactive and extensible framework, the proposed system provides a valuable foundation and reference for future research and applications in related fields.

In addition, this project also implements a sampling logic in Unity, which is able to capture RGBD data with a sampler for both BiRP and HDRP Unity scenes and converts these data to a input dataset. These datasets can be used for GS-IR after running convert.py from 3DGS.

To use RUGS in Unity, you need to prepare the model file trained by GS-IR which should be a .pth file. You can get this file using the instruction of GS-IR. And if you want to train datasets of a virtual scene, you can refer to the sampler aforementioned and try to use the sampler in /utils/UnitySample.

After you get your .pth file, you should use the python script in /utils/pth2ply.py to convert it into .ply file by putting it into the root folder of GS-IR and running: python pth2ply.py [model_name]

Finally, you can import your .pth file into the RUGS open Tools -> Gaussian Splats -> Create GaussianSplatAsset menu within Unity and modify its properties in Inspector to get the desired rendering effect.

Configuration

• 1.0.1: Modified formats of normals data. This should fix the problem that some graphic cards doesn't support R16G16B16_SFloat.

• 1.0.2: Modified data strcture in the file package/Editor/Utils/GaussianFileReader.cs. Now this will match the original GS-IR's data strcture.

For the specific implementation process or if meet any issues while reimplementing it, feel free to contact me.