GaussianShader

GaussianShader: Enhancing 3D Rendering with Shading Functions

Overview

GaussianShader is a cutting-edge project at the intersection of neural rendering and computer graphics. Developed by a team of researchers including Yingwenqi Jiang, Jiadong Tu, Yuan Liu, Xifeng Gao, Xiaoxiao Long, Wenping Wang, and Yuexin Ma, this project leverages the potential of neural 3D Gaussians to significantly enhance the quality and efficiency of 3D renderings, especially in scenes with reflective surfaces.

Problem Statement

With the rise of neural rendering, achieving real-time high-quality renderings has become possible. Yet, reflective surfaces pose a unique challenge due to their intricate lighting and material characteristics, which conventional methods struggle to handle effectively. GaussianShader addresses this gap by introducing a streamlined shading function combined with 3D Gaussians, enabling better rendering of reflective scenes without sacrificing speed or quality.

Key Features

• Real-time Rendering: GaussianShader ensures fast rendering speeds while producing high-fidelity images.

• Universal Application: It adapts to various surfaces, rendering both general and reflective materials seamlessly.

• Distinguishing Lighting Environments: The method allows rendering objects from any viewpoint in diverse lighting scenarios, enhancing flexibility and realism.

• Advanced Shading Techniques: By incorporating shading attributes alongside traditional attributes, it enriches the captured view-dependent appearances.

• Differentiable Lighting Map: The project uses an environment lighting map for more realistic lighting effects, crucial for reflective surfaces.

How It Works

GaussianShader starts with neural 3D Gaussian spheres which integrate potential reflective attributes and shading capabilities. It encompasses an end-to-end training approach, resulting in a robust model that provides high material and lighting fidelity for both reflective and non-reflective surfaces. The outcome is a more lifelike rendering experience.

Installation and Usage

Setting up GaussianShader involves the following steps:

1. Clone the Repository:

   git clone https://github.com/Asparagus15/GaussianShader.git
   cd GaussianShader
   

2. Install Dependencies:

   conda env create --file environment.yml
   conda activate gaussian_shader
   

3. Running the Model: Download the example data provided and place it into the data folder. Use the command below to execute the optimizer:

   python train.py -s data/horse_blender --eval -m output/horse_blender -w --brdf_dim 0 --sh_degree -1 --lambda_predicted_normal 2e-1 --brdf_env 512 
   

4. Rendering: For rendering, use:

   python render.py -m output/horse_blender --brdf_dim 0 --sh_degree -1 --brdf_mode envmap --brdf_env 512
   

Dataset

GaussianShader has been evaluated using various datasets such as NeRF Synthetic, Tanks&Temples, Shiny Blender, and Glossy Synthetic. Conversion tools like nero2blender.py facilitate data preparation for different formats.

Future Enhancements

The project is under continuous development with plans to:

• Provide detailed argument explanations.
• Introduce residual color training capabilities.

Acknowledgements

This project builds upon the work of several foundational projects, notably gaussian splatting, Ref-NeRF, nvdiffrec, and Point-NeRF. Special thanks to their authors for sharing their insights and codes.

Conclusion

GaussianShader advances 3D rendering technology with its innovative approach, paving the way for more realistic and efficient rendering solutions in interactive applications and beyond. Through its novel integration of shading functions, it enhances the portrayal of reflective surfaces—a testament to the evolving capabilities of neural rendering.