Physics3D - Explore the Application of Video Diffusion Models in Understanding 3D Gaussian Physics Dynamics

Physics3D Project Introduction

Overview

Physics3D is an advanced framework designed to merge the realms of simulation and rendering using 3D Gaussians. This innovative project revolves around learning the physical dynamics of objects through a video diffusion model. By leveraging this approach, the Physics3D platform offers an enriched understanding and representation of physical properties in a 3D space.

Features

Currently, the Physics3D repository is under development, with several features already made available:

Training Code Release: Users can access the training code to understand or contribute to the learning process of 3D physical properties.
Synthetic Data Release: A collection of synthetic data is accessible to aid in the training and testing of the models.
Upcoming Features: The team plans to release a detailed tutorial and a local demo to guide users through the system effectively.

Getting Started with Physics3D

System Setup

To start working with the Physics3D project, a Linux system is recommended. Follow these steps to set up the environment:

Create a virtual environment and install necessary Python packages:
```
conda create -n Physics3D python=3.9
conda activate Physics3D
```

Install required dependencies and clone relevant repositories for Gaussian processing:

pip install -r requirements.txt
git clone https://github.com/graphdeco-inria/gaussian-splatting
pip install -e gaussian-splatting/submodules/diff-gaussian-rasterization/
pip install -e gaussian-splatting/submodules/simple-knn/

Quick Start

Download Gaussian Models: Obtain pre-trained Gaussian models from OneDrive or integrate your own models following the gaussian-splatting guidelines.

Using Video Diffusion Models: Utilize text-to-video diffusion models like ModelScope to optimize physical parameters with the following command:

python simulation.py --model_path ./model/ball/ --prompt "a basketball falling down" --output_path ./output --physics_config ./config/ball_config.json

Tips for Optimal Performance

Parameter Initialization: Begin with parameters that are in harmony with real-world physics to accelerate and enhance the training process.
Particle Density: For objects characterized by high-frequency elasticity, increasing the particle density during simulations can lead to better outcomes.

Acknowledgements

The Physics3D project borrows heavily from several existing repositories, including DreamPhysics, threestudio, warp-mpm, and PhysGaussian. Additionally, the project makes use of open-source datasets such as PhysDreamer and BlenderKit.

Citation

For those who find Physics3D beneficial in their research or projects, please consider citing it as follows:

@article{liu2024physics3d,
  title={Physics3D: Learning Physical Properties of 3D Gaussians via Video Diffusion},
  author={Liu, Fangfu and Wang, Hanyang and Yao, Shunyu and Zhang, Shengjun and Zhou, Jie and Duan, Yueqi},
  journal={arXiv preprint arXiv:2406.04338},
  year={2024}
}

Contact Information

For any inquiries or more information about Physics3D, feel free to reach out to Fangfu Liu at [email protected] or Hanyang Wang at [email protected].

This introduction provides a comprehensive overview of what Physics3D is all about, how it operates, and how you can engage with the project for learning or contribution.