lerf

Introduction to LERF: Language Embedded Radiance Fields

LERF, short for Language Embedded Radiance Fields, is an exciting project that merges language processing with radiance field technology. At its core, LERF aims to advance rendering technologies by incorporating language as a key component in creating and visualizing complex radiative phenomena.

Installation Guide

To begin using LERF, it's important to first ensure you have the necessary dependencies. The LERF installation process follows specific integration guidelines designed for custom methods within Nerfstudio. Here’s a step-by-step guide to help you get started:

Install Nerfstudio Dependencies:
- Follow the detailed instructions available on the Nerfstudio website up to the point where you install "tinycudann."
Clone the LERF Repository:
- Use the command git clone https://github.com/kerrj/lerf to clone the LERF project repository to your local environment.
Install as a Python Package:
- Navigate to the cloned folder and install it by running python -m pip install -e .
Run Installation CLI:
- Execute ns-install-cli to complete the setup.
Check the Installation:
- Verify the installation by running ns-train -h. You should see commands including lerf, lerf-big, and lerf-lite.

Getting Started with LERF

Once you've set up LERF, you can start exploring its features. Launch the model training with the command ns-train lerf --data <data_folder>, specifying the folder where your data is stored.

Viewer Connection: Connect to your viewer by forwarding the viewer port, preferably using VSCode, and follow the link provided in the output to access the viewer at viewer.nerf.studio.
Visualizing Relevancy Maps: In the viewer, enter text queries and choose the relevancy_0 output to observe the relevancy maps linked to your input.

Relevancy Map Normalization and Resolution

Normalization: By default, the relevancy maps are scaled using the turbo colormap. To enhance visualization, set the range parameter to (-1.0, 1.0) and enable the Normalize option to use a complete colormap stretch.
Resolution Adjustment: The viewer dynamically adjusts resolution to maintain optimal training throughput. For higher resolution (512px or more), pause the training session as rendering can take additional time.

Variants: `lerf-big` and `lerf-lite`

LERF comes in multiple variants to suit different hardware capabilities:

lerf-lite: Designed for systems with limited GPU memory, this version reduces the network capacity and ray sample count.
lerf-big: For those with more GPU power, this version employs a larger model with ViT-L/14 to harness increased computational resources.

Extending LERF and Contributing

LERF is a continually evolving project with more features planned for integration into Nerfstudio. Users and developers can customize and extend LERF by leveraging its modular design, which supports various image encoders. The main script to study is lerf.py, which integrates language fields and visualization functionalities.

Issues and Contributions: For any challenges faced, users are encouraged to report them on GitHub. More lightweight versions may be created to support diverse platforms.
Custom Image Encoders: Implement different encoders by following the BaseImageEncoder interface, with examples available in files like clip_encoder.py.

Citation

For those who find LERF beneficial, you're encouraged to cite the project using the provided Bibtex entry, ensuring proper recognition of the developers' efforts at the International Conference on Computer Vision (ICCV) 2023.

LERF represents a cutting-edge approach to integrating language processing with radiance field technology, setting the stage for innovative applications in computer vision and rendering.