toon3d - Innovative 3D Reconstruction of Cartoons Beyond Geometric Consistency

Toon3D: Bringing Cartoons to Life in 3D

Toon3D is an innovative project that tackles the intriguing challenge of 3D reconstruction from images that lack geometric consistency—specifically, cartoons. While humans can effortlessly perceive three-dimensional details from two-dimensional drawings, machines have struggled with this capability. Toon3D bridges this gap, allowing users to recreate the scenes from their favorite cartoons in three-dimensional space. The project details and additional resources are available on the Toon3D website.

Setting up the Environment

To get started with Toon3D, users need to set up their environment. This involves creating a new Conda environment and installing Toon3D through a simple command:

pip install -e .

Downloading the Toon3D Dataset

The next step involves obtaining the necessary data. Users can download the Toon3D dataset from Hugging Face and place these datasets in the data/processed directory.

Running the Structure from Motion (SfM) Pipeline

For users who wish to work with custom data, Toon3D offers an optional SfM pipeline. This involves several steps for data preparation and processing:

Obtaining Images

Users can either download prepared image collections from Google Drive or use their own images. The images should be organized within the data/images/[dataset] directory.

Processing Data

This step involves depth estimation and segmentation. Users need to install the necessary tools using:

pip install segment_anything @ git+https://github.com/facebookresearch/segment-anything.git

Additionally, SAM weights can be downloaded for processing. To initialize data processing, use:

tnd-process-data initialize --dataset [dataset] --input_path [input_path]

This command helps prepare the images and organize them into datasets.

Adding More Images

If users decide to expand their dataset later, they can easily incorporate more images by running:

tnd-process-data add --dataset bobs-burgers-dining --input_path data/images/more-bobs-burgers-dining-photos

Labeling Images

To label images efficiently, Toon3D provides a dedicated tool at labeler.toon3d.studio. Users have two main options:

Upload ZIP folders containing processed data.
Host processed data publicly using a command such as:
```
tnd-server --path data/processed --port 8000
```

Then, navigate to the online labeler to annotate your dataset.

Running Structure from Motion

The magic of 3D reconstruction happens here. Users run the SfM pipeline to generate dense 3D models from images:

tnd-run --dataset bobs-burgers-dining

Dense Reconstruction

For generating high-quality 3D representations, Toon3D supports dense reconstruction through:

ns-train toon3d --data data/nerfstudio/bobs-burgers-dining

Users can render a camera path to visualize the 3D scene or generate videos by interpolating between various views with:

tnd-render camera-path --load-config [load-config] --camera-path-filename [camera-path-filename] --output-path [output-path].mp4

tnd-render interpolate --load-config [load-config] --output-path [output-path]

Understanding Project Structure

All outputs from the Toon3D processes are organized in the outputs folder, categorized by the type of experiment:

outputs/[dataset]/run/[timestamp] for SfM experiments
outputs/[dataset]/toon3d/[timestamp] for MVS experiments

Citing Toon3D

Academics and researchers who benefit from Toon3D are encouraged to cite the associated research paper to acknowledge its contributions to the field:

@inproceedings{weber2023toon3d,
title = {Toon3D: Seeing Cartoons from a New Perspective},
author = {Ethan Weber* and Riley Peterlinz* and Rohan Mathur and
    Frederik Warburg and Alexei A. Efros and Angjoo Kanazawa},
booktitle = {arXiv},
year = {2024},
}

With Toon3D, the world of cartoons transcends the flat confines of screens, offering a new perspective and inviting explorations into the dynamic realm of 3D reconstruction.