SDT - Handwriting Generation Enhanced Through Writer and Character Style Separation

🔥 Disentangling Writer and Character Styles for Handwriting Generation

📢 Introduction

The Style-Disentangled Transformer (SDT) project focuses on generating online handwriting that aligns with specific content and stylistic preferences. Whereas previous methods relying on recurrent neural networks (RNNs) predominantly captured a person's overall writing style, the SDT addresses this by recognizing and disentangling subtle style differences even within a single person's writing. This results in enhanced style imitation capabilities. The SDT project also introduces an offline-to-offline framework aimed at improving the quality of offline Chinese handwriting generation.

📅 News

The SDT project has seen notable progress and attention:

In July 2024, a new handwritten text generation method, One-DM, was accepted by ECCV 2024.
In January 2024, a tutorial and code for synthesizing handwriting with customizable styles were added.
In December 2023, a top video blogger on Bilibili featured the project, attracting almost a million views.
2023 marked several milestones, including acceptance to the CVPR 2023 conference and reports by various media outlets. Initial datasets and codes were also released.

📺 Handwriting Generation Results

The SDT project showcases impressive results in handwriting generation, including:

Online Chinese handwriting, demonstrating the SDT’s capabilities in producing engaging and realistic characters.
Application across various scripts, showcasing versatility by generating different handwriting styles akin to various alphabets.
Offline Chinese handwriting enhancement, extending SDT’s efficiency to offline scenarios.

🔨 Requirements

To set up the project environment, the following steps are recommended:

conda create -n sdt python=3.8 -y
conda activate sdt
# install all dependencies
conda env create -f environment.yml

📂 Folder Structure

The SDT project is structured as follows:

SDT/
│
├── train.py - main script to start training
├── test.py - generate characters via trained model
├── evaluate.py - evaluate generated samples
│
├── configs/*.yml - configuration files for training
├── data_loader/ - manage data loading
├── model_zoo/ - pre-trained content encoder models
├── data/ - directory for experimental datasets
├── model/ - networks, models, and losses
├── saved/ - stores trained models, visualizations
├── trainer/ - training scripts
└── utils/ - utility functions

💿 Datasets

Datasets for Chinese, Japanese, and English handwriting are available on Google Drive and Baidu Netdisk (PW:xu9u). Users should download, unzip, and move them to the /data directory.

🍔 Pre-trained Models

Pre-trained models are available to facilitate easy setup and testing:

Content encoder models, available on Google Drive and Baidu Netdisk.
SDT models, available on both platforms for immediate application without retraining.

🚀 Training & Testing

Training

For the Chinese dataset:

python train.py --cfg configs/CHINESE_CASIA.yml --log Chinese_log

For the Japanese dataset:

python train.py --cfg configs/Japanese_TUATHANDS.yml --log Japanese_log

For the English dataset:

python train.py --cfg configs/English_CASIA.yml --log English_log

Testing

For generating Chinese handwritings:

python test.py --pretrained_model checkpoint_path --store_type online --sample_size 500 --dir Generated/Chinese

Evaluation

Set data_path and run for evaluating generated samples:
```
python evaluate.py --data_path Generated/Chinese
```

🏰 Practical Application

An exciting application has been developed by P0etry-rain, with a pipeline that transforms SDT-generated models into TTF files, supporting flexible layout modifications. This practical application highlights the project's real-world potential in customizable font creation.

❤️ Citation

Researchers and developers who find SDT's contributions valuable are encouraged to cite the project:

@inproceedings{dai2023disentangling,
  title={Disentangling Writer and Character Styles for Handwriting Generation},
  author={Dai, Gang and Zhang, Yifan and Wang, Qingfeng and Du, Qing and Yu, Zhuliang and Liu, Zhuoman and Huang, Shuangping},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={5977--5986},
  year={2023}
}

This detailed overview summarizes the SDT project’s goals, structure, achievements, and potential applications, highlighting its innovative approach to handwriting generation.