ACG2vec: Vectorizing Anime, Comics, and Games
Introduction
ACG2vec, short for Anime Comics Games to vector, is an innovative project dedicated to the exploration and application of deep learning within the realm of anime, comics, and games. It aims to convert various elements related to these fields into vector representations for a myriad of applications.
The project showcases online capabilities like text search, image-to-image search, text-to-image search, and image scoring prediction. Interested individuals can explore these features through the online preview available at cheerfun.dev. The project's open-source repositories for both backend and frontend development can be found on GitHub.
Modules
The ACG2vec project consists of several key modules encompassing model development, web service deployment, and containerized application setups:
-
Model Module:
- acgvoc2vec: A sentence-transformers model fine-tuned on anime-related sentence pairs to generate text feature vectors for tasks like tag recommendation and search. This model is available for online use via Hugging Face.
- dclip: An adaptation of the CLIP (Contrastive Language–Image Pre-training) model fine-tuned on the Danbooru2021 dataset.
- pix2score: A multi-task model based on ResNet101, designed for predicting attributes like favorite count and view count for anime illustrations.
- illust2vec: A semantic feature extraction model for illustrations, derived by modifying the DeepDanbooru model.
- real-cugan_tf: A TensorFlow implementation of the Real-CUGAN model, regarded as a top-tier tool for super-resolution in anime images, capable of running in browser environments.
- recSys4Pix: A streamlined system for modern recommendation practices.
-
Webapp Module: Offers web services like illustration tag prediction, image feature extraction, and text feature extraction ready for immediate use.
-
Docker Module: Provides a containerized deployment setup, enriched with necessary configuration and resource files (currently under development).
Features Preview
The platform allows users to engage with innovative animation-related tasks like:
Semantic Text Search
Semantic Image Search
Illustration Scoring Prediction
Image-to-Image Search
Image Super Resolution
Technical Architecture
ACG2vec employs a sophisticated architecture combining TensorFlow for model training, Spring Boot for web services, TF-serving for model deployment, Milvus for vector retrieval, and Tendis for metadata storage. The entire setup can be deployed efficiently using Docker Compose.
Deployment
To deploy ACG2vec, clone the repository, download the necessary model packages, and utilize Docker Compose for deployment. Instructions are provided for easy setup and operation.
# Clone the repository
git clone https://github.com/OysterQAQ/ACG2vec-docker.git
# Download model package from release (1.0.0_for_tf_serving) and extract it to tf-serving/models
# Deploy using Docker Compose
docker-compose up -d
Conclusion
ACG2vec stands at the intersection of technology and creativity, pushing the boundaries of what's possible in anime-related AI applications. Its integration of deep learning techniques with practical, real-world applications make it a valuable resource for developers and enthusiasts alike.