Super JSON Mode: A Framework for Accelerated Structured Output Generation
Introduction
Super JSON Mode is an innovative Python framework designed to enhance the efficiency of generating structured outputs from large language models (LLMs). By deconstructing a target schema into smaller, atomic parts and enabling parallel processing, this framework facilitates quick and reliable output creation. It is compatible with high-performance LLMs such as those offered by OpenAI, as well as open-source models from Hugging Face Transformers and vLLM. As technology evolves, support for additional LLMs will be incorporated.
Key Benefits
Super JSON Mode offers a significant performance boost compared to traditional JSON generation methods that use prompting and Hugging Face Transformers. Users can experience up to a 10-fold increase in speed while enjoying greater predictability and reduced parsing issues.
The framework is extremely user-friendly, with a straightforward installation process: simply run pip install super-json-mode.
How Does It Work?
Super JSON Mode capitalizes on the inherent structure of formats like JSON or YAML, which can be parallel or hierarchical. By focusing on independent extraction of schema components, it avoids the inefficiencies of traditional methods.
For instance, consider a structured schema for a real estate listing that requires details such as address, price, and number of bedrooms. Instead of generating the entire schema in one go, Super JSON Mode handles each field separately. This reduces token usage, thus saving time on syntax already determined by the schema. Moreover, processing queries in batches rather than individually facilitates faster generation times due to the parallel capability of LLMs.
Installation and Usage
Users can easily install Super JSON Mode through PyPI by running pip install super-json-mode. For those preferring a manual setup, instructions for creating a conda environment and installing necessary dependencies are provided.
The framework is designed for ease of use and features a collection of examples in its examples folder. This includes integration with OpenAI's gpt-3-instruct-turbo and HuggingFace's Mistral 7B model. For illustration, it showcases how to extract character details and quarterly financial reports, highlighting flexibility across different applications.
Future Directions
Super JSON Mode is continuously evolving. Planned enhancements include qualitative output analysis, structured sampling to enforce type constraints, and dependency graph support for handling interdependent JSON keys. Additionally, local model support and integration with advanced frameworks like TRT-LLM are on the roadmap to further improve performance and usability.
Community and Citation
Super JSON Mode was developed as part of a systems-oriented machine learning course (CS 229), benefiting from the support and guidance of its teaching team. It is an open-source project available on GitHub, and users are encouraged to cite the repository when utilizing the library in their work:
@misc{ShenoyDerhacobian2024,
author = {Shenoy, Varun and Derhacobian, Alex},
title = {Super JSON Mode: A Framework for Accelerated Structured Output Generation},
year = {2024},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/varunshenoy/super-json-mode}}
}
Conclusion
Super JSON Mode stands out as a powerful tool for developers and data scientists looking for efficient and reliable structured output generation from LLMs. With its emphasis on parallel processing and schema deconstruction, it brings a robust framework to the world of data extraction and automation.