jtokkit

🚀 JTokkit - Java Tokenizer Kit

JTokkit is a sophisticated Java library designed for tokenizing text, specifically crafted for use with OpenAI models. This versatile tool is particularly beneficial for those involved in natural language processing tasks, providing essential functionalities such as token counting to optimize requests for models like GPT-3.5.

📖 Introduction

JTokkit bridges the gap for Java developers by offering capacities similar to the Python-based tiktoken library. With an easy-to-use interface, it allows seamless processing of input text, facilitating natural language tasks by efficiently encoding and decoding text data.

🤖 Features

JTokkit boasts a rich feature set aimed at enhancing usability and performance:

• Encoding and Decoding Capabilities: It supports various encoding schemes such as r50k_base, p50k_base, p50k_edit, cl100k_base, and o200k_base.

• User-Friendly API: Designed for simplicity, JTokkit allows easy interaction with its API.

• Custom Encoding Extensibility: Users can extend the library to incorporate custom encoding algorithms as needed.

• No External Dependencies: This ensures a clutter-free integration into projects.

• Compatible with Java 8 and Above: Broad compatibility ensures it can be used in a wide array of Java applications.

• Efficient Performance: JTokkit is optimized for speed, being 2-3 times faster than comparable tokenizers.

📊 Performance

Performance is a paramount feature of JTokkit, being significantly faster than other similar libraries. Detailed performance benchmarks demonstrating this speed advantage are available for users interested in the technical specifics.

🛠️ Installation

To integrate JTokkit into a Maven project, developers can add the following dependency to their project setup:

<dependency>
    <groupId>com.knuddels</groupId>
    <artifactId>jtokkit</artifactId>
    <version>1.1.0</version>
</dependency>

For Gradle users, the following dependency can be used:

dependencies {
    implementation 'com.knuddels:jtokkit:1.1.0'
}

🔰 Getting Started

Using JTokkit is straightforward. Developers can instantiate a new EncodingRegistry and fetch the desired encoding through getEncoding. The encode and decode methods facilitate the processing of text data.

EncodingRegistry registry = Encodings.newDefaultEncodingRegistry();
Encoding enc = registry.getEncoding(EncodingType.CL100K_BASE);
IntArrayList encoded = enc.encode("This is a sample sentence.");
String decoded = enc.decode(encoded);

These classes are designed to be thread-safe, making them highly adaptable for integration into diverse applications.

➰ Extending JTokkit

For users looking to adapt JTokkit for specific needs, the library is easily extendable. Developers can implement the Encoding interface for custom encodings or augment existing algorithms with new parameters.

Here is an example of registering a custom encoding:

EncodingRegistry registry = Encodings.newDefaultEncodingRegistry();
Encoding customEncoding = new CustomEncoding();
registry.registerEncoding(customEncoding);

For more details, users can refer to the comprehensive JavaDoc provided with the library.

📄 License

JTokkit operates under the MIT License, ensuring that it is free to use, modify, and distribute. More information is available in the LICENSE file.

JTokkit empowers developers with the tools necessary for efficient and effective text processing. Its flexibility and performance make it an invaluable resource for any Java-based natural language processing endeavor.