similarity-search-kit

Introduction to SimilaritySearchKit

SimilaritySearchKit is a Swift package designed for iOS and macOS applications to provide efficient text embeddings and semantic search capabilities directly on users' devices. It stands out for its emphasis on speed, extensibility, and privacy, allowing developers to implement sophisticated natural language processing (NLP) features without relying on cloud services. The package comes with a range of state-of-the-art NLP models and similarity metrics and supports custom options for enhanced flexibility.

Key Use Cases

SimilaritySearchKit can be applied across various scenarios, primarily focused on privacy and local data processing:

• Privacy-focused Document Search Engines: Users can build a search engine that processes sensitive documents entirely on their devices, ensuring that no sensitive data is exposed to external services.

• Offline Question-Answering Systems: Developers can implement systems that answer user queries by finding relevant information from a local dataset.

• Document Clustering and Recommendation Engines: The package enables automatic document grouping and organization based on textual content, enhancing the way documents are managed on devices.

Through these applications, SimilaritySearchKit supports the creation of powerful, privacy-preserving applications without compromising on performance.

Installation Process

Installing SimilaritySearchKit is straightforward thanks to its integration with the Swift Package Manager. You can easily add it to your Swift project by searching for the package URL in Xcode or by adding a line in your Package.swift file to include it as a dependency. This allows you to choose which pre-built models to include, optimizing the size of your final application.

How to Use

To utilize SimilaritySearchKit in your project, you begin by importing the framework and creating an instance of SimilarityIndex with the desired distance metric and embedding model. From there, you add text entries you want to search through to the index and then query the index to find similar items matching a specific query. The querying process outputs a search result array, highlighting the entries that closely match the query text.

Examples and Demonstrations

In the Examples directory, you can find several sample applications that showcase how SimilaritySearchKit can be employed:

• BasicExample: A simple multiplatform app demonstrating text similarity indexing.
• PDFExample: A macOS Catalyst application allowing semantic search across PDF contents.
• ChatWithFilesExample: An advanced macOS app for indexing an array of text files on a computer.

These examples serve as a practical guide to understanding the full potential of SimilaritySearchKit.

Available Models and Metrics

SimilaritySearchKit features several built-in models and metrics for different purposes like text similarity and question-answering:

Models:

• NaturalLanguage: Best for fast text similarity tasks with built-in Apple support.
• MiniLMAll and MiniLMMultiQA: Offer fast inference, ideal for text similarity.
• Distilbert: Provides the highest accuracy for question-answering tasks.

Metrics:

• DotProduct, CosineSimilarity, and EuclideanDistance: Metrics for measuring how similar or different text items are based on their vector representations.

These models and metrics align with the package's goal to offer efficient, on-device solutions for text processing.

Custom Implementations

Developers have the flexibility to customize major components by implementing protocols such as EmbeddingsProtocol and DistanceMetricProtocol. This customization capability allows for tailored solutions aligned with specific needs, from embedding creation to indexing and search logic.

Project Foundations

SimilaritySearchKit draws inspiration from prominent projects and innovations in the NLP space, such as HuggingFace Transformers and Sentence Transformers. The package aims to deliver the benefits of advanced NLP models in a format that aligns with Apple's privacy-focused ecosystem.

Looking Forward

There are exciting plans for expanding SimilaritySearchKit's functionality, including performance enhancements, more embedding models, and new features like summarization models and Metal acceleration. The project aims to keep widening its applications for developers seeking robust and private NLP solutions. Additionally, community feedback is welcomed to guide further development.

By providing flexible, on-device NLP functionalities, SimilaritySearchKit is paving the way for innovative applications that respect user privacy without sacrificing performance or capability.