datachain

Introduction to DataChain

DataChain is a cutting-edge Python library designed to manage data efficiently, especially for applications in artificial intelligence. It aims to organize unstructured data into well-structured datasets and allows users to handle data at scale directly on their local machines.

Key Features

• 📂 Storage as a Source of Truth

  • DataChain allows users to process unstructured data without creating redundant copies. It supports data from sources like Amazon S3, Google Cloud Platform (GCP), Azure, and local file systems.
  • It can handle various types of data, including images, videos, texts, PDFs, JSONs, CSVs, and parquet files.
  • Files and metadata are unified into persistent, versioned, columnar datasets.

• 🐍 Python-Friendly Data Pipelines

  • Using DataChain, users can manipulate Python objects and their fields directly.
  • The library includes built-in parallelization and supports out-of-memory computation without needing to employ SQL or Spark.

• 🧠 Data Enrichment and Processing

  • Users can generate metadata using local AI models and LLM APIs.
  • The library provides capabilities to filter, join, and organize data by metadata and allows for vector embedding searches.
  • Data can be passed to machine learning libraries like Pytorch and Tensorflow or exported back to storage.

• 🚀 Efficiency

  • DataChain supports parallelization and out-of-memory workloads, alongside data caching.
  • It allows vectorized operations on Python object fields like summing, counting, averaging, etc., and optimizes vector searches.

Getting Started

To start using DataChain, you can install it via pip:

$ pip install datachain

Working with JSON Metadata

DataChain can select files based on JSON metadata. For example, if you have images of cats and dogs annotated with ground truth and model inferences, you can filter these files using their metadata. This allows you to download only the images with high-confidence predictions.

Data Curation with AI Models

DataChain supports batch inference using AI models. For instance, you can use the transformers library to perform sentiment analysis on text files, filtering and copying positive sentiment files to a specified directory.

LLM Judging Chatbots

Large language models (LLMs) can be employed as universal classifiers. Using a service like Mistral, you can evaluate chatbot dialogues for success, processing multiple files simultaneously.

Serializing Python Objects

DataChain can serialize the entire response from LLMs for analytics, such as analyzing the number of tokens used or the model's performance parameters.

Iterating Over Python Data Structures

DataChain allows iteration over dataset objects, supporting workflows that operate beyond in-memory limits. Users can retrieve datasets and process stored information efficiently.

Vectorized Analytics

Users can perform operations within the database without the need for data deserialization. For instance, calculating the cost of API usage based on token consumption can be achieved swiftly.

PyTorch Integration

Data prepared using DataChain can be exported or directly used in PyTorch data loaders. This facilitates the process of training machine learning models using structured data.

Tutorials and Support

DataChain offers various tutorials to get started and delve deeper into its capabilities:

• Getting Started Guide.
• Tutorials on multimodal data handling and JSON metadata reading.
• Guides on evaluating chatbots using LLMs.

For community support, documentation, or reporting issues, DataChain provides resources through its website, GitHub issues page, Discord chat, email, and Twitter. Contributions to the project are welcomed, and the Contributor Guide is available for those interested in participating in its development.