peaks-consolidation - Cross-Platform Solution for Large CSV File Preview and Query Optimization

Introduction to the Peaks-Consolidation Project

Overview

The Peaks-Consolidation project offers a suite of cross-platform applications designed to enhance file handling and data processing, particularly for large CSV files. The tools created in this project are built using different programming languages like Golang, Rust, and Python, showcasing flexibility and adaptability for various user needs. This document provides an insight into the primary functionalities and concepts behind the project.

Instant File Preview and Validation

One of the core features of this project is its ability to provide instant file previews and validations, especially useful for handling large CSV files. Unlike typical CSV file tools that assume commas as delimiters, this application can automatically detect different delimiters, ensuring accurate file parsing. It divides a massive file into 100 partitions by default, validating the first row of each partition. This results in a more efficient and quicker way to sample data (up to 100 sample rows), with the first 20 rows available for a quick view on the screen. Users interested in altering the preview settings can adjust the partition number easily within the source code.

To access these tools, users can explore the following download links:

Golang Version: main.go
Rust Version: main.rs
Python Version: Peaks.py

A detailed demonstration of these functionalities is available via a demo video, which can be found here.

Query Statement for Diverse Data Sources

Peaks-Consolidation enables users to write custom query statements for various data sources, such as files, in-memory tables, and network streams. While the syntax may seem intricate at first, it simplifies complex data operations by employing user-defined functions for data extraction, transformation, and loading.

Examples of Use Cases:

File Expansion
```
ExpandFile = from Fact.csv to 1BillionRows.csv
.ExpandFactor: 123
```
The above statement indicates an operation to expand a file's data size.
Data Joining and Transformation
```
JoinTable = from 1BillionRows.csv to Test1Results.csv
.Filter: Saleman(Mary,Peter,John)
```
This example includes filtering by specific criteria before joining tables.
File Splitting
```
SplitFile = from Test1Results.csv to FolderLake
```
This functionality allows users to split large files into more manageable datasets.

These are just a few examples highlighting the diverse capabilities of this toolset, allowing users to perform sophisticated data manipulations effortlessly.

Command List - Core Features

The project also provides a rich set of commands for diverse data operations. Below is a summary of key commands:

AddColumn: Perform mathematical operations, such as addition or multiplication, to create new columns.
Filter and Group: Filter data using a wide range of comparison operators and group data for aggregated insights like count, sum, max, or min.
Join: Merge datasets using multiple join types, including AllMatch and InnerJoin.
Order and View: Sort data by various criteria and generate views for analysis.
Read and Write: Import and export data from/to CSV files, with capabilities to "Expand" datasets by specified factors.

These commands empower users to handle large datasets efficiently, supporting both exploratory data analysis and large-scale data processing tasks.

In conclusion, the Peaks-Consolidation project equips users with robust tools to manage, analyze, and derive insights from complex datasets, all while offering the flexibility of cross-platform usability and customization. Whether dealing with massive CSV files or integrating data from various sources, this project stands out as a valuable resource for data professionals and enthusiasts alike.