Introduction to Steampipe
What is Steampipe?
Steampipe is an innovative tool that allows users to query APIs and services using SQL, eliminating the need for ETL (Extract, Transform, Load) processes. This approach enables users to access data sources directly and in real-time, leveraging SQL as the standard language for data queries.
Key Features of Steampipe
-
SQL Query Language: Steampipe uses SQL, a long-standing, widely accepted standard for data access, which makes it easy for users to leverage their existing SQL skills.
-
Real-Time Data Access: Users can query APIs and services live, which means they can always access the most up-to-date information.
-
Performance and Speed: Steampipe offers rapid query speeds, allowing users to access data faster than traditional methods.
-
Parallel Querying: It can query multiple data sources simultaneously, improving efficiency and reducing wait times.
-
Single Binary Deployment: Steampipe is simple to install and deploy, whether locally or within CI/CD (Continuous Integration/Continuous Deployment) pipelines.
Getting Started with Steampipe
To begin using Steampipe, users need to install it on their system. It is available for MacOS, Linux, and Windows (using WSL2). Here’s a quick installation guide:
-
MacOS:
brew install turbot/tap/steampipe -
Linux or Windows (WSL2):
sudo /bin/sh -c "$(curl -fsSL https://steampipe.io/install/steampipe.sh)"
Once installed, users can enhance Steampipe’s functionality by installing plugins for various services like AWS, Azure, GCP, GitHub, Kubernetes, and more. An example command for installing a plugin for the Hacker News service is:
steampipe plugin install hackernews
After setting up, users can begin querying data. For instance:
steampipe query
> select * from hackernews_new limit 10
Documentation and Resources
Steampipe provides comprehensive documentation, covering topics such as running queries, managing Steampipe, using CLI commands, integrating services, and developing plugins.
Steampipe Plugins
The community has developed a vast range of plugins that map APIs to database tables. With support for over 2000 tables and many services like AWS, Azure, GCP, GitHub, Salesforce, and more, each plugin is documented for easy application.
Steampipe Distributions
Steampipe comes in various distributions to fit different use cases:
- Steampipe CLI: Utilize CLI to query APIs converted into tables within a bundled Postgres instance.
- Steampipe Postgres FDWs: Use native Postgres Foreign Data Wrappers to work with foreign tables.
- Steampipe SQLite Extensions: Translate APIs into SQLite virtual tables.
- Steampipe Export Tools: Separate binaries for exporting data straight from APIs, requiring no database.
- Turbot Pipes: A cloud-based solution to operate Steampipe collaboratively.
Development and Contributions
For those interested in contributing to Steampipe’s development, it supports open-source involvement. One can begin by cloning and building the Steampipe binary or developing new plugins. Steampipe operates under the AGPL 3.0 license.
Collaboration and Community
Steampipe invites users to contribute and engage with the community. Contributors can join discussions on Steampipe’s Slack channel and adhere to the community’s code of conduct and Contributor License Agreement.
Conclusion
Steampipe revolutionizes real-time data querying by harnessing the power of SQL and simplifying integration with various APIs and services. With its rapid deployment, extensive plugin support, and active community, Steampipe stands out as a powerful tool for developers and organizations seeking immediate access to actionable data across multiple platforms.