Secrets Patterns Database: A Comprehensive Guide
The Secrets Patterns Database stands out as the most extensive open-source resource for identifying secrets such as API keys, passwords, tokens, and more. This database is a vital tool designed to support secret scanning engines by providing regular expressions (regex patterns) that enable the detection of hidden or sensitive information.
π Key Features
- Extensive Regex Collection: The database includes over 1600 regular expressions tailored for uncovering a wide variety of secrets including passwords, API keys, and tokens.
- Format Versatility: The patterns are compatible with different secret detection tools, like Trufflehog and Gitleaks, thanks to their adaptable single format.
- Thorough Testing: Each regex pattern has been carefully reviewed and tested, especially for vulnerabilities such as ReDoS (Regular Expression Denial of Service) attacks.
- Confidence Levels: Patterns are organized according to their confidence levels, helping users gauge the reliability of each regex in detecting secrets.
β The Rationale
The challenge of identifying secrets lying within codebases is significant due to the limited regex resources available online. While tools like TruffleHog offer around 700 built-in rules, and GitLeaks provides about 60, these figures only scratch the surface of what's needed. The database fills this gap by offering a more robust and continuously updated collection of regex patterns.
This project is crucial as it consolidates the scattered efforts to maintain a comprehensive database for detecting potential security vulnerabilities stemming from exposed secrets. The database serves as a community-driven initiative, inviting security teams globally to contribute and enhance its breadth and accuracy.
π» Community Contribution
The project thrives on community involvement. Contributors are encouraged to report any issues on GitHub and submit pull requests to introduce new features or improvements.
π Starting Points for Contribution
To contribute to the Secrets Patterns Database, here are some potential avenues:
- Developing support for severity levels in patterns.
- Improving classification by type or tagging.
- Extending the database's compatibility with other detection tools.
Using the Database
For those using Trufflehog or Gitleaks, the database supports easy integration with the following commands:
- Trufflehog v2:
./convert-rules.py --db ../db/rules-stable.yml --type trufflehog - Gitleaks:
./convert-rules.py --db ../db/rules-stable.yml --type gitleaks
Moreover, users have the option to export patterns using specific filename extensions relevant to each tool (e.g., .toml for Gitleaks, .json for Trufflehog).
π Licensing and Credits
The Secrets Patterns Database operates under the Creative Commons Attribution 4.0 International License, which allows for sharing and adaptation with proper attribution.
The project is spearheaded by Mazin Ahmed, who is a prolific figure in the cybersecurity community. You can learn more about his work through his website or connect with him on Twitter and LinkedIn.
This project is in its Beta phase, implying there are numerous opportunities for refinement and advancement. Contributors and users alike are invited to participate actively in the evolution of this valuable resource.