How to Find Secrets that are Accidentally Committed to Version Control System (Git)

Meme Credit: https://twitter.com/DZoneInc/status/1361420207793659904

Sensitive secrets being exposed accidentally or hard-coded to the backed code can be a significant concern for the DevSecOps team, of any organization. Even though the repository of an organization is private, it takes less than a minute for an insider- to get hold of sensitive secrets. Obtaining these secrets gives them the power to access or make modifications to the current infrastructure. Secrets that can be exploded to the internet include Slack tokens, Database credentials, cloud access, secret keys and developer tokens, etc.

Why Secrets in Git Repository are a big problem 🔐

When a secret makes its way to a Git repository, it stays there forever, sitting in one or more of your commits, waiting to be found and used against you. Developers often forget that Git-based repository history is never deleted. Not only do the secrets sit in a Git server, but every clone and fork save this secret on different machines without the clone or fork creator being aware of it.

How to Improve Organization Security 💡

Many tools in the market can scan your repository, or commits before pushing, to ensure that no secrets are stored or pushed to the remote origin. Here are some ways you can improve your organization's security :

1. Configure a tool to actively scan your repository all the time for files that contain secrets.
2. Configure on each machine that pushes code to the remote repository, for this you can use git-secret.
3. You can integrate a tool in the CI/CD pipelines. You can do it using GitHub Actions, Jenkins or Travis CI.

How its Work? 🔨

How do scanning tools work? Let's find this out. I will scan my local repository with the Git-Secrets tool first, and then I will scan the repository with the TruffleHog tool.

Git-Secrets 🧰

Git Secrets is a tool released by AWS Labs that will scan commits and commit messages. It lets you scan a file or an older one recursively to look for secrets. This work great for trying to find AWS, Azure, or Google Cloud secret key in your repository. Let's see how it works.

Git-Secrets Installation & Working

A full installing guide is available on the Git-Secrets Repository and you can follow it accordingly.

git clone https://github.com/awslabs/git-secrets.git
cd git-secrets
make install

Open up a terminal window and perform the following steps. You have to do these steps for every repo you want to use git — secret with.

cd /path/to/my/repo
git init 
git secrets --install
git secrets --register-aws

So, I am using the AWS plugin, you can also use multiple cloud plugins in your repository

git secrets -register-azure

git secrets -register-aws

git secrets — register-gcp

git secrets --scan 
git secrets --scan-history 
git secrets --scan /path/to/file

For more information and documentation on git-secrets → README.
Now, on to the more popularly used scanning tool — truffle hog. This GitHub repository scanner will look into your commit history and spot anything that looks like a password or confidential information using regex and entropy.

Truffle hog 🧰

The second tool which I like the most is the truffle Hog. TuffleHog will scan the entire commit history of each branch and check each diff from each commit, and check for secrets. This is both by regex and by entropy. For entropy checks, truffle Hog will evaluate the Shannon entropy for both base64 char set, and hexadecimal char set for every blob of text greater than 20 characters comprised of these character sets in each diff. If at any point a high entropy string > 20 characters is detected, it will print to the screen.

Truffle Hog Installation & Working

Complete installation documentation is available on the Truffle Hog repository README and you can follow it accordingly. I installed the Truffle Hog using Pip Package manage on Ubuntu Machine using these commands

sudo apt-get install python3-pip
pip install truffleHog

To check whether TruffleHog is installed or not, try this command

trufflehog git --help

To run it against a repository to find the secrets, you can use the following command

trufflehog --regex --entropy=False </path/to/directory/of/repo>

To scan a Git Repository, try these commands

trufflehog --regex --entropy=False https://github.com/dxa4481/truffleHog.git

trufflehog git https://github.com/trufflesecurity/trufflehog.git

Other Scanning Tools 🧰

There are other available scanning tools, which you can easily implement in your CI/CD Pipelines, so scanning can be done in automatic ways. Here are the results :

Conclusion 🚀

So I just started my journey with DevSecOps, and whatever I learn, I will try my best to help my community, and suggest best practices to secure the infrastructure from Evil Eyes. Any suggestions, improvements, Collaboration, or opportunities please feel free to drop a message on LinkedIn. I will be very happy to connect with you and share some cool ideas :)

https://www.linkedin.com/in/adilshehzad7/