Scavenger - OSINT Bot - 3.0

bot in action (currently not running)

Just the code of my OSINT bot searching for sensitive data leaks on paste sites. email:password combinations are always collected.

It detects sensitive information based on the file searchterms.txt and some regexes in classes/utility.py.

Search terms:

mysqli_connect(
BEGIN RSA PRIVATE KEY
-----BEGIN
The name of the database for WordPress
Return-Path:
insert into
INSERT INTO
.onion
github_pat_
AIza
sk_live_
rk_live_
client_secret
jdbc:
mongodb://
mongodb+srv://
mysql://
postgresql://
redis://
authorization: basic

Search terms can be customized. You can learn more about it in the configuration section.

• https://jakecreps.com/2019/05/08/osint-collection-tools-for-pastebin/
• https://jakecreps.com/2019/01/08/scavenger/
• https://youtu.be/VCwiZ2dh17Q?t=51 (the bot is mentioned here)

For pastebin.com the bot has two modes:

• looking for sensitive data in the archive via scraping
• looking for sensitive data by tracking users who publish leaks

For pastes.io:

• tries to guess pastes and looks for sensitive information
• do not forget to add your api key under configs/pastesio.json and set if you have a free or pro tier

For GitHub Gist:

• scrapes latest gists and looks for sensitive information

Additional features:

• customizable search terms

0. Execute setup.sh to install are needed non standard python libraries
1. Delete the README.md files in every subfolder as they are only placeholders
2. The bot searches for email:password combinations and other kinds sensitive data by default. If you want to add more search terms edit the configs/searchterms.txt file or use the -2 switch in the control script Default configs/searchterms.txt configuration:

If you want to add other search terms just add them to file line by line. You know a useful search terms which is missing here? Tell me! :-) 3. For the user tracking module of pastebin.com you need to add the target users line by line to the configs/users.txt file.

Program help:

$ python3 scavenger.py -h
╭── Scavenger 2.0 ─────────────────────────────────╮
│ pastebin credential-leak crawler                 │
│                                                  │
│   -0  pastebin archive scrape module             │
│   -1  pastebin user track module                 │
│   -2  pastes.io random-ID scraper                │
│   -3  GitHub gist scraper                        │
│   -4  edit search terms                          │
│   -5  edit tracked users                         │
│                                                  │
│ python3 scavenger.py -0 -1  (combine flags)      │
╰──────────────────────────────────────────────────╯
usage: scavenger.py [-h] [-0] [-1] [-2] [-3] [-4] [-5]

control script

options:
  -h, --help          show this help message and exit
  -0, --pbincom       Activate pastebin.com archive scraping module
  -1, --pbincomTrack  Activate pastebin.com user track module
  -2, --pastesio      Activate pastes.io random-ID scraper
  -3, --githubgist    Activate GitHub gist scraper
  -4, --editsearch    Edit search terms file for additional search terms (email:password combinations will always be searched)
  -5, --editusers     Edit user file of the pastebin.com user track module

example usage: python3 scavenger.py -0 -1

Crawled pastes are stored at different locations depending on their status.

• Paste crawled but nothing was detected -> data/raw_pastes
• Paste crawled and an email:password combination was detected -> data/raw_pastes and data/files_with_passwords
• Paste crawled and other sensitive data was detected -> data/raw_pastes and data/otherSensitivePastes

Pastes get stored in data/raw_pastes until they reach a limit of 48000 files. Once there are more then 48000 pastes they get ziped and moved to the archive folder.

Start the pastebin.com archive scraping module

$ python3 scavenger.py -0

Start pastebin.com user tracking module

$ python3 scavenger.py -1

When starting one of these modules, a tmux session with the running module is created in the background.

List tmux sessions

$ tmux ls
pastebincomArchive: 1 windows (created Sun Apr 14 06:33:32 2021) [204x58]
pastebincomTrack: 1 windows (created Sun Apr 14 06:33:32 2021) [204x58]

Interact with a tmux session example

$ tmux a -t pastebincomArchive
$ tmux a -t pastebincomTrack

To detach from a session hit STRG+b d.

If you want to start a module without using the control software you can do this by calling them directly.

Pastebin.com archive scraper

$ python3 pbincomArchiveScrape.py

Pastebin.com user tracker

$ python3 pbincomTrackUser.py

If you miss anything and want me to add features or make changes, just let me know via Twitter or GitHub issue :-)