Mr. Crawley in Termux – Website Crawling & OSINT Tool

Mr. Crawley is a simple web crawler and OSINT tool that helps collect publicly available information from websites. It can crawl web pages, discover links, and gather useful data for website analysis and reconnaissance. You can easily install and use Mr. Crawley in Termux using a few simple commands.

Here’s what you can do with Mr. Crawley in Termux:

Crawl websites.
Discover URLs and web pages.
Analyze website structure.
Save crawl results to a file.
Use custom headers and User-Agent.
Configure threads and request delay.

Install Mr. Crawley in Termux

Below are the simple commands to install Mr. Crawley in Termux. Copy and run each command one by one to install the tool.

Update Termux packages.

pkg update && pkg upgrade -y

pkg update && pkg upgrade -y

Install Git and Python.

pkg install git python -y

pkg install git python -y

Install the required Python modules.

pip install requests beautifulsoup4 termcolor

pip install requests beautifulsoup4 termcolor

Clone the Mr. Crawley repository.

git clone https://github.com/t0nyc23/mr_crawley.git

git clone https://github.com/t0nyc23/mr_crawley.git

Move into the project directory.

cd mr_crawley

cd mr_crawley

Run the tool.

python crawley.py

python crawley.py

Mr. Crawley is now installed and ready to use.

Use Mr. Crawley in Termux

After installing Mr. Crawley, you can use different commands to crawl websites and customize the scan.

Display the help menu.

python crawley.py -h

python crawley.py -h

Crawl a target website.

python crawley.py -u https://example.com

python crawley.py -u https://example.com

Use custom HTTP headers.

python crawley.py -u https://example.com -H "Authorization: Bearer TOKEN"

python crawley.py -u https://example.com -H "Authorization: Bearer TOKEN"

Set the number of threads.

python crawley.py -u https://example.com -t 20

python crawley.py -u https://example.com -t 20

Save the results to a file.

python crawley.py -u https://example.com -o results.txt

python crawley.py -u https://example.com -o results.txt

Use a custom User-Agent.

python crawley.py -u https://example.com -a "MyUA/0.1"

python crawley.py -u https://example.com -a "MyUA/0.1"

Add a delay between requests.

python crawley.py -u https://example.com -s 1000

python crawley.py -u https://example.com -s 1000

Use an HTTP proxy.

python crawley.py -u https://example.com -p http://127.0.0.1:8080

python crawley.py -u https://example.com -p http://127.0.0.1:8080

Enable verbose mode.

python crawley.py -u https://example.com -d

python crawley.py -u https://example.com -d

Enter the target website URL and run the command. Mr. Crawley will start crawling the website and display the collected information.

End Note

Mr. Crawley is a simple web crawler that helps you collect publicly available information from websites. It is easy to install, easy to use, and a good tool for learning website crawling and basic OSINT in Termux.