This tool builds a clean, visual-ready hierarchy of links from any website you point it at. It digs through pages, maps relationships, and helps you understand a site's structure without the guesswork. It’s built for anyone who needs fast, reliable link discovery at scale.
Created by Bitbash, built to showcase our approach to Scraping and Automation!
If you are looking for Get Urls Pro you've just found your team — Let’s Chat. 👆👆
This project crawls a website starting from a single URL and produces a structured map of its internal links. It solves the common challenge of understanding how pages connect, especially on larger sites where navigation paths aren’t always obvious. It’s ideal for developers, SEO analysts, data engineers, and anyone who needs deeper insights into a site's link architecture.
- Recursively follows links and builds a parent–child hierarchy.
- Lets you limit crawl depth and children per link to stay efficient.
- Supports both lightweight HTML parsing and full browser rendering.
- Filters out unwanted file types to keep output clean.
- Optionally restricts crawling to the same domain for focused analysis.
| Feature | Description |
|---|---|
| Configurable crawling depth | Control how deep the crawler follows links. |
| Dual parsing engines | Choose between fast HTML parsing or browser-powered crawling for JS-heavy sites. |
| Link filtering | Exclude specific extensions for cleaner output. |
| Smart deduplication | Avoids repeated URLs unless explicitly allowed. |
| Domain control | Stay within the same domain to keep results relevant. |
| Proxy support | Add another layer of flexibility and stability for distributed crawling. |
| Field Name | Field Description |
|---|---|
| url | The exact page address discovered during the crawl. |
| name | The detected label or title of the link when available. |
| query | Query parameters extracted from the URL. |
| depth | Numerical level representing how far the link is from the starting point. |
| parentUrl | The URL from which this link was found. |
Example:
[
{
"url": "https://jamesclear.com/five-step-creative-process",
"name": null,
"query": "",
"depth": 0,
"parentUrl": null
},
{
"url": "https://jamesclear.com/",
"name": null,
"query": "",
"depth": 1,
"parentUrl": "https://jamesclear.com/five-step-creative-process"
},
{
"url": "https://jamesclear.com/books",
"name": "Books",
"query": "",
"depth": 1,
"parentUrl": "https://jamesclear.com/five-step-creative-process"
},
{
"url": "https://jamesclear.com/articles",
"name": "Articles",
"query": "",
"depth": 1,
"parentUrl": "https://jamesclear.com/five-step-creative-process"
},
{
"url": "https://jamesclear.com/3-2-1",
"name": "Newsletter",
"query": "",
"depth": 2,
"parentUrl": "https://jamesclear.com/"
},
{
"url": "https://jamesclear.com/events?g=4",
"name": "Speaking",
"query": "g=4",
"depth": 2,
"parentUrl": "https://jamesclear.com/"
}
]
Get Urls Pro/
├── src/
│ ├── runner.py
│ ├── crawler/
│ │ ├── html_parser.py
│ │ ├── selenium_engine.py
│ │ └── link_utils.py
│ ├── outputs/
│ │ └── json_exporter.py
│ └── config/
│ └── settings.example.json
├── data/
│ ├── inputs.sample.json
│ └── sample_output.json
├── requirements.txt
└── README.md
- Researchers map out complex websites to understand information flow and content structure more clearly.
- SEO teams analyze internal linking patterns to identify gaps and improve crawlability.
- Developers audit large web projects to verify link integrity and navigation hierarchy.
- Data analysts gather structured link datasets for downstream processing or visualization.
- Content strategists detect hidden or orphaned pages to refine content architecture.
Does this scraper handle JavaScript-heavy sites? Yes—switching to the Selenium mode enables full browser rendering, which captures dynamic links standard parsers would miss.
How do I prevent too many pages from being crawled?
Set limits using maxDepth and maxChildrenPerLink, which let you shape the crawl to fit your needs.
Can I avoid crawling assets or file downloads? Absolutely. Add extensions like pdf, jpg, css, or others to the ignore list to keep your output focused.
What if I need URLs outside the starting domain? You can toggle the domain restriction setting to explore external links as well.
Primary Metric: Handles an average of 120–180 pages per minute using standard parsing, depending on page weight and structure.
Reliability Metric: Maintains a typical stability level above 98% during multi-depth crawls even on moderately dynamic sites.
Efficiency Metric: Optimized queueing ensures minimal redundant requests and predictable memory usage during long runs.
Quality Metric: Produces link hierarchies with consistently high completeness, often capturing more than 95% of reachable internal paths on medium-sized websites.
|
"Bitbash is a top-tier automation partner, innovative, reliable, and dedicated to delivering real results every time." Nathan Pennington
|
"Bitbash delivers outstanding quality, speed, and professionalism, truly a team you can rely on." Eliza
|
"Exceptional results, clear communication, and flawless delivery. Syed
|