V0.1.0OPEN SOURCERUST

UNIVERSAL WEB CRAWLER
AND KNOWLEDGE GRAPH

High-performance, parallel web crawler and knowledge graph system built in Rust. Open source and community-driven. Extract, analyze, and graph data from the web at scale.

INSTALLDOCUMENTATIONSOURCE
FEATURES
>>
Fast

Process 10,000+ pages per minute with Tokio async runtime

[]
Respectful

Built-in robots.txt compliance and rate limiting

<>
Graphs

Build entity-relationship graphs automatically

##
Smart

AI-powered entity recognition and classification

::
Storage

RocksDB, PostgreSQL, and graph databases

()
Browser

Handle JavaScript-heavy sites with automation

INSTALL
HOMEBREW
BASH
001brew tap Pranav-Karra-3301/omnivore
002brew install omnivore
DOCKER
BASH
001docker run --rm -it omnivore:latest omnivore --help
FROM SOURCE
BASH
001git clone https://github.com/Pranav-Karra-3301/omnivore.git
002cd omnivore
003cargo install --path omnivore-cli --force
FIRST CRAWL
BASH
001omnivore crawl https://example.com --workers 5 --depth 3
BUILD GRAPH
BASH
001omnivore graph results.json --output knowledge-graph.db
FULL GUIDE