2 releases
| 0.1.1 | Aug 11, 2025 |
|---|---|
| 0.1.0 | Aug 9, 2025 |
#116 in #web-crawler
34 downloads per month
Used in omnivore-cli
180KB
4.5K
SLoC
omnivore-core
Core crawler and knowledge graph engine for Omnivore.
- Fast async crawling with Tokio
- Respectful politeness and robots.txt support
- HTML parsing and extraction utilities
- Pluggable storage and graph building
License
MIT OR Apache-2.0
Dependencies
~58–85MB
~1.5M SLoC