enh: crawler by thiswillbeyourgithub · Pull Request #269 · mwmbl/mwmbl

thiswillbeyourgithub · 2025-06-18T17:14:50Z

fix: import HTTPException from ninja to resolve undefined name errors
refactor: move crawler configuration to env_vars with environment variable support
docs: add initial README for mwmbl crawler
feat: add configurable rate limiting for crawler to reduce request bursting
feat: import CRAWLER_WORKERS from env_vars and simplify worker configuration
docs: add CRAWLER_WORKERS env var description to crawler README
docs: Add comprehensive crawler architecture details to README
remove burst limiting
enh: print error reason in the logger
feat: add configurable threads for crawl_batch with env var
docs: clarify CRAWLER_WORKERS documentation to specify OS process spawning
enh: add check for crawler worker value
docs: clarify crawler process and thread architecture in README
minor: use logger instead of print for the results
feat: add rate limiting with configurable delay and random fuzz
docs: Add docstrings to process_batch and run_indexing, update README with workflow details
docs: clarify DOMAIN_GROUPS purpose in README for domain authority scoring
feat: add crash tracking for index process with 5-crash-per-hour threshold
chore: replace pip with uv for faster Python package management
feat: add crawler version tracking to batch uploads
refactor: move hardcoded constants from env vars to respective files
minor: more compact log line for crawling
feat: import REDIS_URL from env_vars and use in Redis connection
minor: var rename
feat: add retry mechanism with exponential backoff for API requests
bump python version to 3.11.12
upgrade version of pybloomfiltermmap3
fix wrong requirements.txt
bump python version to 3.11.12
enh: add check for redis connection very early
fix: botched merge forgot some lines
ignore aider files
ignore redis_data folder
minor: better presentation of the env var in the docker compose
ran poetry.lock
fix: duplicate crawler version + better declaration
feat: Add configurable contact info to user-agent for responsible crawling

Okay so I sort of branched from the previous PR (#242) then did a first visual pass to fix what I botched during the merges (forgot a few lines here and there) then ran poetry.lock, fixed a few things here and there, added a configurable contact info and made the PR.

I don't have the time yet to give it a go, there might be fatal errors somewhere but I prefered to do the draft asap to give you more time to review. If there are issues they are small.

The main difference with #242 in the end is the lack of LMDB, not using black, not using loguru.

Feedback welcome

…iable support

…rsting

…uration

Signed-off-by: thiswillbeyourgithub <26625900+thiswillbeyourgithub@users.noreply.github.com>

…pawning

Signed-off-by: thiswillbeyourgithub <26625900+thiswillbeyourgithub@users.noreply.github.com>

…ADME with workflow details

…scoring

…shold

because loguru takes care of the tracing Signed-off-by: thiswillbeyourgithub <26625900+thiswillbeyourgithub@users.noreply.github.com>

Signed-off-by: thiswillbeyourgithub <26625900+thiswillbeyourgithub@users.noreply.github.com>

…wling

Signed-off-by: thiswillbeyourgithub <26625900+thiswillbeyourgithub@users.noreply.github.com>

thiswillbeyourgithub · 2025-06-18T17:19:00Z

I included the what is in #264 and addressed #265

thiswillbeyourgithub · 2025-06-18T17:30:00Z

Update: pytest working ( DJANGO_SETTINGS_MODULE="mwmbl.settings_dev" pytest), the container builds fine, up is fine so far. I'm making as "ready for review" but will let it run overnight

TechnologyClassroom · 2025-06-18T17:38:09Z

I would recommend a default CRAWL_DELAY_SECONDS value of of 2.0 instead of 0.0 .

thiswillbeyourgithub · 2025-06-18T17:45:26Z

But the delay is for all URLs no? I mean it makes sense to not burst on a single domain but across all domains it's fine right?

Also I initially set it to 0.0 to make it "opt in" because that was my idea and I didn't want to trouble the owner too much.

Signed-off-by: thiswillbeyourgithub <26625900+thiswillbeyourgithub@users.noreply.github.com>

daoudclarke · 2025-06-22T15:09:09Z

The delay should not be necessary since we only use a single URL from each domain in each batch of URLs.

daoudclarke · 2025-06-22T15:14:46Z

If we're upgrading Python, is there a reason to use 3.11 instead of 3.13?

thiswillbeyourgithub · 2025-06-22T19:08:46Z

The delay should not be necessary since we only use a single URL from each domain in each batch of URLs.

Keeping it at 0 by default then. But th idea was to put a cap on the strain on my end, not on the remote server.

If we're upgrading Python, is there a reason to use 3.11 instead of 3.13?

Iirc it was because I had trouble installing a dep on later versions but let me revisit that

thiswillbeyourgithub · 2025-06-22T20:38:52Z

I tried with 3.13.3 but have issues with pybloomfilter (!), numpy, pandas etc. I'm sure it's possible without too much trouble but I'll leave it up to you as I'm not familiar with the tradeoffs involved.

daoudclarke · 2025-06-22T21:16:23Z

Ok that's fair enough! Good to know, thanks for checking.

I will need to check that the main site works on before we merge this but it's looks like great work.

daoudclarke · 2025-06-23T20:42:11Z

There are a few things that could be improved here - I'll make some fixes on a branch off your PR and we can merge from there.

thiswillbeyourgithub · 2025-06-23T21:02:47Z

Alright!

daoudclarke · 2025-06-23T21:29:59Z

Added a commit in #274 - will test out over the next few days, including deploying to beta.mwmbl.org and running against that, then hopefully we can merge 🎉

thiswillbeyourgithub added 30 commits May 30, 2025 14:02

fix: import HTTPException from ninja to resolve undefined name errors

ccf25cf

refactor: move crawler configuration to env_vars with environment var…

fde29c4

…iable support

docs: add initial README for mwmbl crawler

5d25317

feat: add configurable rate limiting for crawler to reduce request bu…

0de9a6a

…rsting

feat: import CRAWLER_WORKERS from env_vars and simplify worker config…

661879f

…uration

docs: add CRAWLER_WORKERS env var description to crawler README

706c8d6

docs: Add comprehensive crawler architecture details to README

bff1ba5

remove burst limiting

0dcdd61

Signed-off-by: thiswillbeyourgithub <26625900+thiswillbeyourgithub@users.noreply.github.com>

enh: print error reason in the logger

2343671

Signed-off-by: thiswillbeyourgithub <26625900+thiswillbeyourgithub@users.noreply.github.com>

feat: add configurable threads for crawl_batch with env var

6d2424a

docs: clarify CRAWLER_WORKERS documentation to specify OS process s…

81c4615

…pawning

enh: add check for crawler worker value

c1dfd0f

Signed-off-by: thiswillbeyourgithub <26625900+thiswillbeyourgithub@users.noreply.github.com>

docs: clarify crawler process and thread architecture in README

63ff2b2

minor: use logger instead of print for the results

2e23d3f

Signed-off-by: thiswillbeyourgithub <26625900+thiswillbeyourgithub@users.noreply.github.com>

feat: add rate limiting with configurable delay and random fuzz

9188909

docs: Add docstrings to process_batch and run_indexing, update RE…

7420378

…ADME with workflow details

docs: clarify DOMAIN_GROUPS purpose in README for domain authority …

e6c14e6

…scoring

feat: add crash tracking for index process with 5-crash-per-hour thre…

25d1c79

…shold

chore: replace pip with uv for faster Python package management

bd75726

feat: add crawler version tracking to batch uploads

bc46c41

refactor: move hardcoded constants from env vars to respective files

1e6b874

Merge branch 'main' into enh_crawler

45dca75

minor: more compact log line for crawling

8259226

because loguru takes care of the tracing Signed-off-by: thiswillbeyourgithub <26625900+thiswillbeyourgithub@users.noreply.github.com>

feat: import REDIS_URL from env_vars and use in Redis connection

62ded03

minor: var rename

c311981

Signed-off-by: thiswillbeyourgithub <26625900+thiswillbeyourgithub@users.noreply.github.com>

feat: add retry mechanism with exponential backoff for API requests

562a4fe

bump python version to 3.11.12

f7fa56b

Signed-off-by: thiswillbeyourgithub <26625900+thiswillbeyourgithub@users.noreply.github.com>

upgrade version of pybloomfiltermmap3

3d033df

Signed-off-by: thiswillbeyourgithub <26625900+thiswillbeyourgithub@users.noreply.github.com>

fix wrong requirements.txt

cd47aa8

Signed-off-by: thiswillbeyourgithub <26625900+thiswillbeyourgithub@users.noreply.github.com>

bump python version to 3.11.12

b8431e7

Signed-off-by: thiswillbeyourgithub <26625900+thiswillbeyourgithub@users.noreply.github.com>

thiswillbeyourgithub added 9 commits June 18, 2025 18:57

ignore redis_data folder

86ce0dd

Signed-off-by: thiswillbeyourgithub <26625900+thiswillbeyourgithub@users.noreply.github.com>

minor: better presentation of the env var in the docker compose

a83f774

Signed-off-by: thiswillbeyourgithub <26625900+thiswillbeyourgithub@users.noreply.github.com>

ran poetry.lock

b2ff86a

Signed-off-by: thiswillbeyourgithub <26625900+thiswillbeyourgithub@users.noreply.github.com>

fix: duplicate crawler version + better declaration

cc4f5c2

Signed-off-by: thiswillbeyourgithub <26625900+thiswillbeyourgithub@users.noreply.github.com>

feat: Add configurable contact info to user-agent for responsible cra…

0d7cfad

…wling

add check for api key

9a9ceaa

Signed-off-by: thiswillbeyourgithub <26625900+thiswillbeyourgithub@users.noreply.github.com>

set default workers to 2

c34b058

Signed-off-by: thiswillbeyourgithub <26625900+thiswillbeyourgithub@users.noreply.github.com>

store persistent root folder

2fe4e8f

Signed-off-by: thiswillbeyourgithub <26625900+thiswillbeyourgithub@users.noreply.github.com>

style for the api key

3ea0faa

Signed-off-by: thiswillbeyourgithub <26625900+thiswillbeyourgithub@users.noreply.github.com>

Merge branch 'main' into enh_crawler_chunk_1

d2f4415

thiswillbeyourgithub mentioned this pull request Jun 18, 2025

Feature request: Allow users to configure the user-agent with a link to their instance or provide contact information #265

Open

thiswillbeyourgithub marked this pull request as ready for review June 18, 2025 17:30

ignore all data folders

4a1661c

Signed-off-by: thiswillbeyourgithub <26625900+thiswillbeyourgithub@users.noreply.github.com>

thiswillbeyourgithub mentioned this pull request Jun 19, 2025

enh: crawler + lmdb + orjson #270

Closed

Merge branch 'main' into enh_crawler_chunk_1

ba2c580

thiswillbeyourgithub mentioned this pull request Jun 22, 2025

FR: upgrade docker to latest python (3.13.3) #273

Open

daoudclarke mentioned this pull request Jun 23, 2025

Enh crawler fixes #274

Merged

thiswillbeyourgithub closed this Jul 6, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

enh: crawler#269

enh: crawler#269
thiswillbeyourgithub wants to merge 46 commits into
mwmbl:mainfrom
thiswillbeyourgithub:enh_crawler_chunk_1

thiswillbeyourgithub commented Jun 18, 2025

Uh oh!

thiswillbeyourgithub commented Jun 18, 2025

Uh oh!

thiswillbeyourgithub commented Jun 18, 2025

Uh oh!

TechnologyClassroom commented Jun 18, 2025

Uh oh!

thiswillbeyourgithub commented Jun 18, 2025

Uh oh!

daoudclarke commented Jun 22, 2025

Uh oh!

daoudclarke commented Jun 22, 2025

Uh oh!

thiswillbeyourgithub commented Jun 22, 2025

Uh oh!

thiswillbeyourgithub commented Jun 22, 2025

Uh oh!

daoudclarke commented Jun 22, 2025

Uh oh!

daoudclarke commented Jun 23, 2025

Uh oh!

thiswillbeyourgithub commented Jun 23, 2025

Uh oh!

daoudclarke commented Jun 23, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

thiswillbeyourgithub commented Jun 18, 2025

Uh oh!

thiswillbeyourgithub commented Jun 18, 2025

Uh oh!

thiswillbeyourgithub commented Jun 18, 2025

Uh oh!

TechnologyClassroom commented Jun 18, 2025

Uh oh!

thiswillbeyourgithub commented Jun 18, 2025

Uh oh!

daoudclarke commented Jun 22, 2025

Uh oh!

daoudclarke commented Jun 22, 2025

Uh oh!

thiswillbeyourgithub commented Jun 22, 2025

Uh oh!

thiswillbeyourgithub commented Jun 22, 2025

Uh oh!

daoudclarke commented Jun 22, 2025

Uh oh!

daoudclarke commented Jun 23, 2025

Uh oh!

thiswillbeyourgithub commented Jun 23, 2025

Uh oh!

daoudclarke commented Jun 23, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants