Skip to content

Conversation

@WGH-
Copy link
Collaborator

@WGH- WGH- commented Oct 9, 2020

This allows one to pass context.Context to colly.Collector, which in turn will passed to stdlib's http.Request.

This enables colly scraping to be cancelled cleanly at any time.

See #255 #240

@WGH-
Copy link
Collaborator Author

WGH- commented Oct 9, 2020

Given that the word Context collides with already existing type in colly, I couldn't think of anything better than calling the CollectorOption function StdlibContext. Better ideas are more than welcome.

This allows one to pass context.Context to colly.Collector,
which in turn will passed to stdlib's http.Request.

This enables colly scraping to be cancelled cleanly at any time.
@WGH- WGH- force-pushed the collector-context branch from fafdcbc to b02f0ab Compare October 9, 2020 18:20
Copy link
Member

@asciimoo asciimoo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great, thanks! I think StdlibContext is perfectly fine

@asciimoo asciimoo merged commit 8252c34 into gocolly:master Oct 13, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants