I am trying to implement a pausable/resumable crawler. From the implementation that I have seen I believe that currently it is not possible to stop a queue after calling Run until it is empty or has 0 active requests as can be seen in https://github.com/gocolly/colly/blob/v2.1.0/queue/queue.go#L150.

I could implement this at the collector level by avoiding to call AddURL when needed but the crawler would keep going if the queue got big.

Has someone been able to implement this? Maybe I am missing something.

If not, is this a desirable behavior? Maybe it could be worth implementing