open source, distributed, restful crawler engine

this is a project base on scrapy,we reuse most of scrapy code in crawler part

I design my distributed architecture by imitating elasticsearch. it spire me to do a engine for distributed crawler

• cluster
• node
• webservice
• schedule
• crawl
• engine

you can see it in issues

it will be in wiki. so please look forward to it