This project is a BBC news webpage crawler and parser, used to collect corpus. We need about 10,000 documents from BBC News.

Process:

1. Using nekohtml tagsoup to parse HTML into a DOM
2. Filter unneccesary DOM nodes
3. maintian a crawled webpages urls

Reference:

1. nekohtml
2. hadoop
3. nutch

Notice: I used it to crawl 10,000 bbc news webpages. This project can also be used to collect data from other web pages. It is a crawler and a parser. It maintains a linkDB for web crawling. More information please contact: xinghuangxu@gmail.com