Setting up the code
-------------------

The project depends on Nutch v1.0 (http://lucene.apache.org/nutch/) and Hadoop 
Core v0.19 (http://hadoop.apache.org/).  Please, download and include the 
corresponding jars in your classpath.

Running the code
----------------

1. Data preprocessing

The first step is to extract and process data in a nutch database and handle 
incremental updates.  The pre-processing stage is split in the following steps:

 a. Extract pages from a nutch database (babel.prep.extract.NutchPageExtractor).
    Versions of each page fetched by multiple nutch crawls and containing parse
    and content metadata along with parsed content are aggregated and collected
    into a page dataset.

 b. Merge two existing page datasets (babel.prep.merge.PageMerger).

 c. Collect page language information (babel.prep.langid.LangIdentifier). Page
    content language is identified for pages in a dataset with missing language
    metadata.

 d. Generate per-language dataset (babel.prep.corpus.CorpusGenerator).  A 
    dataset is split per-language and (optionally) saved as a set of XML 
    documents.

2. More coming...