Use robots.txt parsing support now in crawler-commons #42

The robots.txt parsing code was recently ported to crawler-commons.

Bixo should be modified to leverage crawler-commons, though we need a way to get crawler-commons builds into a Maven repo first.

This issue is the same as https://issues.apache.org/jira/browse/NUTCH-1008