Currently, the code in DomainNames that tries to find the "paid level domain" of the input domain treats ae.com, gq.com, and hm.com as if they were domains within the Arab Emirates, Equitorial Guinea, and Heard & McDonald Islands (respectively). These are instead domains within the United States.
The problem stems from the (over)use of DomainNames.CC_ALWAYS_TLDS. The purpose of this set of country codes is to list those that should be treated as top-level domains after a global top-level domain prefix is added. For example, com.de is considered a paid level domain, because de is a member of CC_ALWAYS_TLDS, whereas com.jp is itself a top-level domain, making honda.com.jp a paid-level domain.
Unfortunately, this set is also (inappropriately) being used to list those country codes that should be treated as top-level domains after a global top-level domain suffix is added. For example, de.com is a German top-level domain (see http://www.de.com/ for details), whereas (as mentioned before) ae.com, gq.com, and hm.com are paid level domains in the United States.
We need to add a new set of country codes that includes de which DomainNames.getPld can check to determine what to do with input domains like de.com, ae.com, gq.com and hm.com. I'm not exactly sure how to determine what other country codes besides de should be put into the new set. Unfortunately, the source document I used to build the constants in DomainNames.java (https://github.com/bixo/bixo/blob/master/doc/effective_tld_names.dat) doesn't specify which CCs should be treated this way.
Ken suggests trying out the equivalent domain processing support that currently exists in crawler-commons.
Currently, the code in DomainNames that tries to find the "paid level domain" of the input domain treats ae.com, gq.com, and hm.com as if they were domains within the Arab Emirates, Equitorial Guinea, and Heard & McDonald Islands (respectively). These are instead domains within the United States.
The problem stems from the (over)use of DomainNames.CC_ALWAYS_TLDS. The purpose of this set of country codes is to list those that should be treated as top-level domains after a global top-level domain prefix is added. For example, com.de is considered a paid level domain, because de is a member of CC_ALWAYS_TLDS, whereas com.jp is itself a top-level domain, making honda.com.jp a paid-level domain.
Unfortunately, this set is also (inappropriately) being used to list those country codes that should be treated as top-level domains after a global top-level domain suffix is added. For example, de.com is a German top-level domain (see http://www.de.com/ for details), whereas (as mentioned before) ae.com, gq.com, and hm.com are paid level domains in the United States.
We need to add a new set of country codes that includes de which DomainNames.getPld can check to determine what to do with input domains like de.com, ae.com, gq.com and hm.com. I'm not exactly sure how to determine what other country codes besides de should be put into the new set. Unfortunately, the source document I used to build the constants in DomainNames.java (https://github.com/bixo/bixo/blob/master/doc/effective_tld_names.dat) doesn't specify which CCs should be treated this way.
Ken suggests trying out the equivalent domain processing support that currently exists in crawler-commons.