In order to index the web properly, corbia spider only