Solr is a great search engine. It won’t help you with the crawling, but if you manage to get the data into Solr you have a come far.