…Solr or Elasticsearch . Grub was an open source distributed search crawler that Wikia Search used to crawl the web. Heritrix is the Internet Archive 's archival-quality crawler, designed for archiving periodic snapshots of a large portion of the Web. It was written in Java . ht:/…