There is probably a neat wget oneliner that could crawl everything on the open web. The real challenge is how to index all the information. That might be a neat Perl oneliner.