WebWizardtoAsk Lemmy@lemmy.world·5 months agoWhat's your favourite search engine?message-squarearrow-up154arrow-down14message-square59fedilink
arrow-up150arrow-down1message-squareWhat's your favourite search engine?WebWizardtoAsk Lemmy@lemmy.world·5 months agomessage-square59fedilink
minus-squareyeehawarrow-up5arrow-down0·5 months agolinkfedilinkHow does that work, does it go out and start indexing the internet for you?
minus-squarewhalerossarrow-up5arrow-down0·5 months agolinkfedilinkIt’s a meta search engine. It queries other search engines and compiles you results.
minus-squareludarrow-up3arrow-down0·5 months agolinkfedilinkThat’s a shame, It would be more interesting to run your own crawlers. (Yes I realise that would be computationally intense and hard, but it would nonetheless be cool)
minus-squarewhalerossarrow-up3arrow-down0·5 months agolinkfedilinkThere is probably a neat wget oneliner that could crawl everything on the open web. The real challenge is how to index all the information. That might be a neat Perl oneliner.
My own self-hosted SearXNG.
How does that work, does it go out and start indexing the internet for you?
It’s a meta search engine. It queries other search engines and compiles you results.
That’s a shame, It would be more interesting to run your own crawlers.
(Yes I realise that would be computationally intense and hard, but it would nonetheless be cool)
There is probably a neat wget oneliner that could crawl everything on the open web. The real challenge is how to index all the information. That might be a neat Perl oneliner.