Luu TuyentoTechnology@lemmy.worldEnglish·9 days agoTikTok’s parent launched a web scraper that’s gobbling up the world’s online data 25-times faster than OpenAI(fortune.com)external-linkarrow-up1571arrow-down17message-square131fedilink
arrow-up1564arrow-down1external-linkTikTok’s parent launched a web scraper that’s gobbling up the world’s online data 25-times faster than OpenAI(fortune.com)Luu TuyentoTechnology@lemmy.worldEnglish·9 days agomessage-square131fedilink
minus-squarejagged_circleEnglisharrow-up11arrow-down0·8 days agolinkfedilinkI think a common nginx config is to just redirect malicious bots to some well-cached terrabyte file. I think hetzner hosts one iirc
minus-squareSomething Burger 🍔Englisharrow-up16arrow-down0·8 days agolinkfedilinkhttps://github.com/iamtraction/ZOD 42kB ZIP file which decompresses into 4.5 PB.
minus-squareWhyJiffieEnglisharrow-up3arrow-down0·7 days agolinkfedilinkwouldn’t it be trivial to defend against that with a hash check if the size matches? though I guess it’s possible to create your own that differs
I think a common nginx config is to just redirect malicious bots to some well-cached terrabyte file. I think hetzner hosts one iirc
https://github.com/iamtraction/ZOD
42kB ZIP file which decompresses into 4.5 PB.
wouldn’t it be trivial to defend against that with a hash check if the size matches?
though I guess it’s possible to create your own that differs