HubitoReddit@lemmy.worldEnglish·7 months agocross-posted to: reddit@lemmy.mlReddit shares plunge almost 25% in two days, finish the week below first day close(www.cnbc.com)external-linkarrow-up1861arrow-down115message-square173fedilink
arrow-up1846arrow-down1external-linkReddit shares plunge almost 25% in two days, finish the week below first day close(www.cnbc.com)HubitoReddit@lemmy.worldEnglish·7 months agocross-posted to: reddit@lemmy.mlmessage-square173fedilink
minus-squareLaterRedditorarrow-up3arrow-down2·7 months agolinkfedilinkWasn’t there robots.txt at least?
minus-squareSacrificedBeansarrow-up12arrow-down0·7 months agolinkfedilinkRobots.txt is a compliance standard afaik, more like an advisory guide. It cannot enforce bans on scrapers, it sets some polite boundaries for the website.
Wasn’t there robots.txt at least?
Robots.txt is a compliance standard afaik, more like an advisory guide. It cannot enforce bans on scrapers, it sets some polite boundaries for the website.
There was, and still is