Cynicus RextoPrivacy@lemmy.mlEnglish·2 months agoHow to block AI Crawler Bots using robots.txt file(www.cyberciti.biz)external-linkarrow-up1110arrow-down132message-square64fedilink
arrow-up178arrow-down1external-linkHow to block AI Crawler Bots using robots.txt file(www.cyberciti.biz)Cynicus RextoPrivacy@lemmy.mlEnglish·2 months agomessage-square64fedilink
minus-squareJackbyDevEnglisharrow-up1arrow-down5·2 months agolinkfedilinkIt’s better than saying something like “there’s no point in robots.txt because bots can disobey is” though.
minus-squarethanks_shakey_snakearrow-up3arrow-down0·2 months agolinkfedilinkMaybe? But it’s not like that’s the only alternative thing to say, lol
minus-squareReversalHatcheryEnglisharrow-up2arrow-down1·2 months agoedit-22 months agolinkfedilinkIs it, though? I mean, robots.txt is the Do Not Track of the opposite side of the connection.
It’s better than saying something like “there’s no point in robots.txt because bots can disobey is” though.
Maybe? But it’s not like that’s the only alternative thing to say, lol
Is it, though?
I mean, robots.txt is the Do Not Track of the opposite side of the connection.