mad_asshattertoTechnology@lemmy.worldEnglish·5 months agoThe Internet Archive is under a DDoS attack(bsky.app)external-linkarrow-up11.36Karrow-down17message-square216file-textfedilink
arrow-up11.36Karrow-down1external-linkThe Internet Archive is under a DDoS attack(bsky.app)mad_asshattertoTechnology@lemmy.worldEnglish·5 months agomessage-square216file-textfedilink
minus-squareGeneral_EffortEnglisharrow-up1arrow-down0·5 months agolinkfedilinkhttps://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ Does this answer the question?
https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/
Does this answer the question?