Cynicus Rex@lemmy.ml to Privacy@lemmy.mlEnglish · 2 年前How to block AI Crawler Bots using robots.txt filewww.cyberciti.bizexternal-linkmessage-square46linkfedilinkarrow-up139arrow-down18
arrow-up131arrow-down1external-linkHow to block AI Crawler Bots using robots.txt filewww.cyberciti.bizCynicus Rex@lemmy.ml to Privacy@lemmy.mlEnglish · 2 年前message-square46linkfedilink
minus-squarebreadsmasher@lemmy.worldlinkfedilinkEnglisharrow-up2·2 年前It isn’t an enforceable solution. robots.txt and similar are just please bots dont index these pages. Doesn’t mean any bots will respect it
It isn’t an enforceable solution. robots.txt and similar are just please bots dont index these pages. Doesn’t mean any bots will respect it