Cynicus Rex@lemmy.ml to Privacy@lemmy.mlEnglish · 2 years agoHow to block AI Crawler Bots using robots.txt filewww.cyberciti.bizexternal-linkmessage-square46linkfedilinkarrow-up139arrow-down18
arrow-up131arrow-down1external-linkHow to block AI Crawler Bots using robots.txt filewww.cyberciti.bizCynicus Rex@lemmy.ml to Privacy@lemmy.mlEnglish · 2 years agomessage-square46linkfedilink
minus-squareCynicus Rex@lemmy.mlOPlinkfedilinkarrow-up2arrow-down2·edit-22 years agoLies, as in that it’s not really “blocking” but a mere unenforceable request? If you meant something else could you please point it out?
minus-squaredabaldeagul@feddit.nllinkfedilinkarrow-up8·2 years agoThat is what they meant, yes. The title promises a block, completely preventing crawlers from accessing the site. That is not what is delivered.
minus-squareJackbyDev@programming.devlinkfedilinkEnglisharrow-up2arrow-down1·2 years agoIs it a lie or a simplification for beginners?
minus-squarethanks_shakey_snake@lemmy.calinkfedilinkarrow-up8·2 years agoLie. Or at best, dangerously wrong. Like saying “Crosswalks make cars incapable of harming pedestrians who stay within them.”
minus-squareJackbyDev@programming.devlinkfedilinkEnglisharrow-up0arrow-down2·2 years agoIt’s better than saying something like “there’s no point in robots.txt because bots can disobey is” though.
minus-squarethanks_shakey_snake@lemmy.calinkfedilinkarrow-up2·2 years agoMaybe? But it’s not like that’s the only alternative thing to say, lol
minus-squareReversalHatchery@beehaw.orglinkfedilinkEnglisharrow-up2arrow-down1·2 years agoIs it, though? I mean, robots.txt is the Do Not Track of the opposite side of the connection.
minus-squareEager Eagle@lemmy.worldlinkfedilinkEnglisharrow-up1·2 years agothe word disallow is right there
Lies, as in that it’s not really “blocking” but a mere unenforceable request? If you meant something else could you please point it out?
That is what they meant, yes. The title promises a block, completely preventing crawlers from accessing the site. That is not what is delivered.
Is it a lie or a simplification for beginners?
Lie. Or at best, dangerously wrong. Like saying “Crosswalks make cars incapable of harming pedestrians who stay within them.”
It’s better than saying something like “there’s no point in robots.txt because bots can disobey is” though.
Maybe? But it’s not like that’s the only alternative thing to say, lol
Is it, though?
I mean, robots.txt is the Do Not Track of the opposite side of the connection.
the word disallow is right there