tekeous@usenet.lol to Lemmy Shitpost@lemmy.world · 10 days agorobots.txt is a suggestionusenet.lolimagemessage-square25fedilinkarrow-up1621arrow-down13
arrow-up1618arrow-down1imagerobots.txt is a suggestionusenet.loltekeous@usenet.lol to Lemmy Shitpost@lemmy.world · 10 days agomessage-square25fedilink
minus-squareAlbum@lemmy.calinkfedilinkarrow-up1·10 days agoUser agent catching is rather effective. You can serve different responses based on UA. So generally people will use a robots.txt to catch the bots that play nice and then use useragents to manage abusers.
User agent catching is rather effective. You can serve different responses based on UA.
So generally people will use a robots.txt to catch the bots that play nice and then use useragents to manage abusers.