Aggressive BOTS that ignore robots.txt

Author	Topic »
Marshall Ecommerce Template Guru USA 2001 Posts	Posted - 08/23/2026 : 08:29:14 Recently I have had several sites hammered with bots, bots which I have as disallow in the robots.txt file, but apparently ignores it. Here are a few: Baiduspider Yandex ClaudeBot GPTBot Scrapy AhrefsBot SemrushBot Sogou Bytespider ev-crawler osentix-crawler meta-externalagent The only recourse is block them either via .htaccess or web.config. Believe me when I say, they will chew up a lot of bandwidth. I have one site that is at 190% bandwidth usage just for August thanks to these bots. Marshall CENLYT Productions - ms designs Affordable Web Design Custom Ecommerce Designs Responsive Websites Cenlyt.com
dbdave ECT Moderator USA 10733 Posts	Posted - 08/23/2026 : 11:49:40 Have you tried blocking them bv IP address. Most of them should be using the same IP or range. David
Marshall Ecommerce Template Guru USA 2001 Posts	Posted - 08/24/2026 : 04:01:41 Yes, but some use a wider range that one might expect, especially Sogou. Marshall CENLYT Productions - ms designs Affordable Web Design Custom Ecommerce Designs Responsive Websites Cenlyt.com
	Topic »