Block bad bots htaccess

0 votes

Hello, I'd like to blocks all bad bots from crawling my website such as Ahrefs, SEOPROFILER...

I see some articles guiding me to search bots ips then block them, but very hard to know all bots 

Do you know how to block all of them?

asked Aug 11, 2015 by vinhvinh (570 points)

1 Answer

0 votes
Best answer

Yes, you just simple add these lines to top of your htaccess. I will block all bad bot as we know well. You can add more if you like

SetEnvIfNoCase User-Agent .*rogerbot.* bad_bot
SetEnvIfNoCase User-Agent .*exabot.* bad_bot
SetEnvIfNoCase User-Agent .*mj12bot.* bad_bot
SetEnvIfNoCase User-Agent .*dotbot.* bad_bot
SetEnvIfNoCase User-Agent .*gigabot.* bad_bot
SetEnvIfNoCase User-Agent .*ahrefsbot.* bad_bot
SetEnvIfNoCase User-Agent .*sitebot.* bad_bot
SetEnvIfNoCase User-Agent .*sitebot.* bad_bot
SetEnvIfNoCase User-Agent .*BLEXbot.* bad_bot
SetEnvIfNoCase User-Agent .*Blekkobot.* bad_bot
SetEnvIfNoCase User-Agent .*SEOkicks-Robot.* bad_bot
<Limit GET POST HEAD>
Order Allow,Deny
Allow from all
Deny from env=bad_bot
</Limit>

 

answered Aug 11, 2015 by super (900 points)
Welcome to Expert Customize Forum, where you can ask any IT questions and get free support by Our experts
You have projects or tasks? Want to hire Quality expert? Try our Web expert service