Lag 06/10/2020

That might of been me. I was running a system check that pulled on a lot of resources.
 
That was probably it Steve. There is still something there that wasn't before. Sort of a hiccup before pages load.
 
I moved to Cloudflare to try and speed load times up. Maybe the pause is from there? I don't know. I can't find an explanation for the high usage.
 
Could members posting photos directly from computer instead of a hosting site be dragging down resources?
 
Could members posting photos directly from computer instead of a hosting site be dragging down resources?

No, we found the problem. It is search bots indexing the site. They are running 5 request per second ( that's like pulling up 5 webpages per second) and there are dozens of them on the site.
We've added a crawl-delay robot text file to slow their request rate down. It might take a little while to take effect, but it should help.
 
Top 5 bad bots


What we will do now is to choose top 5 bad bots that we don’t want visiting our website and lock them out on a server side through the .htaccess file. When they visit our website, they will get a 403 Access Forbidden error.
  1. Ahrefs – seo tool bot
  2. Semrush – seo tool bot
  3. MJ12bot or Majestic bot – seo tool
  4. DotBot – we are not an ecommerce site
  5. CCBot – marketing
There is a huge list of other bots that you can block at tab-studio. We won’t bother with so many, but will block only the most active spiders.
Block crawlers with .htaccess


Here is the code to insert into your .htaccess file to block the bots:
#bad bots start
RewriteCond %{HTTP_USER_AGENT} \
ahrefs|\
semrushbot|\
mj12bot|\
dotbot|\
ccbot|\
[NC]
RewriteRule .* - [F]
#bad bots end
This code didn’t work for some of our websites that had other blocks on.

Here is another version of how to block multiple bots in one statement in .htaccess:
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^.*(ahrefs|semrushbot|mj12bot|dotbot|ccbot).*$ [NC]
RewriteRule .* - [F,L]
 
Nothing has changed here. Still the same sluggish response time, but nothing I can't deal with. Thanks for taking a shot anyhow to find a fix. It's easy for me or anyone else to sit on the sidelines and be a Monday morning quarterback, but I know how tedious these things can be.

7de472c6e309d20.gif
 
Mike, it should be better. We've banned whole blocks of bot IP addresses and have our CPU usage down to 35%.
I'm getting fast page loading pretty regular now with the occasional hiccup pause.
 
That sounds about right for what I am seeing on my end right now as well Steve. Initially getting on the site is really fast with no lag time. I see a little stutter navigating between threads, but no biggie. I appreciate what gets done in the background to keep all the gears in sync for a smooth running forum experience. Thanks!
 
Back
Top Bottom