facebookexternalhit crawl delay or slow down [closed]


Facebook crawler sending 400-500 requests/minute with the following User Agent:
facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)

My site goes down 503 error. I tried robot txt file to crawl delay or submitted post in Facebook Developer Support. Is there any way to slow down without blocking the bot?

Here’s sample raw access logs of my site, any suggestions would be greatly appreciated!

enter image description here

Rahman 4 months 0 Answers 10 views 0

Leave an answer