facebookexternalhit crawl delay or slow down [closed]
Question
Facebook crawler sending 400-500 requests/minute with the following User Agent:
facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
My site goes down 503 error. I tried robot txt file to crawl delay or submitted post in Facebook Developer Support. Is there any way to slow down without blocking the bot?
Here’s sample raw access logs of my site, any suggestions would be greatly appreciated!
0
4 months
0 Answers
10 views
0
Leave an answer
You must login or register to add a new answer .