Support Forum
Support Forum
Offline
I got an email from Google Analytics that Googlebot couldn't crawl my site. Any ideas how to fix this?
Here is the error:
Googlebot couldn't crawl your URL because your server either requires authentication to access the page, or it is blocking Googlebot from accessing your site.
The folder causing the error is: /wp-content/sp-resources/forum-smilies
Offline
why are you trying to crawl that directory??? no forum content there... no real reason to crawl anything below wp-content... sounds like you need a robots.txt file...
Visit Cruise Talk Central and Mr Papa's World
Offline
*I'm* not trying to crawl that directory, lol, Googlebot is! I am not privy to the ways of Googlebot, he goes where he wants 
I have never gotten this error before installing Simple Press, why would I need a robots.txt file?
Offline
yeah, I meant why are you letting bots crawl it... if you dont want to make your own robots.txt, there are many wp plugins that will do it for you...
you may be getting it now, because bots are detecting that plugin because of smilies in your forum content... but having a robots.txt to control how bots behave on your site has always been a google recommendation...
Visit Cruise Talk Central and Mr Papa's World
All RSS1 Guest(s)