Support Forum

Google webmaster tools and the plugins / themes folders

JV Joan Alamo Vallejo
Joan Alamo Vallejo
Member

Recently I received an alert by the google webmaster tools about a huge increase of crawling errors, it turned out to be plugins and theme files like

wp-content/sp-resources/forum-plugins/plupload/ahah/sp-plupload-attachments-display.php
wp-content/sp-resources/forum-plugins/private-messaging/template-files/spPMThreadsView.php
wp-content/sp-resources/forum-plugins/sp-import/admin/spimport-setup.php
wp-content/sp-resources/forum-themes/custom/templates/backup/spProfileEdit.php
wp-content/sp-resources/forum-themes/custom/templates/spDefault.php
wp-content/sp-resources/forum-themes/custom/templates/spNewPostsView.php
wp-content/sp-resources/forum-themes/custom/templates/spSearchView.php

How can I avoid this? Should I modify the robots.txt and how?

Also, I discovered the folders and files are visible if I go to urls like:

http://domain/wp-content/sp-resources/forum-themes/custom/templates/

Which is the suitable files and folders permissions? Currently I have 755 and 644 for folders and files

8 Answers

New Answer

MP Mr Papa
Mr Papa
Member

why are you letting google (or other bots) crawl  you wp-content folders and sub folders?  there is not content there…

you should have a robots.txt file in your wp root to tell bots where they can and cannot go…  there are wp plugins to help with this or you can create the file itself…  just google wordpress and robots.txt for many topics…

JV Joan Alamo Vallejo
Joan Alamo Vallejo
Member

That’s my point, but I thought maybe here I could find someone who already had the problem and tell me how to fix it

I added this

Disallow: /wp-content/
Disallow: /wp-content/sp-resources/
Disallow: /wp-content/sp-resources/forum-plugins/
Disallow: /wp-content/sp-resources/forum-themes/

But I don’t know if it’s enough.. Maybe I should put some wildcards somewhere

MP Mr Papa
Mr Papa
Member

its a broader wp site issue, not just sp…   and you need to do a user agent too…  and why I suggested goolging it because many opinions out there…  something simple like this would help:

User-agent: *

Disallow: /wp-content/
Disallow: /wp-icludes/
Disallow: /trackback/
Disallow: /wp-admin/
Disallow: /archives/
Disallow: /category/
Disallow: /tag/
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
Disallow: /*.php$

JV Joan Alamo Vallejo
Joan Alamo Vallejo
Member

Thanks, I’ll try that

Any ideas on which permissions should the sp resources folders have? Because I tried to change them but sp needs to be able to see those resources

MP Mr Papa
Mr Papa
Member

curious, why?  wont affect the google crawling with robots.txt…

and it can be different per server set up… typically, its 0755 for directories and 0644 for php type files…  should pretty much mimic what you have set up for wp-content… in fact, will typically inherit when created at install time…

JV Joan Alamo Vallejo
Joan Alamo Vallejo
Member

It just it doesn’t feel very secure to be able to list the files of those folders

 

Edit: Seems that adding Options -Indexes to the htaccess did the trick

MP Mr Papa
Mr Papa
Member

again, not a simple press issue…

in  your .htaccess, add:

Options -Indexes