Drop statistics pages
Virtuelvis suggests the search engines drop statistics pages from their indexes.
I’m sure that could be done, if they wrote specific exclusion parameters based on common statistics programs?
Virtuelvis suggests the search engines drop statistics pages from their indexes.
I’m sure that could be done, if they wrote specific exclusion parameters based on common statistics programs?
March 13th, 2005 at 9:31 am
I think the stats page authors should make their pages non-indexed by default. It’s so easy to add in the proper meta tags.
I ran into a similar situation with a PHPBB bulletin board I manage. I get spam user account signups where the bogus user leaves a link to their spam website. Some spammers will even do create these accounts manually (I have a basic bot blocker). My solution was to set the robots meta tag to noindex, nofollow on the pages that list the members and their accounts. Unfortunately, that will probably not stop the tactic unless the authors would make that the default.
Having the stats page authors change their indexing settings is a long-term solution, but a good one I think.
March 13th, 2005 at 10:31 am
Could you do a tutorial on your site about how you disallow certain pages or sections in an .htaccess file?
I mean, what especially irritates me is when Googlebot starts following every little link on forums. Say for instance those “last post” links on forums. And the quote links and other stuff. Googlebot gets SO lost, it’s not even funny! In my logs, the links are even truncated because of the session ID’s of the URL’s on IPB.
March 13th, 2005 at 11:37 am
See http://www.searchengineworld.com/robots/robots_tutorial.htm for a tutorial on how to instruct robots not to index or follow links to certain pages.
March 13th, 2005 at 11:56 am
That’s the easy part.
I’m thinking more complex things, like specific URL’s.
viewtopic.php?p=
but not
viewtopic.php?t=10
See what I mean?
March 13th, 2005 at 12:50 pm
Here’s how I did it for PHPBB, but it only uses meta ‘robots’ tags.
http://boonedocks.net/mike/index.php?/archives/70-PHPBB-Member-List-Link-Spam-Part-2.html
I’m sure you could set up .htaccess controls like you’re suggesting with some sort of regexes.
March 13th, 2005 at 1:15 pm
Sooo, if I hack the templates, I could add those nofollow tags into the links I need to remove from Google et al? Yeah, I’ll look into that!
March 13th, 2005 at 1:57 pm
If you wanted to block robots from an entire forum site, you could add the robots meta tag to the PHPBB templates/subSilver/overall_header.tpl file. If you only want to do it for certain pages of a forum, like the member list or user profile pages, you would have to try a method like I described in the blog.