Comments on: Drop statistics pages

By: Mike Boone

Mike Boone — Sun, 13 Mar 2005 19:57:07 +0000

If you wanted to block robots from an entire forum site, you could add the robots meta tag to the PHPBB templates/subSilver/overall_header.tpl file. If you only want to do it for certain pages of a forum, like the member list or user profile pages, you would have to try a method like I described in the blog.

By: Administrator

Administrator — Sun, 13 Mar 2005 19:15:40 +0000

Sooo, if I hack the templates, I could add those nofollow tags into the links I need to remove from Google et al? Yeah, I’ll look into that!

By: Mike Boone

Mike Boone — Sun, 13 Mar 2005 18:50:54 +0000

Here’s how I did it for PHPBB, but it only uses meta ‘robots’ tags.

http://boonedocks.net/mike/index.php?/archives/70-PHPBB-Member-List-Link-Spam-Part-2.html

I’m sure you could set up .htaccess controls like you’re suggesting with some sort of regexes.

By: Administrator

Administrator — Sun, 13 Mar 2005 17:56:15 +0000

That’s the easy part.

I’m thinking more complex things, like specific URL’s.

viewtopic.php?p=
but not
viewtopic.php?t=10

See what I mean?

By: Arve

Arve — Sun, 13 Mar 2005 17:37:37 +0000

See http://www.searchengineworld.com/robots/robots_tutorial.htm for a tutorial on how to instruct robots not to index or follow links to certain pages.

By: Administrator

Administrator — Sun, 13 Mar 2005 16:31:49 +0000

Could you do a tutorial on your site about how you disallow certain pages or sections in an .htaccess file?

I mean, what especially irritates me is when Googlebot starts following every little link on forums. Say for instance those “last post” links on forums. And the quote links and other stuff. Googlebot gets SO lost, it’s not even funny! In my logs, the links are even truncated because of the session ID’s of the URL’s on IPB.

By: Mike Boone

Mike Boone — Sun, 13 Mar 2005 15:31:48 +0000

I think the stats page authors should make their pages non-indexed by default. It’s so easy to add in the proper meta tags.

I ran into a similar situation with a PHPBB bulletin board I manage. I get spam user account signups where the bogus user leaves a link to their spam website. Some spammers will even do create these accounts manually (I have a basic bot blocker). My solution was to set the robots meta tag to noindex, nofollow on the pages that list the members and their accounts. Unfortunately, that will probably not stop the tactic unless the authors would make that the default.

Having the stats page authors change their indexing settings is a long-term solution, but a good one I think.