Comments on: How to use wildcards in robots.txt http://seogadget.co.uk/wildcards-in-robots-txt/ UK SEO Services Company in London - SEOgadget.co.uk Mon, 15 Mar 2010 08:29:05 +0000 http://wordpress.org/?v=2.9.2 hourly 1 By: Door Manufacturer Keith http://seogadget.co.uk/wildcards-in-robots-txt/comment-page-1/#comment-10283 Door Manufacturer Keith Fri, 08 Jan 2010 01:40:17 +0000 http://seogadget.co.uk/?p=271#comment-10283 Awesome - thanks for the info. This would be really useful for me - to have crawler bot disabled for certain wordpress posts with a particular keyword in the URL... (just amend the slug accordingly) But this only applies to Google yeah? Awesome – thanks for the info. This would be really useful for me – to have crawler bot disabled for certain wordpress posts with a particular keyword in the URL… (just amend the slug accordingly) But this only applies to Google yeah?

]]> By: pah http://seogadget.co.uk/wildcards-in-robots-txt/comment-page-1/#comment-6356 pah Wed, 30 Sep 2009 12:37:05 +0000 http://seogadget.co.uk/?p=271#comment-6356 agree there... 301 redirect, scoop any relevant paras into a cookie, new site map . lovely. if only only i could do actually get that implemented on the site i'm working on agree there… 301 redirect, scoop any relevant paras into a cookie, new site map . lovely. if only only i could do actually get that implemented on the site i'm working on

]]> By: JohnMu http://seogadget.co.uk/wildcards-in-robots-txt/comment-page-1/#comment-602 JohnMu Thu, 02 Oct 2008 07:54:00 +0000 http://seogadget.co.uk/?p=271#comment-602 One thing you need to keep in mind is that disallows in the robots.txt will just disallow crawling, it will have less of an impact on actual indexing. If we have reason to believe that a URL which is disallowed from crawling is relevant, we may include it in our search results with whatever information we may have (if we've never crawled it, we may just include the URL -- if we've crawled it in the past, we may include that information). To prevent URLs like these from being indexed, I would recommend that you have the server 301 redirect to the appropriate canonical (and of course not link to the incorrect one). John One thing you need to keep in mind is that disallows in the robots.txt will just disallow crawling, it will have less of an impact on actual indexing. If we have reason to believe that a URL which is disallowed from crawling is relevant, we may include it in our search results with whatever information we may have (if we’ve never crawled it, we may just include the URL — if we’ve crawled it in the past, we may include that information).

To prevent URLs like these from being indexed, I would recommend that you have the server 301 redirect to the appropriate canonical (and of course not link to the incorrect one).

John

]]>