The morning after I arrived home from my post SMX Advanced holiday I got up early to check my site traffic. Not to mention the personal achievement of being up and working before 7am on a Monday (this is good for me…), I actually managed to resist the temptation of logging into Analytics on my iPhone for an entire week while I was in Italy too. Surely the start of a very productive week. Sadly, I found a bit of a suprise. My blog traffic had dropped.
To ease myself back into blogging (and to provide some traffic dropping, Google page penalty based entertainment for SEOgadget readers) here’s the story on what happened:
Comparing the first half of June to the last half of May, overall traffic on the site had dropped by 4.52%.
This was pretty disappointing so I drilled down a little deeper by traffic source, showing a decrease in search engine traffic as the main culprit:
The great thing about Google analytics is how quickly and easily you can drill down to the specifics. Most of us use Google Analytics every day but it’s often only until something a bit unpleasant happens to your site that you really appreciate the beauty of this (free) software.
Drilling down to the keyword level I quickly found the culprit:
Those Virtualbox keywords are from the top of the tail generated by a guide to installing Virtualbox guest additions. “Virtualbox guest additions” is was my 2nd most popular keyword generating 1500 visits in May 2009. So what was going on?
Comment spam, missed by Akismet. Don’t get me wrong, I think Akismet is amazing, but it can miss some types of comment spam. It’s probably my fault for not adding a verification or a CAPTCHA to my comments are but I don’t enjoy the experience on other blogs personally, so I choose to leave that off.
Here’s a sample of what was on the page: (The following image may offend some readers)
Those links were going to some seemingly bonafide domains that happened to have some very hacked looking, spammy page URLs on them. As you can see, they were left on June 3rd, at 6.10am.
Nearly all of the traffic coming to the Virtualbox post dropped within 24hours and stayed that way until I cleared the comments on Monday 15th June at 7.00am (ish). Here’s the screenshots of the Google.co.uk results:
Before:
After:
Thankfully, the page was reincluded within 24 hours of clearing out the comments. Here’s the timeline of events according to Google Analytics (and Snagit, of course.) – Click to enlarge:
So, the traffic has returned and SEOgadget is comment spam free once more. What did I learn from this? I thought it was interesting to see how quickly the page was dropped from the index and how quickly it ranked after the clean up. 24hours each way, no reinclusion request needed. That’s good to know. I discussed the issue with Adam from work and we concluded that it would be quite interesting to look at whether it was the words used in the comments or the links used (nofollowed) or a combination of the two. That’s quite an easy test to do…
Just until the cached copy updates, you can see what Google saw just here.
PS – We stayed in Palinuro, Italy at a little farm called Isca Del Donne. Highly recommended!
Blog RSS Feed
Blog Posts by Email
{ 42 comments… read them below or add one }
Nice piece of work. And also great to know that your page came back in the SERP within 24 hours.
Probably it helps that your site has a lot of readers therewith increasing the indexrank of the page.
It also brings up the necessity to keep moderating pro actively.
Nice sum up and this show how fast Google react to things happening in our pages. Happily they are fast to get your ranking as it was before when you take the time to fix the damn spam.
Seems you had a great trip in Italy, a week off worth the little spam that get through
Nice catch Rich – it’s very interesting to see how quickly this took effect. I’d love to know the outcome of any tests done into whether it was the link or the text. I’d suspect the text, but I could easily be wrong.
Nice piece of analysis, like the way you show a clear correlation of cause and effect – tweeting…
Hi Richard,
Glad to see such a quick reversal after the comments were cleared. I think you’re right not to add a captcha for comments. Some can be notoriously difficult to bypass.
@Tom – sure will. The test is running now
@David – agreed. It’s good that Google clear this type of spam penalty so rapidly. Very clever.
Nice post Richard, it sure does illustrate JUST how seriously the old ‘Don’t link to bad neighbourhoods’ advice should be taken.
@Old Welsh Guy
Exactly – though the question on my mind is, what cause the filter? Is it the links (nofollowed?), the text, or both. That’s what we’re testing now – just to be sure. Thanks for your comments! Love the name, by the way!
@Richard It raises the question (again) of just how ‘nofollow’ Google treats the NOFOLLOW tag. Could this ‘penalty’ (for want of a better word), be the result not so much of the adult content itself, but a simple case of non semantically related content close to links, thus dropping the Google trust?
Ironic that Google, a search engine built on the backrub backlinking citation algorithm, appear to be doing all in their power to stop people linking out with confidence.
Interesting. *whistles*
Was there a reinclusion request submitted, or just comment scrubbing?
@richardbaxter,
I almost for sure know it is the text that kicked the filter.
The drop cant be caused by a nofollow on the links, Google even states you should put a nofollow on links you dont trust.
On several domains i have seen a drop in rankings for a certain page that contained just a little piece of text one could see as related to sexual content.
@Tonnie, That is a real assumption to say it is the text that caused this, as the algo is way more complex than that isn’t it.
It could simply be that the comments were allowed to show (regardless of nofollow), and as a result Google decide the site has lost some editorial integrity and drops the trust element accordingly.
So by that token it wouldn’t actually have been the text, it was the fact it went unmoderated that caused it. I know this is semantics, but many people (not saying you), read things differently and would take it that the use of those words caused the problem, when they didn’t, as there are a lot of sites who use those words, who rank for those words as that is what their site is about.
Great, informative post…what will you add to eliminate future spam?
I use WP-Spamfree in addition to Akismet, because I never like Captchas or Disqus or other kinds of interventions either.
Very informative Richard thanks for taking the time to put the post together. This certainly makes a case for keeping a close eye on the comments.
@Old Welsh Guy
“it was the fact it went unmoderated that caused it”
I like that point – well made.
Guys, I am using a couple of your images in my blog post. Please let me know if this is a problem and I will take them down asap.
Very interesting. It would be very nice if you could test this. I know you probably dont wann risk too much. But it would be funny to see what happens if you reactivate these comments from time to time
Thanks for sharing
Oh yeah, maybe I should post the URL to the blog entry: Moderieren Sie Ihre Kommentare oder riskieren Sie schlechtere Rankings which translated would read something like: Moderate your comments or risk losing your rankings
I’m really starting to think that I better start using paged comments. Presumably Google would start sending traffic again when the bad comments got off the main page (ie onto page 2 of the comnments)?
Who knows – but this along with the recent PageRank sculpting changes make paged comments look better and better (with the canonical tag to prevent duplicate content penalty).
I am suprised that these effects and changes happened so quickly. I could understand if the site was PR5-6 and being crawled frequently. It seems so easy to inflict negative seo on a competitor…I may have to start up Xrumer!
Really good analyse ! But when we look the last image, the visits have already started to grow up before you clear the comments ! Is it normal ?
@sylvain not quite. I cleared those comments very early on the day in question. If I started getting traffic again before the end of that day, the graph will appear to increase.
@old welsh guy
Aint that difficult, the algo has not much to do with it, just a filter that kicks in on certain words.
That there are sites ranking high that still use those words, might be, on the other hand, there are still sites that do spam and rank too
Dont think to difficult when it comes to Google. It aint that complicated.
Further more, if a ‘unmoderated’ penalty would have taken place, how come it did so fast?
Wouldn’t it be more logical that Google throws one in after a certain period?
Just one week and your out of business?
I tend to agree with Tonnie here – it looks a bit like a text filter. I know it’s too late now, but I’d like to know if your website would show up with SafeSearch turned off…
Also, the complete top 4 for ‘virtualbox guest additions’ is different in both screen shots. Could you think of an explanation for this?
Really Interesting this post.
It´s a flag to pay more attention in the comment links, mainly after the changes on nofollow rules.
@Wiep Hello mate. Actually Will at Distilled said the same thing last night.
As far as the totally different rankings go, perhaps a possible indicator of how a shorter term index may be deployed in case the main index changes unexpectedly when a penalty occurs?
Really interesting stuff.
I really hate those comment spammers, and I always make sure all posts are closed when I have no way of moderating the comments while I’m kicking it on the beach. But it makes me sad that my readers don’t have the chance of interacting with me while I’m sipping on a cocktail.
Thanks for sharing this. Very helpful
Dennis
Great post! We allow comments on our site and it’s good to look out for this stuff. Fortunately we don’t allow comments to be posted without approval so hopefully we never run into this problem!
Nice post. Interesting, but not sure that a bunch of comment can attract the attenction of Google penalty system.
I’ve a site of a company who I manage that seems page level penalized, and it’s a standard plain html web site, without comment, so the reason must be searched somewhere else.
I have my comments set to approval before they are published, so this sort of problem won’t happen to me. Indeed, there are actually quite a few spam comments that escape Askimet and that is not normal, automation only goes up to a point.
Thanks for the insight. I didn’t know it would affect a site so fast.
Keep up the good work
Sorry, I meant to say that it is not abnormal that some spam escapes Askimet
wow.. nice post. I thought akismet is close to perfection. huhuhu.. i learned a lot, comments do affect rankings.
thank u!!
Askimet works with a list of known spam messages. In that sense it is perfect, but you cannot do anything against a person who fills in a spam message by hand.
But it is a lot less, and I mean really a lot less then without the protection from Askimet, so I would donate some money towards those guys as they really do a good job.
Am I alone in thinking “so what”. If you are naive enough in this day and age to allow comments onto your blog without moderation, then why would you be surprised by Google’s response when they find them?
@IBL builder
I recieve a lot of comments. I would never be able to manually moderate them all, not to mention the fact that comments moderatated tend to extinguish conversation between commenters. It’s not “naive” to allow comments on a blog, it’s intentional.
Since you have to check your comments anyway to filter out the bad ones, I do not see why moderating them upfront would be a problem. Especially blogs with open commenting are prone to be misused by spammers and now that they have find you out, you are going to have to monitor your comments very closely from now on. Therefore, I think IBL builder has point although he puts it rather bluntly.
But I understand where you come from, moderating conversations doesn’t work encouraging, that is absolutely true.
However, you can get around this problem by allowing trusted commenters to publish automatically. There is a setting for it this. In other words: if you have approved a comment for the first time, the commenter will then be able to comment next time without the need for approval. Secondly, you can filter comments that have more then one link. That’s another firewall for spam.
Hope this helps?
Excellent job paying close attention to your traffic. Even better job communicating this information in a clear, concise manner. Makes me worry about massive comments and the time necessary to manage them all.
Interesting post, I didn’t really think about the negative effects of Comments on your sites SEO, however I agree with most people here that say it was a text filter.
Also search results can vary if a Data Center is having problems, sometimes searching the same keyword yeilds different results because it depends on which data center you’re hitting. I’d say it was the words (Which I’ll omit) ticking off the safe filter.
This makes me want to write about it myself on http://green.cx but I think I’ll just link you instead! You did such a good and through writeup!
I had a similar experience with a travel blog I ran – I found that if you don’t have the time to check things really really carefully its better to disable comments entirely – its so easy for this sort of thing to happen and for traffic to take a hit.
At least things bounced back and you spotted the issue.
Very well written. This is the kind of information that is useful to those want to increase their SERP’s. Keep up the good work.
quite interesting, but congrat. you got traffic back
Great review. thanks for sharing this very forwarded idea.i use twitter only for this. search engine marketing tips´s last blog.
how your page in SERP within 24 hours!!!!!!
great interesting read. Thankz