Something interesting has just come across one of my networks (hat tip to datadial), just a few days after Digg have announced that they are building a replacement for the much loved Google Reader, they have (coincidentally?) disappeared from the primary google index.
Is it an SEO penalty for links? That seems to be the number one reason that brands are getting booted from google’s index these days… Some conspiracy theorists will no doubt be proclaiming its something to do with their announcement to build a replica of the now defunct Google reader, but personally I really cant see that having any effect. Could there?
Doing a site: search for Digg certainly demonstrates that they are no longer in the index:
Its likely (only if it is link based however) that it would be down to what individuals who submit content do after the fact – ie. sending spammy links at their posts to try and build the pagerank, and create “authority” which they then pass back to their own sites. Digg has long been listed in every “linkwheel” sellers handbook, and if that is the reason then what does it mean for every community site on the internet?
Will we have to manually aprove all new links soon at this rate? Come on Google – WTF – let the internet know what you’re doing please.
UPDATE: Its looking like Digg may have suffered a technical SEO issue with their robots.txt file being unavailable to google’s crawlers. In a nutshell what this means is that google could not see a vital part of their site, which controls access to the spiders. In these circumstances Google may remove the site – however this would probably be the biggest example seen of such a problem.
UPDATE2:
I have had a tweet from Matt Cutts, head of Google’s Webspam team where he references a post he made on Ycombinator, explaining the reason for Digg’s disappearance:
“We’re sorry about the inconvenience this morning to people trying to search for Digg. In the process of removing a spammy submitted link on Digg.com, we inadvertently applied the webspam action to the whole site. We’re correcting this, and the fix should be deployed shortly.
From talking to the relevant engineer, I think digg.com should be fully back in our results within 15 minutes or so. After that, we’ll be looking into what protections or process improvements would make this less likely to happen in the future.”
Some personal notes: Its fantastic that Google have taken the time to let the online community know what has happened, and we cant really hold any ill-will towards them for a human error, its a testament to their professionalism that this hasnt to my knowledge happened in the past, and we applaud their transparency in this matter. I guess the only concern for the average webmaster might be what would happen if this happened to their site, and didnt receive the kind of attention that this did.
Still, thank you Matt, and Google as an organisation for the clarification.
Please, voice your opinion in the comments below!
Martin is Inbound Marketing Director at Expedia EAN. In his career he has executed organic campaigns in industries from Gambling to Entertainment before moving to Travel. He is a regular speaker at conferences worldwide about inbound marketing, and regular writer and contributor to respected publications on the topic.
lol.. People think Google is banning Digg cus of G-Reader-replica? That’s crazy talk.. I guess it’s link-related..
Confirmed. It is back now.
site:digg.com
About 1,070,000 results
Most likely just a coincidence, but pretty hilarious timing. I don’t see why Google would get up in arms over something they shut down.
Time to dive into their link profile and see what dirty tricks they have been doing. Will be interesting to see if it is link related and what link networks/scams Digg is involved in. Would have to be pretty blatant to get banned.
Most likely it’s not even Digg itself but all those fin folks who have been posting their stories on it and then trying to push them in Google
Dave Naylor did there seo, now there gone, see here https://twitter.com/DaveNaylor/status/314400134043676673
I can guarantee that Dave & Bronco will have nothing to do with any penalty, Im pretty sure its a tech problem looking at it further.
Yup I did, for the record I was the guy that got digg to nofollow external links
oh It was in 2009 well before it was sold
“there”? This doesn’t make sense.
Surely it’s a coincidence. Another one. Who will be the next coincidence? #donoevil
Probably link related indeed, for everybody spamming their backlinks.
Nice find!!
They don’t have a sitemap.xml or robots.txt from their root – think that has anything to do with it? Or is it really just petty de-indexing?
They do have robots.txt but it’s cloaked – see examples in my post here http://www.koozai.com/blog/search-marketing/a-brief-guide-to-robots-txt-and-five-mistakes-to-avoid/
Yeah. Every time somebody alleges Google does something less than pristine it’s obviously just silly. After all, Google never does anything shady.
I’m not saying this because of the G reader replica, but can people stop acting like Google is pure as the snow?
Does anybody read about Google’s investigations, privacy issues, less than mannerly (and hypocritical) practices and interesting associations?
I spend a fair bit of time reading and looking at all their investigation and privacy issues and it makes for some interesting reading.
But it doesn’t mater, its Google!!
This doesn’t smell like a penalty to me – they wouldn’t have deindexed them.
I suspect they have accidentally removed themselves via GWT or similar.
That’s what I thought
I Guess you can still use this http://www.google.com/cse/home?cx=008134915465729387641%3Amen4x3sw8va
“Don´t do evil”…
So, a one page website gets banned from Google?
Digg always was about “diggs” (votes if you like) and was always ephemeral, in the old days, you ‘only’ had 24 hours to ‘pop’ a story.
As such there was never any value in “sending spammy links at their posts to try and build the pagerank, and create “authority” which they then pass back to their own sites.”
Depends on the reason why you’d want to link to your Digg submission
The only power that any link from Digg ever wielded was from a front page link.
Linking to a submission never had any bearing on it being frontpaged or not.
Digg wasn’t a one page website. It was filled with a shit ton of drama and history before the mass exodus to Reddit.
There’s a huge history on it’s interaction, but the tldr is that it was bought out by some really shady people and this is the penalty for evil SEO.
Yep. Digg became nothing but a page full of spam. What were once user-up-voted stories were replaced by “stories” from the same five or so Web sites day after day.
They should’ve been removed from searches then.
It wasn’t a one page website, it is a one page website now though.
I know plenty about its history, I got the t-shirt for 200 frontpages
“bought out by some really shady people and this is the penalty for evil SEO.” – tinfoil hat much?
You’re doing it wrong, use proper search engine:
https://duckduckgo.com/?q=site%3Adigg.com
Ouch, that really sucks for Digg. It’s been going downhill for a while.
I remember when Digg used to be my first page to visit once my desktop was on at the office…lol
Good riddance~
WoW, I want to throw a few possible explanations…
1) Google changed their ranking algorithm fairly recently, making digg loose its rank based off new ranking system.
2) Google felt digg was filled with backlink spammers (though lots of sharing sites are).
3) A higher up in google had a personal feud with the founder of Digg which lead to this conspiracy!
It’s gotta be a bug / glitch. Too many sites would be subject to same “link wrath.”
How about some tinfoil hat stuff? Maybe Digg is intentionally telling G to f-off. They still show up in other search engines though this is most likely due to lower crawl rate/archiving delay.
Making a deal with FB maybe? Incorporating into a more social-search landscape than an organic search landscape? Digg is more social anyway, seems like a deal with FB could really boost their value.
If there’s any funny business going on regarding links, Digg will get hit like anyone else by Google. The Penguin and Panda updates last year were targeting excessive and irrelevant links and Digg was unwittingly a part of most “link wheel” equations. I hope it’s the robot.txt error, but as an SEO I’ve seen numerous major sites get smacked by Google in the past year.
It’s probably something related to an alert from Google, something like “be carefull digg.com guys, you’ve crossed the yellow line for your aggregated content”.
probably it’s manuallly set to force them to adjust their link building.
Its good that Matt is updating the community as it was a bit of shock, and def the most attention Digg has gotten over the past 2 years. Thanks for the update
Time to dive into their link profile and see what dirty tricks they have been doing. Will be interesting to see if it is link related and what link networks/scams Digg is involved in. Would have to be pretty blatant to get banned.
Don’t let Matt Butts off the hook so easily.
If the web site was just some little web site owned by somebody like you or me, and it disappeared because of this kind of mistake, good luck ever getting anything restored again at Google.
Transparency really? They got caught with their undies down, sure they had to say something and explain wtf is oing on, if that’s transparency it was really forced on them. In case of some smaller site, we would probably never know what the story was, nor would the site owner.
You might say Ive made enough trouble for myself for one month
It’s hard to believe that Google can do such a mistake
Digg must be getting thousands of spammy link bookmarks everyday & Google must be trying hard to identify those links & avoid indexing them.
I believe that I’m having a similar issue to Digg. Check out the image here with a screenshot of Google Webmaster tools.
http://www.biggerpockets.com/renewsblog/wp-content/uploads/2013/05/crawl-errors.png
I’m getting a ton of 401 errors since the Panda update 2 days ago. Google is telling me it can’t index pages on the site that are readily available to all visitors.
Any ideas?
Whats the domain name, also any specific URL that’s 401′ing?