“I don’t think Mr. Page is sitting in a large chair in a dark tower, wearing a large pinkie ring and stroking a hairless cat while plotting world domination. (At least I hope he’s not.)

But as Google has grown, and the company sees the threat of others on the horizon, it seems that “do the right thing” may have been paused to prevent itself from fading like a Yahoo or an AOL.”

Nick Bilton, Growing Too Big for a Conscience, New York Times

“I don’t think Google was ever “not evil.” Nor do I think that Google was ever “not good.” I think, like any company, it’s not perfect. But unlike most companies, it created an entire “Don’t Be Evil” mantra for itself that it could have never lived up to.”

Danny Sullivan, On Google & Being Evil, Marketing Land

“For months, ‘Don’t be evil’ was like a secret handshake among Googlers. An idea would come up in a meeting with a whiff of anticompetitiveness to it, and someone would remark that it sounded … evil. End of idea.” …

“In his ‘Owner’s Manual to Google,’ Page put front and center the unofficial motto of Google, ‘Don’t be evil.” “We aspire to make Google an institution that makes the world a better place,” he wrote. “We believe strongly that in the long term we will be better served – as shareholders and in all other ways – by a company that does good things for the world even if we forgo some short-term gains. This is an important aspect of our culture and broadly shared within the company.”

Steven Levy, In The Plex: How Google Thinks, Works, and Shapes Our Lives

The New York Times’ Nick Bilton wrote a strong piece arguing that a growing Google is struggling to live up to its founding values. Search industry expert Danny Sullivan argues that “Don’t be evil” was never attainable and somewhat misguided.

Without wading into the important points Nick and Danny raise, I want to take a moment to celebrate the young Google’s decision to take the “Don’t Be Evil” pledge, and to spend over a decade trying to live up to it.

Last year at South by Southwest I heard Senator Al Franken state that corporations are legally required to maximize revenues.

And yet Google clearly told potential investors that they were devoted to doing good, and that they should only buy the stock if they were comfortable with that. Google’s aspirations to morality were not just an empty motto. They affected their internal decisions, and they clearly told new investors that the company did not see maximizing returns as its only goal.

Danny Sullivan makes two specific points against “Don’t Be Evil”:

1. It’s shorthand for “don’t be like our evil competitors.”
2. It leads Googlers to simplistic views in dismissing critics’ claims. How can you say we’re not respecting your privacy, didn’t you see our motto?

Danny suggests not that Google drop its aspirations, but that it turn them more positive, into something like “Be good” or “Be good to our customers.” Danny provides quotes from Steven Levy’s In The Plex to show that some early Googlers felt the same way.

I suspect that “Don’t Be Evil” helped drive Google’s values for a decade in a way that “Be Good” would not have.

Absolute, negative phrasing has advantages that positive phrasings do not. The commandments “Don’t murder. Don’t commit adultery. Don’t steal. Don’t perjure” set absolute limits more effectively than positive statements like “Support life.” Granted, “Don’t be evil” is more vague, and is focused on not being rather than not doing.

But my only strong disagreement with Danny is his statement that “‘Don’t Be Evil’ was incredibly dumb.”

“The next time you hear about, or call someone, a hypocrite, remember that subjecting themselves to such labeling is a price people pay for being identified with standards higher than themselves.”

Dennis Prager, Think a Second Time

Yes, Google makes mistakes. Yes, Google probably overestimates its own virtues and its competitors’ vices. Yes, some business decisions may really be about more evil vs less evil. And yes, Google may misdirect its perceived morality into a misguided or evil crusade.

And still. Google has been a great company that has done great things. Google is a very powerful entity that should have internal values that restrain it.

The only consensus Western value is tolerance. People love to celebrate the moral failures of those who publicly aspire to any additional value. I congratulate Google for privately and then publicly declaring a commitment to not be evil, and to put that phrase on the cover of their IPO filing.

Yes, it’s a hell of a challenge living up to that value. But better to occasionally stumble in trying to maintain a high moral standard than to abandon the standard. Good luck, Google.

I gave this presentation at SMX Israel. Rewritten after Google launched Search Plus Your World a few days before the conference.

“I’m so excited that our personalization works so well that people are creating dystopian fantasies about it.”

“How stupid would your friend think you are if he asked for the bus schedule and you gave him schedules for far-away cities? How annoyed would you be at your friend if he wasn’t paying attention to you so he answered your question without considering the previous parts of your conversation?”

Google Product Manager Jack Menzel, Best of SMX East

“Google Plus isn’t Google’s attempt to build a third or fourth social network. It’s the centerpiece of our attempt to understand our users better so we can better serve throughout the product line.”

Google Plus Product Manager Christian Oestlein, Best of SMX East

“Google is portrayed as Godzilla but sees itself as Snuffleupagus”

Jeff Jarvis, Best of SMX East

Yesterday’s release of Search Plus Your World was one more giant step for Google. We are living in the era of Google’s grand unifications.

Search, Social, and Serendipity

Every few months, even at search conferences, somebody gets up and declares the imminent demise of search or SEO. Sometimes they’re just looking for attention, sometimes they’re just idiots. But often the problem is they only understand half the story.

Yes, we often trust our friends more than we trust strangers. Yes, sometimes the thing you want can find you, without you looking for it. Yes, we want more than to type into a text box and see the same 10 blue links that everybody else sees.

But there was some mistaken impression that Google couldn’t or wouldn’t do these things. That Facebook or Apple would get there first. And that the end result would be that search, SEOs, and Google would cease to be involved in how people connect with what they want.

“It is easier to reliably detect social spam than link spam.”

Vanessa Fox, A Holistic Look at Panda

“I always tell people not to ask how does Google know. They just know.”

Vanessa Fox, Best of SMX East

Google is studying the social graph. They know who you are.  They know what you like, what sites you visited, and how long it took you to return to Google. They know how many people who visited those sites clicked +1 or bounced back quickly and hit “Block this site.”

We can argue about whether they’re Big Brother or your best friend. But they’re not a 1999 search engine that only knows about pages and links.

The Search Quality and Spam Groups

The gap between search and social isn’t the only key gap Google’s been filling.

Google decided years ago that the people should be represented by two separate yet equally important groups (as I wrote that, my Android phone made the Law & Order beeping sound that it makes for incoming messages. I think they’re watching me. I should stop using Chrome for writing blog posts). There’s the search quality team, led by Amit Singhal, who rewrote the Google search algorithm in 2001. Blindfolded. In Assembler. While drunk. And the search spam team led by Matt Cutts, who also serves as Google’s ambassador to the search community. [And of whom I do not make fun, because, well, he runs the search spam team, and is therefore probably more powerful than the president of France. Amit probably is too, but he doesn't know who I am.]

Anyway, this division of power worked well, but had some blind spots exposed.

“So we did Caffeine in late 2009.  Our index grew so quickly … we basically got a lot of good fresh content, and some not so good.”

Amit Singhal

“It was like, “What’s the bare minimum that I can do that’s not spam?”  It sort of fell between our respective groups.”

Matt Cutts

Google had a blind spot, a vulnerability. Sites could achieve trust, and then mass produce highly targeted pages with thin content. Amit’s algorithms couldn’t quite recognize quality at a page level well enough to know when Google was overvaluing a highly relevant (but useless) page that was mass-produced by a site they had trusted. Matt’s team couldn’t help, because there was no spam involved.

SEO professional: I have a few sites where I had already grown the content and the links, and I was just about to start pushing out the rest of the pages when this thing hit.

Me: So that’s what you do? You build the trust with good pages, wait a bit, then exploit it with tons of crap?”

SEO professional: Isn’t that what we all do?

Conversation between me and a top SEO at a bar outside SMX West, 2011

SEOs were exploiting this gap between Amit and Matt’s groups to establish Google’s trust and then exploit it. Larger sites like eHow, which were once high quality, had a compelling business interest to create large amounts of content but little interest in paying for high quality. [Yes, I continue to think that eHow was unfairly hit because interested parties raised a deafening roar that prompted Google to modify Panda to crush it. But that's another story].

Closing the Gap

So our heroes joined forces and created the Panda Update, which, unless you’re one of the sites affected, is more important for what it foretells than for what it has done.

First, Panda closes the gap between the two teams. They took some ideas from Amit’s bag of tricks. Panda is a ranking factor. It’s a computational intensive process run every few weeks, like Page Rank. They combined that with some of Matt’s tricks. Panda is a sitewide penalty implemented as ranking factor and updated like Page Rank.

Back in the old days penalties were applied by sledgehammer. -30. -50. -950. If Amit decided your page should rank #2 but Matt hit your site with a -950 penalty, you dropped to 952. And you know the difference between a -30 penalty and a -950 penalty?Right, 920. But in reality, nothing. Maybe in 1998 there was a tiny practical difference between being on page 4 or on page 96. But in 2012, neither will get you any traffic.

The Panda penalty is applied as a ranking factor and is far more subtle. If Google can’t find any pages that it likes better for a particular search, your page may still rank first, even if you’re being weighed down by the Panda. You can still find most eHow pages, for example, but you’ll often find them a few slots lower. Which was enough to do this to its stock price.

Panda’s Polluting Publishers Penalty

With Panda, Google decided to make publishers pay a price for putting out bad pages.

When you expose a page to Google, Google considers it a request to trust you to send you its user. If too many users return unhappy, Google responds as it would to someone who has betrayed its trust.

So your job is to stop that from happening. The two obvious pieces to this are:

• Satisfy the searchers Google sends you.
• Don’t ask Google to send you searchers that you can’t satisfy.

Calling All Data

Google’s biggest change in recent years is that they finally declared that they’re considering every signal they can get their hands on. Well – almost. It’s still presumed that they’re not taking data from publisher-side programs like Google Analytics and using it against you. But Google is considering behavioral signals from user activity on Google’s sites, applications, and toolbars.

“In May 2010 I said that we weren’t using Social Signals. Now we are.”

“What we ended up doing is taking multiple data sources and intersecting them, so if one data source had a false negative, the others wouldn’t.”

Matt Cutts, Google Webmaster Video, December 17, 2010

“I always tell people not to ask how does Google know. They just know.”

Vanessa Fox, Best of SMX East

Previously Matt repeatedly answered questions with statements like “we don’t use that signal, it’s too spammable.” Now he answers similar questions by saying that Google considers so much data that even if you fool it on one signal you’re unlikely to fool it on all the corroborating signals. Former black hat SEOs have responded by saying things like “it’s now easier to build a good page than to fake it.” With Panda, Google explicitly stated that they were using some behavioral signals as corroborating signals.

User-Friendly Role Model: The IRS

One way that Google makes sense of the boatload of data that they have is by comparing data sets to paradigmatic models of good and bad web sites. Panda is a Document Classifier, which means it looks at things and puts them in Pile A or Pile B.

“We actually came up with a classifier to say, okay, IRS or Wikipedia or New York Times is over on this side, and the low-quality sites are over on this side.”

Matt Cutts, Wired Magazine, The Panda That Hates Farms, March 3, 2011

So your site needs to emit the same signals as sites like the IRS, Wikipedia, and the New York Times. So try to be a blood-sucking anti-capitalist rag. Kidding. But when was the last time you heard the IRS named as a pillar of user friendliness? If more of my visitors wanted to kill themselves after visiting me would Panda like me more?

Seriously though, Google’s use of a document classifier is significant. Combine this with the general idea of using many spammable signals and suspecting sites that don’t conform to expected patterns.

“You know the best way to ensure your site has a ‘footprint that sites that focus on users have?’ Focus on users!”

Vanessa Fox, Search Engine Land, Lessons Learned at SMX West, March 12, 2011

I partially agree with Vanessa’s point that if you focus on users you’ll look like you’re focusing on users. But it’s not always true. For example, Answers.com built a great site that spent millions of dollars annually licensing and aggregating quality content, much of it exclusively, from trusted publishers. The site focused on users. But it had the footprint of a scraper. We had some interesting internal arguments about showing Wikipedia content. It was the right thing to do for our users. But, even after we NoIndexed Wikipedia-only pages, it hurt our footprint at Google.

If search traffic is a big part of your business, you need to pay attention to whether you look more like the IRS or like eHow. Even if you think eHow is a better designed and more user-friendly site.

Google’s Timeless Problem, and Their Progress

“‘Junk results often wash out any results that a user is interested in …

The number of documents … has been increasing by many orders of magnitude. …

People are still only willing to look at the first few tens of results.”

Larry Page, The Anatomy of a Search Engine, 1998

Google’s core problem hasn’t changed: publishers keep creating a lot of junk, and users want the best page to jump to the top. In 1998 Larry Page complained that users were “still only willing to look at the first few tens of results.” Three peculiar words here. The word ”still” indicates that Page thought that search engines would keep falling further behind in their battle to highlight the best content. But you’d need to eliminate the words “tens of” to get to today’s reality, where it’s hard for us to even fathom that our ancestors would look at tens of results. Probably while they were walking 3 miles in the snow to school every day. Uphill. Both ways. And Google has been so successful that users keep raising the bar, and complaining if the best page isn’t the top result.

The Future

 ”The job of SEO has been upgraded from SEO to web strategist. Virtually everything you do on the Internet with your website can impact SEO today. That is especially true following Panda.”

Rand Fishkin, How Google’s Panda Update Changed SEO Best Practices Forever

“It’s too hard now to fake a natural footprint well enough to fool Google.”

“Human Engagement is the new Page Rank. Build engagement signals, get links from pages with good engagement signals.”

Greg Boser, Best of PubCon

When I started going to SEO conferences back in 2005 I got the feeling that I was witnessing the end of an era. SEO had been the Wild West, but the great frontier was finally being tamed.

It may take some time for people to acknowledge the new reality. And surely there are still some pockets of rogue activity, especially in the big money spaces like pharmaceuticals and gambling.

But search marketing is increasingly about applying the fundamental principles of marketing to the particular environment of search. And the key strategies are the same.

Specifically:

• Build your reputation.
• Build your relationships.
• Learn what your potential customers want, and how they’re trying to find it.
• Build pages that satisfy their needs and generate positive attention.
• Look like a good business. Make sure you’re giving off the same positive vibes as the places that people trust.

As Google closes the gap between search and social, these fundamentals become increasingly important. For all but a few rogue geniuses who manage to stay one step ahead, SEO is increasingly about reputation and relationships. Earn  people’s friendship and trust. Understand what they want. And only signal Google that you can deliver what their users are looking for when you really can.

Good luck.

” … the old South Indian Monkey Trap … depends on value rigidity for its effectiveness. The trap consists of a hollowed-out coconut chained to a stake. The coconut has some rice inside which can be grabbed through a small hole. The hole is big enough so that the monkey’s hand can go in, but too small for his fist with rice in it to come out. The monkey reaches in and is suddenly trapped—by nothing more than his own value rigidity. He can’t revalue the rice. He cannot see that freedom without rice is more valuable than capture with it.”

Robert Pirsig, Zen and the Art of Motorcycle Maintenance

Yes, Google Panda should make the Web better. Panda penalizes sites for having too many low quality pages, which will stop sites from throwing up millions of crap pages in the hope that a small percentage will rank. It’s like gmail’s spam filter blocking a good e-mail because too many other of that sender’s e-mail were deemed spam.

And yet …

There’s a systems problem.

• Because Panda is a sitewide penalty there is little correlation between the pages that lose traffic and the ones that cause the penalty.
• Because Panda runs infrequently (roughly once a month) the actions that will help your business today have no correlation (or more likely negative correlation) with the actions that will help you escape the Panda.

If that’s not bad enough, it seems to take more than one clean bill of health from Panda to escape the penalty box.

So in the long term, Google Panda incentivizes the behavior that Google wants. But in the short term, executives have a choice. They can try to get back some of the lost traffic. Or they can not only accept the losses but also inflict more short term pain on their sites in the hope that a few months down the line their successors will get out of the Panda penalty box.

If you’re thinking “you shouldn’t have gotten into this mess,” well, first of all, that’s often not a fair claim. I remain of the opinion that eHow is guilty of little more than aggressive pre-IPO PR. And Reference Answers was hit for spending over a million dollars a year licensing non-exclusive content from top publishers in order to serve their users.

eHow's Panda Double Dip (source: Alexa)

eHow can claim (correctly, IMO) that they were unfairly put in the same bucket as spammers by interested parties (for more on this see Rich Skrenta, Praying for Spam and Google’s Decline: Myth or Fact).

But like the monkey in the South Indian trap, eHow must let go of whatever asset is holding it back. Economists talk a lot about sunk costs. Well, sunk assets is an important concept too. The food in the monkey’s hand is a sunk asset if the monkey can never eat it, but it’s against all of our instincts to let it go. Sometimes you have to mentally relinquish ownership of a lost or toxic asset so that you can move forward again.

I was wrong when I wrote that Panda “better aligns our short & long term interests.” Panda takes at least a month or two to recover from so there’s still a critical disconnect between short & long term interests. So Panda is fatal to monkeys. Let’s hope you’re not one.

Image courtesy of Rick Weiss

Excellent piece today by Danny Sullivan describing the different Google update types and Panda. Combining Danny’s post and some other posts, here’s a quick summary:

1. Index updates: Once upon a time, Google updated their index about once a month, then with “everflux” they began refreshing their index pretty much daily, and now with Caffeine Google says they “update our search index on a continuous basis, globally.”
2. Algorithm updates: Google makes frequent algorithm updates. Most are quite small.
3. Factor tweaks: They frequently play with different weightings (Danny gives the example of the importance given to the search words appearing in close proximity). It can implement these tweaks quickly and test them frequently.
4. “Manual” factors: Thus named because somebody decides when to run the algorithm that updates each factor. These factors are expensive to calculate, and are therefore calculated fairly infrequently. PageRank and Panda are two such factors. Additionally, Google has been working on improving the Panda algorithm, so each time they’ve run Panda it’s been a somewhat different algorithm. Presumably as computing power gets cheaper and the algorithms get more efficient today’s “manual” factors will get refreshed more frequently.

There seems to be something extra at play with Panda, that a bad Panda score doesn’t seem to be completely overwritten in the next Panda update. Danny initially referred to Panda as a ranking factor, but later referred to it as a penalty. The fact that both terms are accurate helps show how Panda differs from previous factors and penalties, and why it was a combined effort between the search quality and the anti-spam team. Back in the old days Google often fought spam by applying draconian penalties on pages, the dreaded -30, -50, -60, and -950 penalties, to name a few. Panda seems to be a penalty that gets applied as a fairly subtle ranking factor, such that the page can still rank #1 if it appears much better than the competition, but will rank below competing pages that appear to be almost as good. And just dropping a slot or two on many queries is enough to significantly affect a website.

Panda should mostly make the Web better, though I’ll be happier when there’s a better feedback loop helping sites succeed in delivering quality pages to Googlers.

I’ve mentioned before that the Danny Sullivan – Matt Cutts conversations are my favorite parts of search conferences. In this case, it was also a chance to watch a potentially significant search & social feature get hatched.

Matt had used the forum to announce the launch of Google’s “block this site” feature on their search results page. (They had launched it in Chrome a few weeks earlier). Somebody asked why they only ask for negative signals and not positive signals, and Danny followed through:

Danny Sullivan: Can we have a PageRank This button? They’ll make [Facebook's] Like buttons like yesterday’s Oldsmobile.

Matt Cutts: That’s good feedback.

Danny: We could have suggested anchor text.

Matt: I’ll take that back.

Disa Johnson (from the audience, sitting right behind me): Google Toolbar used to have a smiley button.

Danny: Yeah, you mentioned that at the first conference you spoke at.

Matt: Google toolbar used to have Happy & Frowny buttons. My wife drew them.

Well, it’s only 3 weeks later, and Google just announced +1, which Danny Sullivan called Google’s Answer To The Facebook Like Button.

No word on whether or not Matt’s wife drew the icon.

Go to the Google Experiments page to turn it on.

[The above transcript is from my notes, but if you're interested in more from that session see Barry Schwartz's SMX Live: Ask the Search Engines

Also see more great moments from SMX West.]

Update: Danny referred to this post in his write-up of +1

“Earlier this month, I joked that I wanted Google to launch a “PageRank This” button for web sites. The new +1 button is kind of like that, though it has been in development well before my joke.”

So a few thoughts for Danny:

1. Thanx for the link!
2. Anchor text “joked.” Really? I guess it’s better than “the.”
3. Yeah, you were joking with Matt, especially calling it a PageRank This button and asking to be able to add anchor text. But the wider context was serious, surrounding the “why only negative signals” question, Disa bringing up the old smiley button, and Matt saying he’d take the idea back to Google engineers.
4. Now why point out that the idea was in the works at Google Labs before the SMX session? Who lets the facts get in the way of a good story? I thought you considered yourself a journalist!

As Aaron Wall from SEO Book demonstrated, Google’s algorithm changes change the course of the Web. Aaron took the negative approach by focusing on Collateral Damage, but there are certainly upsides as well as site owners develop their sites to succeed within the Google ecosystem.

Panda is likely to cause the following shifts in how the web evolves:

• More focus on user experience, including fewer ads hiding the content.
• Less use of non-unique content. This includes fewer
  • scraper sites
  • nearly identical sites that take standard product feeds
  • mashups, where sites create a unique experience by combining non-unique elements on a page.
• More community-generated content.

All told, Panda should make the Google ecosystem a better place. As Rich Skrenta pointed out years ago, Google is “the start page for the internet” and “companies will succeed by working within the framework of Google’s industry dominance.” Google’s choices regarding which behaviors to incentivize drive the web’s development.

Speaking from the perspective of product management of a large web site, I’m mostly happy with Panda. It better aligns our short & long term interests. Now on trade-offs between user experience and other business metrics we can factor likely Google traffic changes into our analyses. This better aligns our interests with our users’, and will help us make choices that improve quality of user experience. Ultimately this helps web publishers and users.

My biggest problem with Panda is that the feedback loop is even less direct and immediate than it was before. It’s like a pitcher throwing 100 pitches a game for a month and only afterwards getting some hints as to approximately how many of those pitches were strikes and whether or not he pitched well enough to win most of those games. Google may counter that a pitcher shouldn’t nibble at the corners, he should just throw the ball down the middle, but that’s not a viable path to success in a competitive environment. In our case, throwing the ball down the middle would be removing all ads and NoIndexing any of our pages on which any of the data sources that we license has been subsequently licensed by another site. It’s ironic that Google, which is so associated with fast iterations based on metrics-driven decisions, has created an environment where websites are flying so blind.

The other downside is that, as Aaron Wall pointed out, we’re in a never ending cat and mouse game. Google just incentivized a whole new set of behaviors. Matt Cutts had always said that Google wouldn’t use signals like clicks on search results because they were too spammable. Now they’ve reversed course. Todd Friesen (back when he considered himself a black hat SEO) once said that he loved real time search because it was so spammable. Well, those spammable factors are getting more significant. And the more Google uses social media signals, the more time spammers will spend manipulating those signals. Google’s use of social media signals may be good for Google, but it’s bad for social media.

The most discussed irony of the Panda update is that people thought it was to fight “content farms like eHow” and yet eHow emerged as one of the big winners. Panda rewards sites with unique content that are not over-monetized and that users find valuable, whether or not bloggers and competitors hate them, and whether or not they target searches that we find puzzling. So this was good for eHow. And IMO Panda’s use of user signals will have a positive influence on eHow, forcing them to provide sufficient quality to satisfy their users.

So Panda is a nice step forward for the web. It incentivizes sites to do the right things. It will force sites to focus more user experience. Its incentives for community generated content will also push the web in the direction of greater user participation. Ads won’t hide the content you’re looking for. Sites that don’t have comments sections or forums will consider adding them.

Many sites will respond to Panda by becoming better sites. Others will respond by manipulating these algorithmic shifts in ways that hurt the web. And around we’ll go until the next Google update.

What a week for content farms!

Here’s the brief overview:

• On Friday Matt Cutts acknowledged some influential bloggers’ criticisms that Google had a faustian deal with Content Farms that were making Google worse. IMO Matt’s statement boiled down to:
  • You’re wrong, Google is getting better, not worse, and we keep trying to keep up with “users’ skyrocketing expectations”
  • You’re wrong, these sites aren’t spam
  • You’re wrong, Google doesn’t give better rankings to sites that run AdSense
  • We hear you though, and now that we’ve made so much progress on spam we’ll turn more attention to “content farms” which we define as “sites with shallow or low-quality content.” Google made two big changes in 2010 to combat these sites.
• A slew of bloggers and journalists came out with posts covering Matt’s post. Most celebrated that Google was finally going to knock out sites like eHow (which BTW, apparently gained traffic during the 2 changes Matt talked about), and that this would cripple the Demand Media IPO. Here are 2 more insightful posts (IMO) on the issue: Michael Martinez’s What is a content farm? and Aaron Wall’s Google gears up for relevancy changes.
• Demand Media had their IPO, which was even more wildly successful than most expected. It ended its first day with a market cap of $1.9 billion.
• Mahalo did a second pivot from their “human powered search” model. They didn’t even acknowledge their previous ridiculous claims of Mahalo 2.0′s overwhelming success (not). They announced that they’re doubling their staff this year and will be creating 2,500 videos per week in a model that seems copied directly from eHow.
• Stack Overflow gave their 2010 year in review. More on that on Monday. Short answer is that they’re doing a great job creating content for programmers and other computer enthusiasts and professionals. They’re not successful yet moving beyond that, but they may not have to be.

Nobody calls Stack Overflow a content farm but I include them here to try to define what a content farm is and isn’t.

The term “content farm,” like the term SEO, is often used by its critics to refer to every spammy tactic that they don’t like, or that cause other sites to rank above theirs.

Stack Overflow and eHow both meet the following criteria often used to identify content farms:

• Creates thousands of new pages every day
• Gets the vast majority of its traffic from Google. You might even say they view Google’s results page as their site’s home page (Stack Overflow co-founder Joel Spolsky in fact did say this).
• Content-creation mechanism naturally creates pages where demand has been identified.

The term “content farm” was coined to refer to the systematic and efficient creation of thousands of in-demand pages every day. eHow does that. So does Stack Overflow.

Matt Cutts just defined content farms as “sites with shallow or low-quality content.” By that definition, I don’t think they’re going to punish eHow. Neither does the stock market.

A few weeks ago I asked people to chime in on whether Google search results were getting better or worse.  Well, Matt Cutts just did.

Matt Cutts

Here are his key points:

1. Google results are getting better. “The short answer is that according to the evaluation metrics that we’ve refined over more than a decade, Google’s search quality is better than it has ever been in terms of relevance, freshness and comprehensiveness.” Obviously Matt’s an interested party, but so are the critics, and the critics aren’t offering any data of their own.
2. Google has been making great progress fighting webspam, scraper, and low-quality sites.
3. With Google’s success on those fronts, some people have shifted their attention to “content farms” and calling for “even stronger action” against these sites. Google will try to do better here.
4. Running Google ads does not help a site’s Google’s rankings. This is a key point for Google, especially as any linkage could really hurt Google in future anti-trust cases. Google repeatedly insists there is no linkage, which does little to silence the repeated accusations.

Last week, Cecilia Kang at the Washington Post reported that Matt Cutts was in Washington testifying that “The only reasons I know of to go in and change [search rankings] manually is for security, a court order or spam.” In Danny Sullivan’s words, “Google’s results are determined by an algorithm and not tweaked to get particular sites ranking well. Google steps in only to deal with spam, security threats or due to legal action.”

A few quick points of my own:

1. Condemnations of “content farms” generally focus on eHow. But the critics never ask Google to pressure eHow to increase their content budget, or their commitment to quality content. There’s nothing wrong with eHow’s model, as ReadWriteWeb admitted “As long as search engines like Google continue to rank niche, topical content highly – and we see absolutely no reason why they wouldn’t – then Demand Media will continue to pump out thousands of articles a day to feed that page view generating machine.” before they changed their minds and coined the term “content farms“ (previously they used the term “page-view generating machine”). Speaking for Answers.com, further increasing our content quality is our biggest focus. Sites like ours receive significant search traffic because we very often have the best page for some long tail search.
2. Despite the complaints of some tech bloggers, large scale content sites are among the key reasons for Google’s success, not their decline. IMO the biggest difference between Google and their competitors has been how much better Google is at indexing and ranking on the long tail. If you’re seeing eHow, it’s usually because no other site has a page as relevant to your search as the one eHow is offering. If you don’t like eHow, don’t click on their link.
3. The “anti-content farm” columns (like anti-SEO columns) generally pointed out cases of spam or scrapers and then said “see how bad content farms are?” Spam sites dominated one searcher’s searches for Dishwasher Reviews and somehow that became eHow’s fault.
4. Anti-trust concerns probably make Google hesitant about manually addressing relative quality among the large sites. Which is a pity. Because one potential solution would involve Google working with the large sites on achieving measurable quality goals.
5. I think that if Google really favored sites with Google ads then Wikipedia would stop ranking first on so many of my searches. Google would instead give higher ranks to (for example) the many Answers.com pages that provide great exclusive content from premium sources like Oxford and Gale in addition to Wikipedia content.

What are your thoughts?

My previous posts on this subject:

• Google’s Decline: Myth or Fact?
• Peter Berger Explains Demand Media
• Demand Media: A Story in 5 Numbers
• Large-Scale Content-Creation Sites
• 2009 Top SEO Smackdowns: Arrington vs Demand Media
• Quality is Still King

Update (Jan 26, 2011): The stock market did not share the view that Matt Cutts’ statements signaled doom for Demand Media. Demand Media’s IPO today has so far exceeded expectations. See CNN / Money’s Demand Media shares soar 41% in IPO.

“Prices will rise, politicians will philander, you too will get old, and when you do you’ll fantasize that when you were young prices were reasonable, politicians were noble and children respected their elders.”

– Wear Sunscreen, Baz Luhrmann

“Human beings seem to take a morose pleasure from believing that once there was a Golden Age, some lost Eden or Camelot, or superior ancient civilization, peopled by heroes and demigods, an age of greatness long lost and irrevocable.”

– George Will, Men at Work

This weekend saw a flurry of articles bemoaning the decline in the quality of Google‘s search results, most notably:

• Vivek Wadhwa in Tech Crunch: Why we desperately need a better Google
• Alan Patrick: On the increasing uselessness of Google
• Jeff Atwood in Coding Horror (and reprinted in Business Insider): Trouble in the House of Google
• Anil Dash: Three’s a Trend: The Decline of Google Search Quality

Clearly, Google doesn’t always produce the results we’d like. So, is Google in decline or are people comparing Google 2011 to a mythical Golden Age of Google that never existed? I don’t know. But AFAICT neither do the people who are making those claims.

AFAIU here’s the basis for the aforementioned authors‘ claim of decline:

• In a 2009 article Paul Kedrosky noted problems with certain Google searches. All four pieces cite this article as the primary source. Quotes from that article make up about a quarter of Dash’s piece, and more than a third of Patrick’s piece. It merits 5 links from Wadhwa’s post. This is a post from 2009! How did this become proof that Google is currently undergoing a decline?
• Atwood’s primary claim is that Google started ranking pages that copy others’ content. To show how bad this is, he writes “when was the last time you clicked through to a page that was nothing more than a legally copied, properly attributed Wikipedia entry encrusted in advertisements? Never, right?” Meanwhile, Patrick writes the opposite “The other main scamsite type is one that copies part of the relevant Wikipedia entry and throws lots of Ads at you.” Atwood is way off in citing this as an example of Google’s decline. A few years ago there were many sites that copied Wikipedia content and sometimes outranked it. Google cleaned that up. In fact, we (Answers.com) incorporate Wikipedia entries in our reference source (legally, with full attribution, to serve our users, and as part of a long-standing and mutually beneficial relationship with Wikipedia in which we support their activities). Since 2007 we’ve put a NoIndex on our pages that only have Wikipedia content, because in 2007 Google greatly reduced the amount of traffic it sends to sites that index pages that are just copies of Wikipedia. As for content scrapers outranking the content source, that’s been a big problem for us (and I presume others) for years. Atwood is noticing it now because now it’s happening to him. Atwood’s main proof of decline is actually an area where Google has been improving.
• They cite some specific examples where their results were disappointing. But AFAICT none of us know whether or not those results were better or worse a few years ago.
• They cite each other as proof.
  • Patrick is mostly based on Kedrosky’s 2009 piece.
  • Atwood writes “I can’t help noticing that we’re not the only site to have serious problems with Google search results in the last few months. In fact, the drum beat of deteriorating Google search quality has been practically deafening of late.” This deafening drum beat of articles in the last few months is 5 articles, including 2 articles from 2009 and Patrick’s and Wadhwa’s that quote them.
  • Dash’s “Three’s a Trend” uses Kedrosky, Patrick, and Atwood as proof of Google’s decline.

So more than a year ago Kedrosky had problems with a certain type of search, and this weekend Patrick found Kedrosky and Atwood found Patrick and Dash found Patrick and Atwood.

Ross Hudgens’ Tweet

What we don’t have is any real indication that things were better a few years ago and then got worse. Where I differ with these authors is not in our opinion of today’s Google.  Google is extremely flawed. For example, a bug caused them to ban the world’s greatest blog (this one) for five months. But Atwood writes

“The idea that there could be something wrong with Google was inconceivable to me. Google is gravity on the web, an omnipresent constant; blaming Google would be like blaming gravity for my own clumsiness. It wasn’t even an option.”

Really?

Google (like Jeff Atwood and probably those other authors) is a significant intelligence and talent trying to make sense of a complex and confusing world. They get it right far more than others do, but they’re often wrong. Are they getting it wrong more often than they used to? I don’t know. But as AFAICT, neither do any of the people who wrote these articles.

My guess is that the perceived decline is only relative to a mythical past. What do you know about this? Anybody have any evidence indicating a decline or an improvement?

BTW, also see Andrew Goodman’s Search Isn’t Broken Because One Guy Had Trouble Using Google.

Update (Jan 22, 2011): Matt Cutts just chimed in.  Here’s my summary: Matt Cutts on Search and Spam.

It always feels strange to buy a hard copy of a book about something that’s changing rapidly, but Vanessa Fox‘s Marketing in the Age of Google was worth it.

Vanessa Fox (and me) at SphinnCon

Vanessa’s book’s value lies in:

1. Case studies of what works and what doesn’t
2. Descriptions of tools and tactics
3. Insight into how Googlers view various issues and challenges
4. Presentation of search marketing within the larger historical context of marketing
5. The idea it presents of an “Age of Google”

I liked the last item so much that I put it in this site’s tagline.

I’d list the key elements of The Age of Google as an age where:

1. Starting an information-based business or hobby is about as easy as putting up a lemonade stand. Profitability is hard of course, as it is with a lemonade stand. But it’s remarkably easy to put up a site, get it found, and monetize it.
2. Information is often very easy to find and usually free.
3. Smart geeks are given tremendous opportunities to do great things and are treated well.

Google of course didn’t create all of that, but they probably played a larger part than any other individual company. And “Age” may not be the right word. Facebook, Apple, and Microsoft are contemporaries but their worlds are each different.

If you want to learn how best to market in the Age of Google, read Vanessa’s book. If you’re interested in how other management and product management issues are affected by the Age of Google, I hope we can explore the questions and answers together on this site.