” … the old South Indian Monkey Trap … depends on value rigidity for its effectiveness. The trap consists of a hollowed-out coconut chained to a stake. The coconut has some rice inside which can be grabbed through a small hole. The hole is big enough so that the monkey’s hand can go in, but too small for his fist with rice in it to come out. The monkey reaches in and is suddenly trapped—by nothing more than his own value rigidity. He can’t revalue the rice. He cannot see that freedom without rice is more valuable than capture with it.”

Robert Pirsig, Zen and the Art of Motorcycle Maintenance

Yes, Google Panda should make the Web better. Panda penalizes sites for having too many low quality pages, which will stop sites from throwing up millions of crap pages in the hope that a small percentage will rank. It’s like gmail’s spam filter blocking a good e-mail because too many other of that sender’s e-mail were deemed spam.

And yet …

There’s a systems problem.

• Because Panda is a sitewide penalty there is little correlation between the pages that lose traffic and the ones that cause the penalty.
• Because Panda runs infrequently (roughly once a month) the actions that will help your business today have no correlation (or more likely negative correlation) with the actions that will help you escape the Panda.

If that’s not bad enough, it seems to take more than one clean bill of health from Panda to escape the penalty box.

So in the long term, Google Panda incentivizes the behavior that Google wants. But in the short term, executives have a choice. They can try to get back some of the lost traffic. Or they can not only accept the losses but also inflict more short term pain on their sites in the hope that a few months down the line their successors will get out of the Panda penalty box.

If you’re thinking “you shouldn’t have gotten into this mess,” well, first of all, that’s often not a fair claim. I remain of the opinion that eHow is guilty of little more than aggressive pre-IPO PR. And Reference Answers was hit for spending over a million dollars a year licensing non-exclusive content from top publishers in order to serve their users.

eHow's Panda Double Dip (source: Alexa)

eHow can claim (correctly, IMO) that they were unfairly put in the same bucket as spammers by interested parties (for more on this see Rich Skrenta, Praying for Spam and Google’s Decline: Myth or Fact).

But like the monkey in the South Indian trap, eHow must let go of whatever asset is holding it back. Economists talk a lot about sunk costs. Well, sunk assets is an important concept too. The food in the monkey’s hand is a sunk asset if the monkey can never eat it, but it’s against all of our instincts to let it go. Sometimes you have to mentally relinquish ownership of a lost or toxic asset so that you can move forward again.

I was wrong when I wrote that Panda “better aligns our short & long term interests.” Panda takes at least a month or two to recover from so there’s still a critical disconnect between short & long term interests. So Panda is fatal to monkeys. Let’s hope you’re not one.

Image courtesy of Rick Weiss

SMX London was great. Here’s the best of the best.

Best Lines & Lessons

• David Burgess: [For Extreme Makeover SEO Edition had to get a site from unranked to page 1 in 3 months]: I was told if I couldn’t get the site ranking #1 I’d be sacked. So this is his first and last conference presentation.
• Stephen Pavlovich: We’re not trying to win awards, we’re trying to drive conversions
• John Mueller: Google may crawl internal site search with relevant keywords to find content which isn’t indexed (hat tip: Kevin Gibbons)
• Kelvin Newman: Think about what survey or poll results would people write about and go for it. (hat tip: @DavidCarralon)
• Christine Churchill:
  • Broad match used to be great, but Google kept expanding it and now it’s too broad. Modified broad match is you can put plusses in front of the words. +apple +pie your ad will only show up if it has both words with plusses in front of it.
  • Pay attention to what shows up in Google Instant and Google Suggests when people are typing their search queries.
• Craig Danuloff: In paid search, the amount you pay for a click isn’t directly based on your bid. Your ranking is affected by it, but your payment is determined by the ad behind you and your quality score.
• Max Thomas: Last link building tip: Have Fun! Great content rocks.
• Michael Wyszomierski (from Google): [Regarding reconsideration requests] In most cases there is no penalty, it’s some other issue. That’s why we’re sending those messages now [saying there's no penalty].
• [Answering: What do you do when a company says "we know we have this problem on our site but we don't have the resources to fix it?]
  • Mikkel deMib Svendsen: It’s like a grocery store saying “we know we have bad food that’s old and rotten but we don’t have the time to throw it out.” Sorry. Building a website takes a lot of time.
  • Craig McDonald: The only thing I’ve seen that works is to bring in a 3rd party SEO expert, sometimes they’ll be able to convince where you can’t.
• [Multiple speakers:] More and more sites are getting hacked. Protect your sites! If you’re using WordPress make sure you’re always on the latest version.
• Patrick Altoft [on link building]: Going viral is great. But most people are better off focusing on who they need and giving things that are useful to them.

Best (actually, Worst) Mistakes

• Mikkel deMib Svendsen:
  • Site hosted by RackSpace, suddenly dropped out of search. RackSpace decided that the engines were taking up too much bandwidth so they blocked them all.
  • An engineer redirected all users who didn’t support javascript to a page saying “please install JavaScript.”
• Richard Baxter: Client wondered why they weren’t ranking in Bing. Turns out they’d been blocking the MSNBot for years because it was once overly aggressive.
• Jonathan Hochman: robots.txt file on testing server was blocking spiders. Accidentally moved that file over to the live servers, thus blocking their live site. [Find some other solution to keep spiders off your testing servers. Nothing that could accidentally move over].

Best Stats

• Greg Sterling: Google site links increase CTRs by 3X
• Daniel Ruby: #1 result on Google appears on Bing page 1 67% of time. Page 1 Google results show on Bing’s first page 43% time (hat tip: @wordtracker)
• David Burgess: 50% improvement using the charity’s email address versus a generic email for link building (hat tip: @keyrelevance)

Best Tweets

• maxormark: Another “missing” speaker. “A broken ankle” Didn’t know SEO was so dangerous

Best Coverage

• WordTracker’s Gareth Davies: SMX London: 221 Takeaways
• State of Search: SMX London
• Koozai’s Sam Stratton: SMX London Day 1 and Day 2

Got more? Help me out by Tweeting me at @GilR with more nominations (and corrections). Thanx!

Other coverage from SMX London:

For more in the series, see:

• Best of SMX Toronto 2011
• Best of SXSW 2011
• Best of SMX West 2011
• Best of SphinnCon 2011
• Best of SMX East 2o10
• Best of SMX Advanced 2010
• Best of WordCamp Jerusalem
• Best of 140: Twitter in Tel Aviv
• Best of SMX Toronto 2010
• Best of SphinnCon 2010
• Best of BirdBrain 2010
• Best of SMX Advanced 2009
• Best of PubCon 2009

Remarkably absent from all the coverage of blekko founder Rich Skrenta, his spam clock, and his anti-spam crusade is any discussion about how Rich telegraphed this play four years ago.

Rich Skrenta

He concluded that there was no opportunity for an SEO model:

“I’m thinking SEO has gotta be dead as a startup business model.”

Rich Skrenta, Some Thoughts on Mahalo, August 20, 2007

and that no other business model would work for most web sites

“companies will succeed by working within the framework of Google’s industry dominance”

Rich Skrenta, Winner Take All and Google, January 1, 2007

and

“We know this from experience: No one will ever go to Mahalo directly, just as no one ever went to About.com, dmoz, Tripadvisor, Nextag, IMDB or any other vertical or broad-but-shallow site. Google is where everyone starts“

Rich Skrenta, Some Thoughts on Mahalo, August 20, 2007

He noticed that the real money was in being a search engine

“It turns out that owning the starting point on the Internet is really, really valuable.

Not just because it gets a lot of traffic. It’s because that traffic is so much more valuable than the rest of the page views bouncing around the net. Google’s CPMs are $90-120, vs. $4-5 for an average browse page view elsewhere.”

Rich Skrenta, Winner Take All and Google, January 1, 2007

and that there would be an opportunity for startups in this space

“I believe Google is a fantastic company … But to bet against the search startup space is equivalent to betting that Google is going to bat 1000. And nobody ever bats 1000.”

Rich Skrenta, Search Isn’t Over, January 19, 2007

And finally, he determined the best positioning for a search competitor: claim that Google was being overrun by spam and that you had the solution:

“Spam created the market opportunity for Google, when Altavista succumbed in 97-98. Search startups should be praying to the spam gods for a second opportunity. :-)”

Rich Skrenta, 98% Spam, February 6, 2007

Ah, the wonder of the internet. An entrepreneur articulates his thought process on his blog, and you can sort of watch how the light bulb went on his head.

So, we rejoin our hero in 2011. How’s blekko doing?

“The last announcement we put out said we’re over 1 million queries per day.”

Matt Rosoff interviews Rich Skrenta, Google’s Garbage Search Results Are Not Going Away — Blekko CEO, March 25, 2011

The editor of that piece wrote “by way of comparison, Google gets more than 1 billion.” Well, a tenth of a percent of Google’s traffic is not a bad place for a search engine startup to be.

Are Skrenta’s dire warnings about the spam problem correct? I don’t think so. In their famous 1998 paper the Google founders complained that junk results had become a problem since “people are still only willing to look at the first few tens of results.” Imagine that, the success metric was finding a relevant result within the first few tens of results. Now we get annoyed if we have to scan the entire above the fold region.

Few people understand the internet industry as well as Rich Skrenta. But using him as a source for questions about Google being overrun by spam is like relying on oil companies’ analyses of alternative fuel. Skrenta is smart. But he may be praying to the spam gods.

As Aaron Wall from SEO Book demonstrated, Google’s algorithm changes change the course of the Web. Aaron took the negative approach by focusing on Collateral Damage, but there are certainly upsides as well as site owners develop their sites to succeed within the Google ecosystem.

Panda is likely to cause the following shifts in how the web evolves:

• More focus on user experience, including fewer ads hiding the content.
• Less use of non-unique content. This includes fewer
  • scraper sites
  • nearly identical sites that take standard product feeds
  • mashups, where sites create a unique experience by combining non-unique elements on a page.
• More community-generated content.

All told, Panda should make the Google ecosystem a better place. As Rich Skrenta pointed out years ago, Google is “the start page for the internet” and “companies will succeed by working within the framework of Google’s industry dominance.” Google’s choices regarding which behaviors to incentivize drive the web’s development.

Speaking from the perspective of product management of a large web site, I’m mostly happy with Panda. It better aligns our short & long term interests. Now on trade-offs between user experience and other business metrics we can factor likely Google traffic changes into our analyses. This better aligns our interests with our users’, and will help us make choices that improve quality of user experience. Ultimately this helps web publishers and users.

My biggest problem with Panda is that the feedback loop is even less direct and immediate than it was before. It’s like a pitcher throwing 100 pitches a game for a month and only afterwards getting some hints as to approximately how many of those pitches were strikes and whether or not he pitched well enough to win most of those games. Google may counter that a pitcher shouldn’t nibble at the corners, he should just throw the ball down the middle, but that’s not a viable path to success in a competitive environment. In our case, throwing the ball down the middle would be removing all ads and NoIndexing any of our pages on which any of the data sources that we license has been subsequently licensed by another site. It’s ironic that Google, which is so associated with fast iterations based on metrics-driven decisions, has created an environment where websites are flying so blind.

The other downside is that, as Aaron Wall pointed out, we’re in a never ending cat and mouse game. Google just incentivized a whole new set of behaviors. Matt Cutts had always said that Google wouldn’t use signals like clicks on search results because they were too spammable. Now they’ve reversed course. Todd Friesen (back when he considered himself a black hat SEO) once said that he loved real time search because it was so spammable. Well, those spammable factors are getting more significant. And the more Google uses social media signals, the more time spammers will spend manipulating those signals. Google’s use of social media signals may be good for Google, but it’s bad for social media.

The most discussed irony of the Panda update is that people thought it was to fight “content farms like eHow” and yet eHow emerged as one of the big winners. Panda rewards sites with unique content that are not over-monetized and that users find valuable, whether or not bloggers and competitors hate them, and whether or not they target searches that we find puzzling. So this was good for eHow. And IMO Panda’s use of user signals will have a positive influence on eHow, forcing them to provide sufficient quality to satisfy their users.

So Panda is a nice step forward for the web. It incentivizes sites to do the right things. It will force sites to focus more user experience. Its incentives for community generated content will also push the web in the direction of greater user participation. Ads won’t hide the content you’re looking for. Sites that don’t have comments sections or forums will consider adding them.

Many sites will respond to Panda by becoming better sites. Others will respond by manipulating these algorithmic shifts in ways that hurt the web. And around we’ll go until the next Google update.

Wow, what a conference. Great sessions, great smackdowns, and Matt Cutts is not the most mentioned person (this time that honor goes to Charlie Sheen).

Here’s the Best of the Best.

Best Lines:

• Todd Nemet
  • [Showing log files] Look at this, Google makes their crawlers work on Christmas. So much for Don’t be evil.
  • Oh, don’t take notes. Everything is up on line, with speaker notes, more examples. In fact you don’t need to be here, you can go now, I don’t mind.
• Vanessa Fox: It’s hard to re-read a book that you wrote. You never want to see it again. But I hope you don’t feel that way about it.
• Maile Ohye:  [sitting on the table to take questions] Sometimes I think that if I weren’t an engineer I’d love to sit on a piano and sing. So this is like a dream come true.
• Greg Boser:
  • AJAX is good for cleaning your toilets, it’s not good for SEO.
  • First Google didn’t like scraper sites and said we want you to write a lot of good content. Then some sites figured out how to do that on a massive scale and suddenly they were told no, that’s evil
• Steven Levy: Google’s competitors have done a great job spreading the search neutrality meme. You’d have thought that after what Microsoft went through they wouldn’t wish this kind of anti-trust scrutiny on their worst enemy. But it turns out this is exactly what they wished on their worst enemy.
• Stephan Spencer: Toolbar PR is mythology. It’s a random number generator. The true PR we don’t have access to.
• Byrne Hobart:
  • Congratulations fellow SEOs. We’re now Old Media! [According to many SEOs response to rise of large-scale content sites]
  • We’re not typical users. The fact that we know the difference between Google, the internet, and IE probably puts us in the top 10% of Google users.
• Danny Sullivan:
  • Perception can beat reality. Nofollow solved comment spam! At least Google’s PR issue with it. Problem didn’t go away. Anybody have a blog? Anybody still have comment spam? Who’s spamming a blog right now?
  • JC Penney will hire a new SEO so they have somebody to throw out the window when people find out what they’re doing again.
  • If we don’t rank first when you Google “Search 4.0” write to the NYT and tell them Google sucks. I can give you a whole list of pages that we should rank #1 for that we don’t if you want to report to the NYT.
  • If there’s a link from Charlie Sheen do you rank those higher? Lower? Depends on the time of day?
  • Anybody see Watson on Jeopardy? It was pretty awesome. They had this computer come in. “I will take that question for $1,000.” And beating the humans. But I couldn’t enjoy it because I knew this (Google) was possible. Watson is a joke compared to Google. And BTW it doesn’t crash all the time and have the crashes edited out. The search engines are just incredible creatures.
  • People don’t give links any more. Or it’s like, yeah, but I’ll NoFollow you. Or I’ll do it on the word “the.”
  • People come up to me and say “I used the same word twice on my page. Is that spam?” You perhaps don’t understand the true meaning of spam. In fact, if you’re asking if you are spamming, there’s an excellent chance that you are not.
  • [Asking the search engine reps] Do you recommend that sites hire SEOs so they can throw them to the wind when something bad happens?
• Kevin Lee: Figure out how creepy you want to be in extreme targeting. [Gave example of "still looking for that divorce lawyer?"]
• Conrad Avvo: [Runs a site (Avvo) that lists & ranks lawyers] Incidentally, Google personalization thinks that I’m a drunk driver whose getting a divorce and about to go to jail.

Best Lessons:

• Greg Boser: Google is no longer page focused. The days of Google determining what will or won’t rank based primarily on page-level analysis are gone. Overall content performance is the key.
• Conrad Avvo: Don’t pay agencies based on ranking reports. Incentivizes them to optimize for specific terms and to buy links to them.
• Jim Yu: Separate brand from non-brand performance. Reporting on them separately is critical for identifying issues.
• Steven Levy: A judge once ruled, and this is the reigning precedent, search results are an opinion and are protected as free space. Philosophically and morally Google is out to serve their users. And if publishers fall & rise that’s a byproduct.
• Todd Friessen: Make sure you’re talking to your PPC guys. They can tell you that certain terms are high ROI. You can tell them that you’re #1 for these terms and if they buy ads there most of the increased clicks will be on the free links. Make sure you’re learning from each other.
• Overall key takeaway: Panda (the latest Google update, formerly known as Farmer) is heralding a sea change, a new era where Google pays:
  • much more attention to signals about
    • User experience
    • Social factors
    • Overall quality of the sub-domain
  • and less attention to:
    • Links
    • Quality of individual page

Best Exchanges:

• Danny Sullivan: How do you recover if you were hit by Panda
• Vanessa Fox: I spent 25 hours writing an article about this ..
• Danny: Can you give it to us in a Tweet?

• Matt Cutts: What my colleague from Bing said
• Danny: Are you copying Bing? Just to be clear, if anybody duplicates what their colleague said, it’s not that they’re copying, it’s that they’re getting the feedback from the audience and picking up on the user signals. [A reference to the Bing / Gooogle copying smackdown in January]

• Danny: Now that Bing has 30% of the market, do you do things specifically to rank well in Bing?
• Vanessa: They mostly work the same
• Danny: Well, there’s a 2 week delay before Bing copies Google

• Matt McGee: Myth or Fact: Competitors can’t hurt you
• Stephan Spencer: They can torch you
• Greg Boser: No comment
• Stephan: He [Greg] can torch you
• Greg: The idea that it can’t harm you is a myth. I’ll leave it at that. … [Clarifies that he doesn't do this] … The upside is that it’s more expensive. … The cost involved in meaningfully impacting a competitor may be prohibitive now. Because it’s more on a page basis now, not a site basis
• Jill Whalen: Don’t your run the risk of actually helping them?
• Greg: In Bing [laughter]. Sometimes when you try to do these things you help your competitors because it helps them in Bing. Be nice to people.

Best Stats:

• Thanx to Greg Sterling who either said these or  Tweeted them while moderating:
  • Google’s Ken Norton:
  • 1 in 3 mobile searches have local intention
  • 80% of fortune 500 have not adjusted to mobile
• Gregg Stewart: 64% of local business searchers expect the business results to be within 15 minutes of their location (hat tip: Lisa Barone)
• Seeing 1 ad on YouTube increases tendency to buy a product by 40% (hat tip: Pamela Lund)
• Ty Downing(Facebook):
  • Biggest Facebook usage spikes tend to occur on weekdays at 11 a.m. – 3 p.m. and 8 p.m. ET (out of 1500 random streams) (hat tip: Brian Harnish)
  • 65% of Facebook users only access the site when they are not at work or school – typically early morning or evening (hat tip: Molly Ryan)
  • Reveal tabs (aka Like Gate) increase “Like”ing 50 – 90 % from
• Baris Gultekin:
  • 64% of Google searches have pages without exact matches to all query terms (hat tip: Danny Sullivan)
  • 44% searches on google have more than 3 words (hat tip: Danny Sullivan)
  • 20% of Google queries not seen before in past 3 months (hat tip: Danny Sullivan)
• Shari Thurow: 72% of mobile information needs are triggered by context.
• Manny Rivas: 84% of the share of universal video search results are YouTube videos, no suprise. In Bing, YouTube is only 38% (hat tip: ReelSEO)
• Angie Schottmuller:
  • 22% of Fortune 50 are using mobile barcodes.
  • Mobile barcode scanning was up 1600% in 2010.

Best Smackdowns:

• Will Scott takes on Yelp’s Dylan Swift. Dylan suggest people solicit reviews (though this is against Yelp’s official policy). Will’s angry reaction includes tips on how to spam Yelp. See Lisa Barone’s Up Close with Yelp.
• Meanwhile in the adjacent room Avi Wilenski gave an example of crowdsource-spamming the Google Instant results to restore the good name of a client tarnished by Google Instant results like “Bob’s Carpets scam.” Google Product Manager Othan Hannson made clear that Google viewed this kind of thing as an abuse to which they might take action. Avi said “give us somebody to talk to at Google and I won’t have to spam.” Got a big applause. [Google is the (mostly) benevolent (in my view) dictator of much of the Web. The strong applause helped indicate how much anger there is. And if you're asking, I think any attempt to change this situation will only make things worse.]
• Marshall Simmonds (with About.com since 1999 and then NYT) and Maile Ohye (Google) gave conflicting answers regarding using Canonical URLs for sending a paginated article back to the first page. Marshall said it was fine. Maile said no. “Believe me, if the page isn’t a subset of the other, it’s a signal. You can believe him or you can believe me, who was talking to our indexing people the other day.” She later Tweeted “Let’s be clear: it’s called rel=”canonical” not rel=”theme.” It’s for dups, very similar, or in rare cases a subset.”

Best Conference Tweets:

• @ShaneEubanks: Had a bit of a Freudian slip today in the expo hall at #smx when I asked the bing rep “what are you doing here?” oops…
• @toddmintz: The session speakers who follow @aimclear tend to display that “I just got hit by a truck look”
  • @SusanEsperanza: So do the livebloggers
• @oilman: good way to not be #1 for something anymore is to tell a room full of SEOs that you’re #1 for something

Best Serendipitous Tweets:

• When Google and Bing were discussing the potential that using Social Signals for search would lead to paid Tweets spamming the system, a sponsored Tweet from @problogger promoting eHow scrolled by.
• During the Do Not Track panel paidcontent.org Tweeted their just published article on the issue Just How Many Opt-Ins Will European Web Surfers Have To Wade Through?

Best Intros:

All from Danny:

• Next up we have Vanessa Fox who may sound eerily like Google, but she’s not of Google
• And Jill Whalen, owner of Trademarks on “I told you this in 1996” and “If you’d only listened to me in 1997”
• Rich Skrenta founded the Opening Directory. Now he bans the Open Directory.

Best Mistakes:

• Maile Ohye: People put a rel canonical and the URL it points to is disallowed in robots.txt. Oops.
• The imcharity site used for SMX charity events is ignored between events. It got hacked. People searching for it got cialis ads. Nobody noticed till right before the conference when it was pretty much too late. Result: online registrations way down from last year. Lesson: pay attention to your site even at off-times and keep it safe.
• Steven Levy: Throughout Google’s history there’s been panic that they’re getting big and have to change. In 2001 Larry & Sergey feared that they had too many middle managers and they were going to fire all of them. HR said NO. They did it anyway. And it was terrible. People wanted to be managed. They had to roll it back [Probably a more difficult rollback than a a failed site feature]

Best New Terms:

(New to me, anyway)

• Like-gating: encouraging a FB visitor to like you before they can see content on your FB page. (from Facebook’s Ty Downing)
• Like farms: Spammy groups on a social network that just like each other’s stuff (from Bing’s Paul Yiu)

Best Analogies:

• Steven Levy: I wrote a story about the Educational Testing System. They insisted that their tests couldn’t be gamed, coaching wouldn’t help. I did a story showing that their social scientists did studies proving the opposite, and they suppressed those. Ultimately the admitted it. When Google talks to you it reminds me of what the ETS says now. It’s not coaching. It’s teaching. You’re teaching how to expand your vocabulary and improve your math skills. Google says don’t design for us, just make your web site better.
• Danny Sullivan: People talk about this ideal world where the search engine just knows what’s good and you don’t have to do anything. That’s never happening. We’re never going to get that happy-go-lucky world. It’s like saying I want the WSJ to cover me and not researching that sites that WSJ does cover have PR firms that know how to pitch a story. And you sit back and say “No, the WSJ should find me.”

Best Stories:

• Steven Levy: Google people are suckers for metrics. I remember somebody was arguing something to the board. They asked him “how sure are you that this will work.” He said 87%. They went with it. I asked him where the number came from. He said he just made it up.
• Bruce Clay: I had 2 Fortune 500 companies come to me after losing their traffic on major keywords: 200K visits / day. Lost them. Big companies. In a little corner division of the company some guy got paid for ranking and traffic. So he bought links. It was all under the main domain. That little guy knocked out the main domain. Took forever to find it.

Other Favorite Moments:

• Maybe this is a standard euphemism but I loved how Justin Briggs referred to Targeted Links, including discussing how you want to run a content initiative at the same time that you acquire Targeted Links so that there can be a reason associated with the gain of links. It’s a fascinating charade. Sort of a conspiracy of silence, games, and euphemisms so we can play make believe that buying links isn’t a key part of SEO.
• The Ask the SEOs sessions used to be battles between Black and White. Now, with the same participants, it’s between “focus on users” and “focus on creating the footprint that sites that focus on users have.” It seemed that Vanessa found this debate far more frustrating than when her fellow panelists were simply black hats.

Best Coverage:

• SMX West 2011 by Lisa Barone (Outspoken Media)
• SMX West 2011 by Barry Schwartz (SE RoundTable)
• SMX West 2011 by Jessica Lee (BruceClay)

Got more? Help me out by Tweeting me at @GilR with more nominations (and corrections). Thanx!

[Dear reader: A funny thing happened with this post. I wrote it to show how it's OK for eHow's page to rank on "how to pour a glass of water" since it will only be found by people searching for that information. But then I was surprised to get a lot of search traffic to this page, using those keywords. Can you PLEASE tell me in the comments why you're here? Is your current method of pouring water not working for you? Are you switching from ceramic mugs to glass cups and want to make sure the methodology is the same? Are you looking for the latest technological advances and new best practices on the water pouring front? Are you trying to understand content farms or SEO? Please help! Thank you.]

Fantastic discussion about Content Farms at SMX West. Instead of a mob with pitchforks we got an intelligent discussion of what these sites do right, how they’ll likely evolve, and what other sites need to learn from them.

How to pour water

Matt McGee started by discussing how the definition of a content farm is elusive, seems to be associated with the following characteristics:

• Has a lot of ads
• Targeting search terms
• Large amount of content gets produced

Gave examples of ridiculous content on eHow such as

• How to make toast
• How to boil water

[Another speaker later added examples like

• How to pour water into a glass cup
• How to pour water from a pitcher

Now these pages are indeed funny. But what's exactly the problem, other than that some people's parents never taught them how to boil water? Or that people are inexplicably Googling for info about how to pour water? Yes, this page is funny (http://www.ehow.com/how_7551358_pour-water-glass-cup.html) but as long as it only shows up for searches inquiring about how to pour water, it's not nefarious, evil, spammy, or low quality. It's also an interesting display of an inherent flaw in using the social graph to rank search results. 50 people liked this page on Facebook. Google & Bing have no way to know that it was probably Liked as an example of useless content, so they bump up the score for this content thus risking that it will rank decently for some other searches as well.]

Matt says that these sites are associated with low quality, but in reality also often have higher quality content. He’s successfully relied on content from some of the sites that get labeled as CFs.

For the last couple of months there have been some complaints about Google decline in search quality and blaming it on CFs

More recently, the Farmer / Panda update. In the works since Jan 2010. Effects 12% of search changes. Still US only

Immediate analysis: clear losers, ezinearticles, Associated Content, hubpages

Others generally associated with CFs like eHow did not seem to suffer

Next was Luke Beatty, VP / GM of content at Y!, oversees the Y! Contributor Network, formerly known as Associated Content, which he co-founded

Y! Contributor Network has the same premise today as when Associated Content was founded:

Create an open democratic platform where people can publish content.

>400K contributors

Has multiple formats (text, video, etc) and both solicited and unsolicited content.

They try to push things onto the Y! network, sites like Shine, Y!Sports, Y!News, Y!Finance

Only fragmented content that doesn’t have a home gets published directly on AC.

Panda results: Google referrals up for 1/3 of their content, down for 2/3. [I suspect the Downs were generally of greater magnitude than the Ups. The primary Ups from Panda seem to be just the small rises that come from some of the sites above you getting smacked]

93% of the site’s assets remain indexed on Google [Very misleading stat. Issue is ranking, not indexing]

Changed on an Asset by Asset basis, not a property basis

Overall Y! Traffic is up. Particularly on Y! Answers. Other crowdsourced content went up.

Puts more pressure on contributors to create higher quality content.

Jury still out on whether or not Google is good at which of our content is higher quality than others. [That's one of the rubs of the Panda update. Seems to be based on factors, including paying for human judgments of a site's trustworthiness, that would be too noisy on a page level, and are only useful on a site level. So within a site, little correlation between quality and ranking.]

He discussed the evolution of Crowd-Sourced Content Distribution

2005: Context = Contextual Advertising

2008: Context = Hyper-targeted display. Ace Hardware wants 100s of articles on drills and drillbits.

Interesting stat: Consumer influence. 70% of the content you consume on a daily basis is crowd sourced (source EMC). 30% by someone you know (source PEW) [Includes Facebook, Twitter, etc]

2011: Context = Content Marketing. Brand relevant assignments. Create content that big brands want.

Getting a Walmart fan to participate in the conversations that Walmart needs to be in. “Moms like me” program.

Luke’s summary:

• If content farm = creating content at massive scale then fine.
• Just like eBay is commerce at massive scale.
• If I can verify that you’re human and writing reasonable content I can get your content in front of an editor.
• We’re committed to increasing quality

Next was Tim Ruder from Perfect Market.

He spent his career in marketing content for traditional media, starting in 1995 with Washington Post.

The values that traditional media bring to the table are important values to keep.

But the economics have some challenges

We can look at CFs and see what can be applied.

CFs vs Traditional Media

CFs:

1. Systematic processes to write large amounts of content
2. Data and tools to help them understand what people are interested in
3. Have data that help them understand what kind of commercial demand exists for each of those topics
4. Amortize over a long period of time

There’s an organizational disconnect among news sites.

They think in terms of sections. Health gets Health ads. Entertainment gets Movies. Main gets an odd mix of furniture ads, etc.

Need to think about Atomic content with people coming in directly from search engine & deep links

Newspapers: The daily miracle. Then you start fresh in the morning on the next day’s miracle. Content is not considered in the longer term fashion.

The knock against the CFs, which I think is rightfully placed, is that the content can be very low and thin.

The market pressure is forcing a change on that.

The Panda / Farmer update puts a lot more impetus on the need for quality for everybody, including CFs

Most of the mainstream press was not dramatically affected

CFs will improve their quality over time. [Agree wholeheartedly. The CFs know that long term success and stability requires significant improvements in quality.]

The news published today by NYT, CNN, Washington Post is being subsidized by the offline versions of those products

That subsidization gives those publishers a dramatic advantage. But it’s going to narrow. Fewer offline purchases, and the competitive landscape is changing.

Traditional publishers have to learn lessons from CFs

• Understand and leverage consumer interest
• Capture better revenue streams from thinking about content in a more granular way
• Understand durability of content produced
• Take full advantage of the moment in time that these publishers have today to put these things in place before their competitors get better and better and we lose what we have from those mainstream publishers

Next up was Byrne Hobart from Blue Fountain Media

How did we get here? Where did we start?

The old style: Supply Media

• Content based on outside sources: News, Art, etc.
• Monetization: get a big audience sell ads, hope they notice
• This is a great way to create content that content creators like

“Demand” media

• Give people what they want. That’s happened for years; we’re just targeting better
• This is what pop music is, etc.
• Monetization: Highly targeted contextual ads
• RPM of $13.45 for 2010 up 26% (from Demand filing) [I need to check this]

Worst case scenario for a publisher is that you solved the user problem. Better is to give just-enough content and get the user to click on ads for the full answer. [This is another key issue within CFs. Are you creating Mahalo coupon pages that rank and force users to click on ads to get coupons? Or pages with the content users are looking for? I think eHow usually does the latter.]

Ridiculous economies of scale.

Great for old domains or for celebrities: Rachel Ray, Lance Armstrong, Tyra Banks. Celeb gets the attention. Then long tail, scalable, highly monetized. They already have celeb-driven sections for food, health, and beauty. Don’t be surprised to see somebody like Suze Orman do a personal finance site for them. [Makes me think of what Calacanis called “the façade strategy” regarding Huffington Post. The site has an impressive face. But the real money comes from mass produced long tail content that gets traffic because of the celebrity involvement on the showcased pages.]

Winners:

• Anyone with more money than time
• Online absentee landlords
• Lots of nontraditional employees (Demand has a lot of stay at home moms)
• Searchers looking for ultra long-tail terms
• Shareholders

Losers?

• Anyone with more time than money
• Traditional media
• Active bloggers
• Anyone trying to get into the industry
• Searchers looking for head terms

Congratulations fellow SEOs! We’re now Old Media [Ha!]

This is the Industrial Revolution hits content creation: Content Factories [I love this analogy. And Content Factories was the original term coined by Wired.com. 2 months later ReadWriteWeb changed the term to Content Farms, presumably to make them sound more sinister.]

• Separates content creation from site ownership
• Separates content creators from the marketing process
• More efficiency

[Assembly line vs one proud owner doing it all. Differentiated labor, straight out of Adam Smith, Henry Ford, etc.  Insulting to the craftsmen of the earlier generation. And a key difference is that we may love mass-produced cars and sweaters, but in the case of content, we need differentiation in the result, not many duplicate copies of the same content.]

Biggest labor arbitrage since China, since there’s so much unexploited time [very interesting point. Sort of insourcing, instead of going abroad getting people from home]

Common responses from SEOs:

• Complaining to the press
• Complaining to Google
• Complaining on Twitter

Better options:

• Shoot for quality
• Sneak by with Social Media
• Dust off your e-mail newsletter
• Made- for Made-for-AdSense. Buy ads that will get you on those sites and satisfy those users. Google will figure this out. This is how to be ready for that.

Finally, Matthew Brown from AudienceWise, the former director of search strategy at the NYT (including About.com, the granddaddy of this model)

Weren’t specifically targeting CFs.

They’re looking for low quality across any network.

eCommerce, comparison shopping, scrapers all got hit.

How we got here: Domain authority

• Ability to rank thousands of pages by domain trust / time
• Longtail rankings add up to a critical mass of content

NYT empty topic pages ranked very well, for example http://topics.nytimes.com/topics/news/business/companies/apple-valley-bank-and-trust-company/index.html

Two views of Farmer (aka Panda) update:

• A sitewide filter on domain authority
• Farmer factor applied to the site’s normal ranking ability

Domains that got hit got hit across the domain

Possible signals:

• Quality vs Quantity ratio
• Big sites relying on domain authority
• Small sites with few quality pages
• Sites with an overload of ads / links

Survivors have thatElusive brand smell:

• Google news inclusion
• Blog search inclusion
• Tweets / shares / likes/ Reddit / dig
• Balanced ratio of deep links vs index page links

“eHow got unfairly swept up in this” [That’s backwards. Demand's pre-IPO media campaign that began with the Wired article about eHow started the backlash that has now hit many sites, but not eHow]

How do I get out of this? (or not get into this)

• Clean your site up
• Build out brand signals
• Channels / domains
• Tighten editorial focus
• Scale promotion

Q&A

Q: Do Content Farms have a bad model or is it just an execution problem

A: Byrne: The problem is the quality of the content. Yeah, it’s an execution problem.

Q: Are there other content suggestions recommended to help with top rankings?

A: MattB: I like video. Other things that can get you into the 1-box: images, get into news or at least into blog search. Any signals to show you’re not just targeting a random keyword phrase.

Q: MattM: What’s Google’s role in this? Are they responsible for all of this?

A: Byrne: Sure. It’s not a perfect algo and there will always be an arms race. In quantitative terms the experience hasn’t been degraded. They’re not doing multiple searches or bouncing more. CFs are probably good for Google. We’re not typical users. The fact that we know the difference between Google, the internet, and IE probably puts us in the top 10% of Google users.

MattB: They’re business model relies on user trust and their brand. The money coming in from CFs isn’t worth compromising Google’s trust and brand.

Q: Quality signals: A lot of these relate to the design of the website. Is that more of a factor now?

A: MattB: I think it is. Matt Cutts  used to be asked are you looking at bounce rates, etc., and he said it’s too noisy and too easy to game. I don’t think he’s said that recently. They may be able to extract stuff out of that noise. The Chrome extension. They’re going to start looking at those signals. This is going to get worse as it goes on for sites with poor quality feedback. eHow didn’t get slammed, because their site design is fairly clean even if people think their content is shaky.

Q: Y!CN strategy going forward:

A Luke: Surface the best CS content and push it to Y! properties. We’re not trying to compete with the high quality sites, we’re trying to cover what isn’t covered.

Q: If you have been hit and can remove low quality pages should you remove it all at once or phase it out?

MattB: I’d do it all at once. Remove the pages. Redirect if that makes sense. Get rid of pages that have lots of ads & links and very poor content. Nobody knows how long it takes to get your traffic back. We’ll start seeing that data.

Q: These CFs rank well even for head terms. Should we avoid them from a linking perspective?

A: Byrne: If they rank, probably not

Q: Is writing 20 very similar articles still unique but w/ different titles spam?

A: Byrne: Probably not from a user experience.

Luke: Unless the user is being pushed a Related Content bucket with those articles

Q: Did paid search have any role in this update?

Luke: We’ve never paid for traffic so I don’t know

MattB: Don’t think it was related at all

Tim: No impact.

[In summary, great session. You can and should learn a lot from content factories, in terms of how to create evergreen content to meet the demands of consumers and advertisers. These factories will become higher quality, or they will go away. And they know it.]

Also see the session coverage from:

• Dustin Woodward: Summary of Content Farm Session – SMX 2011
• Barry Schwartz and Keri Morgret: SMX Live: Content Farms Or The Smartest SEOs in the World
• Lisa Barone: Content Farms Or The Smartest SEOs in the World

“Joel Spolsky is changing the world. Maybe not your world, but the world of programmers and software companies and the people who work for them.”

Opening lines of Seth Godin’s Tribes

When last we left our hero (whom BTW I finally met last week when the FogBugz / Kiln tour hit Tel Aviv), Stack Overflow had just brought in an all star team of VCs and we were wondering whether or not Stack Overflow could grow beyond the programmers community.

Joel Spolsky just published Stack Overflow’s 2010 year in review. The key takeaways:

• Stack Overflow grew to 16.6 million unique visitors per month. Their traffic is still about 90% SEO.
• Stack Exchange is up to 1.5 million uniques, and growing nicely.
• 80% of their questions have at least one answer that received at least one vote.
• They raised $6m earlier this year.
• Spolsky is still funny, insightful, and always worth reading.

Joel posts a nice graph showing Stack Exchange’s growth

Stack Exchange growth

He writes “If, as planned, we continue growing at 51% a month, we will be bigger than Facebook in 15 months.” Nice. He’s too modest to mention that not long after they’ll have more visitors than there are people on the planet!

But a slightly different view emerges when you look at their top sites (courtesy of http://stackexchange.com/sites)

Stack Exchange sites

Some points:

• All of their Stack Exchange sites other than Stack Overflow combine to less than one tenth of Stack Overflow’ traffic.
• The next four sites (not including the one about Stack Exchange itself) are all for computer power users.
• Then, in 7th place, at one tenth of Stack Overflow’s usage is the Computer Gaming site, which is hitting an overlapping demographic.
• Behind that, with 0.2% of Stack Exchange’s visits, is their Mathematics site (also a somewhat overlapping demographic), which they tout the way Demand Media touts LiveStrong.com, as though it’s evidence that something other than the primary model is working.

Here’s another way to look at the distribution:

Stack Exchange Sites Pie Graph

OnStartups vs Quora

One thing Joel left out of the year in review is any mention of revenue. Joel and Jeff did a good job of documenting their monetization failures but never let us know about any revenue successes. $6 million is a lot of money. But 27 employees, nice big New York office, plenty of other expenses. Would be nice to know their financial situation.

To sum up, StackOverflow had a great year in terms of traffic, content, and community. And they’ve grown beyond programming to other types of computer professionals and enthusiasts. Can they grow beyond that? I don’t know. Maybe they don’t have to.

What a week for content farms!

Here’s the brief overview:

• On Friday Matt Cutts acknowledged some influential bloggers’ criticisms that Google had a faustian deal with Content Farms that were making Google worse. IMO Matt’s statement boiled down to:
  • You’re wrong, Google is getting better, not worse, and we keep trying to keep up with “users’ skyrocketing expectations”
  • You’re wrong, these sites aren’t spam
  • You’re wrong, Google doesn’t give better rankings to sites that run AdSense
  • We hear you though, and now that we’ve made so much progress on spam we’ll turn more attention to “content farms” which we define as “sites with shallow or low-quality content.” Google made two big changes in 2010 to combat these sites.
• A slew of bloggers and journalists came out with posts covering Matt’s post. Most celebrated that Google was finally going to knock out sites like eHow (which BTW, apparently gained traffic during the 2 changes Matt talked about), and that this would cripple the Demand Media IPO. Here are 2 more insightful posts (IMO) on the issue: Michael Martinez’s What is a content farm? and Aaron Wall’s Google gears up for relevancy changes.
• Demand Media had their IPO, which was even more wildly successful than most expected. It ended its first day with a market cap of $1.9 billion.
• Mahalo did a second pivot from their “human powered search” model. They didn’t even acknowledge their previous ridiculous claims of Mahalo 2.0′s overwhelming success (not). They announced that they’re doubling their staff this year and will be creating 2,500 videos per week in a model that seems copied directly from eHow.
• Stack Overflow gave their 2010 year in review. More on that on Monday. Short answer is that they’re doing a great job creating content for programmers and other computer enthusiasts and professionals. They’re not successful yet moving beyond that, but they may not have to be.

Nobody calls Stack Overflow a content farm but I include them here to try to define what a content farm is and isn’t.

The term “content farm,” like the term SEO, is often used by its critics to refer to every spammy tactic that they don’t like, or that cause other sites to rank above theirs.

Stack Overflow and eHow both meet the following criteria often used to identify content farms:

• Creates thousands of new pages every day
• Gets the vast majority of its traffic from Google. You might even say they view Google’s results page as their site’s home page (Stack Overflow co-founder Joel Spolsky in fact did say this).
• Content-creation mechanism naturally creates pages where demand has been identified.

The term “content farm” was coined to refer to the systematic and efficient creation of thousands of in-demand pages every day. eHow does that. So does Stack Overflow.

Matt Cutts just defined content farms as “sites with shallow or low-quality content.” By that definition, I don’t think they’re going to punish eHow. Neither does the stock market.

“Prices will rise, politicians will philander, you too will get old, and when you do you’ll fantasize that when you were young prices were reasonable, politicians were noble and children respected their elders.”

– Wear Sunscreen, Baz Luhrmann

“Human beings seem to take a morose pleasure from believing that once there was a Golden Age, some lost Eden or Camelot, or superior ancient civilization, peopled by heroes and demigods, an age of greatness long lost and irrevocable.”

– George Will, Men at Work

This weekend saw a flurry of articles bemoaning the decline in the quality of Google‘s search results, most notably:

• Vivek Wadhwa in Tech Crunch: Why we desperately need a better Google
• Alan Patrick: On the increasing uselessness of Google
• Jeff Atwood in Coding Horror (and reprinted in Business Insider): Trouble in the House of Google
• Anil Dash: Three’s a Trend: The Decline of Google Search Quality

Clearly, Google doesn’t always produce the results we’d like. So, is Google in decline or are people comparing Google 2011 to a mythical Golden Age of Google that never existed? I don’t know. But AFAICT neither do the people who are making those claims.

AFAIU here’s the basis for the aforementioned authors‘ claim of decline:

• In a 2009 article Paul Kedrosky noted problems with certain Google searches. All four pieces cite this article as the primary source. Quotes from that article make up about a quarter of Dash’s piece, and more than a third of Patrick’s piece. It merits 5 links from Wadhwa’s post. This is a post from 2009! How did this become proof that Google is currently undergoing a decline?
• Atwood’s primary claim is that Google started ranking pages that copy others’ content. To show how bad this is, he writes “when was the last time you clicked through to a page that was nothing more than a legally copied, properly attributed Wikipedia entry encrusted in advertisements? Never, right?” Meanwhile, Patrick writes the opposite “The other main scamsite type is one that copies part of the relevant Wikipedia entry and throws lots of Ads at you.” Atwood is way off in citing this as an example of Google’s decline. A few years ago there were many sites that copied Wikipedia content and sometimes outranked it. Google cleaned that up. In fact, we (Answers.com) incorporate Wikipedia entries in our reference source (legally, with full attribution, to serve our users, and as part of a long-standing and mutually beneficial relationship with Wikipedia in which we support their activities). Since 2007 we’ve put a NoIndex on our pages that only have Wikipedia content, because in 2007 Google greatly reduced the amount of traffic it sends to sites that index pages that are just copies of Wikipedia. As for content scrapers outranking the content source, that’s been a big problem for us (and I presume others) for years. Atwood is noticing it now because now it’s happening to him. Atwood’s main proof of decline is actually an area where Google has been improving.
• They cite some specific examples where their results were disappointing. But AFAICT none of us know whether or not those results were better or worse a few years ago.
• They cite each other as proof.
  • Patrick is mostly based on Kedrosky’s 2009 piece.
  • Atwood writes “I can’t help noticing that we’re not the only site to have serious problems with Google search results in the last few months. In fact, the drum beat of deteriorating Google search quality has been practically deafening of late.” This deafening drum beat of articles in the last few months is 5 articles, including 2 articles from 2009 and Patrick’s and Wadhwa’s that quote them.
  • Dash’s “Three’s a Trend” uses Kedrosky, Patrick, and Atwood as proof of Google’s decline.

So more than a year ago Kedrosky had problems with a certain type of search, and this weekend Patrick found Kedrosky and Atwood found Patrick and Dash found Patrick and Atwood.

Ross Hudgens’ Tweet

What we don’t have is any real indication that things were better a few years ago and then got worse. Where I differ with these authors is not in our opinion of today’s Google.  Google is extremely flawed. For example, a bug caused them to ban the world’s greatest blog (this one) for five months. But Atwood writes

“The idea that there could be something wrong with Google was inconceivable to me. Google is gravity on the web, an omnipresent constant; blaming Google would be like blaming gravity for my own clumsiness. It wasn’t even an option.”

Really?

Google (like Jeff Atwood and probably those other authors) is a significant intelligence and talent trying to make sense of a complex and confusing world. They get it right far more than others do, but they’re often wrong. Are they getting it wrong more often than they used to? I don’t know. But as AFAICT, neither do any of the people who wrote these articles.

My guess is that the perceived decline is only relative to a mythical past. What do you know about this? Anybody have any evidence indicating a decline or an improvement?

BTW, also see Andrew Goodman’s Search Isn’t Broken Because One Guy Had Trouble Using Google.

Update (Jan 22, 2011): Matt Cutts just chimed in.  Here’s my summary: Matt Cutts on Search and Spam.

Give Business Insider credit on their coverage of Demand Media. They’ve published insightful analyses, ranging from highly critical to extremely bullish. The best by far was Suite101 CEO Peter Berger’s Hater’s Don’t Understand Demand Media which provided useful insights into the model Demand and Suite101 employ.

Peter Berger

[Disclosure: Like Berger, I work at a company (Answers.com) that uses some similar strategies to Demand]

Berger enumerates how life is different for companies using content models oriented around evergreen content that meets the demands of search users and search advertisers:

• Advice content has a very long shelf life. “… articles on Suite101 tend to display a pattern of continuous traffic growth. Our writers can typically expect increasing revenues from their articles on Suite101 for years.”
• Search audiences are very open to action-oriented messaging targeting their objectives. These content pages are highly valuable to niche, non-brand advertisers. For these sites brand advertising is a nice to have incremental lift, not the center of their monetization strategies.
• “Given current online users’ behavior and preferences, receiving the majority of search traffic from Google is a sign of normality … what content businesses must do to ensure their future relevance is invest in the content quality search engines want to present to their users …”
• “… when users are searching, the fewer clicks it takes to achieve their objectives the better (and more valuable) is their experience … ‘quality’ can not be captured by simplistic engagement metrics. Search-driven sites like About.com, StackOverflow, Answers.com, Suite101 and Demand Media will always show low page views per user and extremely high percentage of traffic coming from search engines. StackOverflow gets about 90% of their traffic from Google, which it says it views as their site’s home page. This is not an indicator of low quality, and people who attempt to denigrate Demand Media because of these stats are completely missing the point (or deliberately misleading their readers).

On a related note, Demand Media just re-filed for their IPO, clarifying their accounting practices of amortizing their content creation costs over 5 years. Henry Blodget advises Demand Media to drop the bogus accounting. He may be right that Demand should conform to the more accepted accounting practices and expense their content creation. But it’s not bogus accounting. As Peter Berger makes clear, unlike most companies’ archived content which is yesterday’s news, Demand’s archived content is the company’s primary asset, and it remains valuable for many years. Amortizing its creation costs may be unconventional, but it’s probably the most correct thing to do.

Previous articles on this subject:

• Demand Media: A Story in 5 Numbers
• Large-Scale Content-Creation Sites
• 2009 Top SEO Smackdowns: Arrington vs Demand Media
• Quality is Still King