NONSENSENUMBERS

Wil Harris

Using authenticity, fairness and transparency to beat the benchmarks

Customers don't care about bench- So how do you engage with that kind of marks. Get used to it. tit for tat, lowest denominator advertising? And if savvy customers don't care OK, let's qualify that. Customers don't about your benchmarks, what do they care about your benchmarks. In today's care about? corporate world, where consumers are bombarded with hundreds of advertise- The answer is - disengage. Then, reenments a day, Brand X saying they're bet- gage on your own terms and serve your ter than Brand Y doesn't mean anything customer, not your competitor. any more. There's no trust. Customers are saying - we don't believe you. Identify who your competitor is marketing to when they're pitching At least, the savvy ones are. The down- benchmarks. Then market better. side of the scenario is that the stupid customers - and there are more stupid peo- What is synonymous with better? Authenple in the world than savvy people - often tically. Transparently. Fairly. The followbuy into advertising. This means that ing is a starting point for a benchmarking 'headline-grabbing' benchmark adver- and marketing campaign that engages on tisements - "Brand X is 90% faster than those terms. Brand Y in Test Z!" - will always still have an impact.

Problem 1: Communicating the performance of server products to an audience of server buyers.

that they trust. Sure, you're not going to win every benchmark. But you'll win most of them, right? And you can at least keep score of the overall total. Move the goalposts on the benchmarks, too - be authentic. Performance, as we understand it today, isn't about raw performance. It's about heat, cost, space and power, so solicit the savvy server community input on designing benchmarks, then try them out. Sometimes they'll work, sometimes they won't. At worst, the people buying the technology will understand better the problems trying to compare it. At best, you'll come up with a metric that makes sense to a lot of people and shifts the goalposts away from stupid SPEC numbers to real-world benchmarks suites, designed by server savvy people, to help server savvy people.

Wil Harris

Here's the issue: companies love their SPEC numbers. SPEC numbers are easy to put on a graph. They're also easy to manipulate, and anybody can make a SPEC say anything. Companies who use SPEC continue to give credibility to something which should, in a world of authenticity and transparency, deserve none. To overcome an opponent playing on SPEC, disengage.

Then reengage - and do so with insane gusto. Savvy server buyers are their own masters. They believe that they understand, better than anyone, what their own needs are. They don't trust companymade benchmarks, although SPEC numbers are attention grabbing. Do something that that grabs their attention more Involve your buyers, your savvy transparently, and more fairly, and leads server community, in your performto a more authentic result. ance metrics and they will buy into your brand. In this case, transparency means a blog. An Intel Server Performance Analysis Disengage from inauthentic, attention blog. Give an employee or two a really simple brief - to take AMD and Intel chips, benchmark them, and publish the results. And the methodology. And photos of the rigs. And videos of the benchmarks in progress. And thoughts about problems with benchmarks. And notes about hardware quirks. And details of what they had for lunch. Solicit input from the server savvy community in the Comments section of the blog, and implement them. Run a high-profile campaign asking for input from these guys. Don't aim for headline grabbing single numbers from single benchmarks. Get your team to design workloads that accurately reflect real-world performance situations, and then get them to fairly benchmark the chips in these situations. If a Commenter on the blog points out a discrepancy, or something unfair in the test setup, go back and re-do the test. Involve the savvy server community in the testing, and then you can give them back results that they are invested in and

grabbing headlines and reengage with authentic ones. "Intel wins 20/30 tests of server performance designed by, and transparently conducted in association with, savvy server buyers in a community wide participation exercise in authentic benchmarking" isn't quite as snappy as "15% in SPEC", but that's just down to the turn of phrase. Authentic beats inauthentic.

Problem 2: Communicate the transparent, are not fair and authentic, performance of desktop prod- and thus will be found lacking by the customer. ucts to consumers
As for the problem of undefinable perThe fundamental problem with desktop formance - don't push them raw numprocessing, from a competitive point of bers, push them dreams, aspirations view, is that almost all desktop proces- and capabilities. sors are now fit for general desktop use. In the high-end world of computer enthuIt's almost impossible for an average siasts, raw performance has long since consumer to sit down and really tell been dropped in favour of capability testthe difference between an AMD and an ing. In other words - assuming it's fast Intel processor in the things that they enough, what more can it do? Apply this use their computers for 75% of the metric. time. Disengage from the tit for tat and reenHere's the second problem: aside from gage on your own terms, better. Transbeing mostly irrelevant to general usage, lating Intel's superior capabilities into any performance related benchmark is better-than-benchmark benchmarks will going to be accused of being biased. do more for the consumer than performance based benchmarks. What's the answer? To be more authentic, transparent and fair. The good news Where is Intel superior? Process techis that this is easier in the desktop space nology, for example. Take a $200 procthan it is in the server space. Unlike in essor from each company, then measure servers, where independent testing is how much electricity it uses. Assuming difficult and expensive, there are hun- one (on 65nm) is better than the other dreds of independent publications world (on 90nm), tell consumers that whatever the performance benefits, Intel processors are better for the environment. Intel changes socket platform less often than AMD, having been on LGA775 for ages. That means easier upgrades and less wastage, again better for the environment. The environment is hot right now, in case you hadnt noticed. (In more ways than one, natch). AMD has fallen into the same trap that Intel did in the Pentium 4 days - marketing based on a number. In those days, AMD successfully shifted wide that already do testing and already the game from the megahurtz race to cahave trust with the consumer. Utilise pabilities (think the XD-bit and 64-bit). those brands to do your talking for you. Now, it's gone back to touting numbers so play them at their own game and shift A good publication will publish all its back to capability benchmarking. It's a benchmark methodologies and will have more representative view of the conevolved them, in conjunction with the sumer experience, it's easily verifiable by community, to a point where they are al- independent means and it talks to conmost beyond reproach. Communicating sumers in language that they can underperformance, then, is simply a case of stand. In other words, its more transparrepublishing what they say in a manner ent, more fair and more authentic. that is fair. That means quoting fairly, providing a balanced view of editorial coverage, but proving your point overwhelmingly - assuming that's what the consensus is. Taking quotes out of context, or making a benchmark look better than it might be, or using a publication of ill repute - all of those things, whilst

Wil Harris 3

Problem 3: Communicating to non-savvy business people the Intel advantage over AMD.

Wil Harris

The problem - these are exactly the kind of high-powered types that fall foul of sheep-driven marketing. The kind of people that will authorise millions of pounds of budget because they saw somewhere that AMD was faster than Intel in a 'benchmarket'. How do you communicate with these kind of people when they are precisely the kind of people that are won over by headlinegrabbing half-truths?

much does it cost to equip a server room with Intel processors compared to AMD? How much will the next upgrade cost? That's one way of approaching the money game. Alternatively, flip that argument on its head - "Last year, server savvy firms spent $x million on Intel processors and $x million on AMD processors. What's the reason?"

And the real kicker? Turn their own lack of tech savvy on its head, and laud them for it. These people delegate what they don't know, and know what they do know, and thats why theyre paid a lot of money. So put it in these terms - "This year, x million spotty kids bought Intel The answer is simple - be more processors to spend 8 hours a day in headline-grabbing whilst simultane- World of Warcraft. They don't know your ously being more authentic. How on business, but they know Intel's procesearth do you do that? sors. Harness their geek, so you don't have to." Everybody trusts a computer geek, because, usually, computer geeks are amongst the most blunt - and authentic people around.

Benchmarketing conclusions
All of these things require moving out of the comfort zone. They all require disengaging from traditional marketing methods and reengaging in new ones. But heres the rub - you dont win an argument by tit for tatting back and forth. You win by moving the goalposts to a point that your competitor cant continue to By playing on their predispositions, that's play the game. Its a more tricky play, but how. Somebody reading the Wall Street its the ultimate end-game. Journal is already predisposed to the Wall Street Journal. They already trust it The bottom line is this - the best way enough to pay money for it. What the t o b e a t i n a u t h e n t i c , f l a w e d WSJ says carries weight, more weight benchmarking-based advertising is to than an advert. So combine the two. employ advertising that is authentic, What was the last thing Walt Mossberg transparent, fair, and which moves the said about Intel technology? Probably benchmarking goalposts. something flattering in relation to the latest Apple notebook. So take that quote, Anything else is just playing the game. put it on a full page ad, and give it a suitable tagline, like 'No Nonsense Numbers. Beyond Benchmarking.' Anybody reading that ad is going to give it more weight than anything AMD can put out that For illustrations, check out the seminal quotes SPEC numbers, because readers Hugh McLeod at gapingvoid.com. of the WSJ trust the WSJ more than AMD. Communicate in language they understand - money. People at the high end understand money more than they understand performance, so communicate in that language, authentically. How