How to Manage a Pilot Project

“As someone who makes buying decisions for clients, I can honestly say that the biggest
barrier to closing sales right now is a fear of being sucked into a never-ending, no-win project
that turns into a corporate embarrassment. Vendors all make the same promises, the same
promises that have been made for many years, and no one believes these promises any more.
That’s why a trial or pilot program is so important. It helps the client figure out if this is a
classic bait-and-switch situation, or perhaps just another case when programmers have
absolutely no idea of how to come up with a truly cost-effective business solution.”
That’s how marketing consultant Kristin Zhivago describes the growing sense of skepticism
among buyers of big-ticket software. It’s no secret that a painfully high percentage of software
installations fail (or partially fail) during enterprise-wide rollouts. In fairness, the blowup is
often the fault of customers with oddball hardware environments and dysfunctional internal
systems. But the bottom line is that more buyers than ever now insist on pilot projects--small-
scale, carefully-managed installations at a few of their sites. If the “proof-of-concept” works
as promised, then they’ll talk about writing a million-dollar check.
Sounds reasonable, but pilot projects are notoriously tricky to manage. Developers
sometimes find themselves giving away hundreds of hours of free engineering work to
customers who aren’t seriously committed to wider implementation. And sometimes the test
itself fails to resolve critical questions about scalability and integration. “We do everything we
can to avoid pilot projects, including walking away from the sale,” says Dick Werner of Shaker
Computer & Management Services. “They’re usually a disaster.”
Still, a well-run pilot installation may provide just the nudge a skeptical customer needs to
make a firm buying decision--or to walk away from a deal that was probably doomed from the
start. We recently asked Soft•letter readers and friends to share what they’ve learned about
running successful pilot projects. Here’s their advice:

• Get high-level buy-in: “I think the most important issue is to get a significant
commitment of your customer’s staff time, which means you need a decision maker
who has authority to allocate these resources,” says sales consultant Chuck DeVita. “To
get that access, the deal size has to be significant. Also, once you have the
commitment for their time, it often evaporates unless they have sufficient skin--
money--in the game. My experience is that free trials have a low probability of success
because there’s no commitment on the customer side.”

Chuck DeVita, president, Growth Process Group, Box 1266, Los Altos, Calif. 94023; 650/917-8010. E-
mail: cdevita@growthprocess.com

*THE FINE ART OF PILOT PRICING

There are endless opinions about how to price pilot projects. Since the client
presumably believes that the project is a long shot, some marketers argue in
favor of deeply discounted (or even free) foot-in-the-door pricing. Others point
out that raising prices after a discount can be enormously difficult.
Hark Vasa, president of PDSC, says his company has developed pricing
guidelines for CRM pilots that take a middle-of-the-road approach to
discounting. His advice:
• License fees: “Vendor and client both need to make a commitment for the
 pilot. However, if the vendor believes that the pilot project is relatively risk free,
they can afford to charge the client a relatively small fee for the pilot--
approximately a 75% discount for the software license for the pilot phase.”
• Other fees: “All other out-of-pocket expenses, such as consulting,
customization, data conversions, and training services, should be charged at a
20% discount from standard rates during the pilot phase.”
• High-risk pilots: “If the pilot is viewed as a highly risky project, the cost should
be split equally between the vendor and the client. If the pilot is successful, the
client should also pay for the vendor’s share of the pilot costs, since the client is
now assured of the benefits of a full rollout.”
• Recalculation: “The agreement should also state that if the pilot is successful
and a full rollout is carried out, all of the fees during the pilot will be
recalculated and charged at the normal contract rate. However, if the pilot phase
is viewed as unsatisfactory (by the client and vendor) and product use is
discontinued within 30 days of that decision, the vendor should refund all license
fees and only keep fees and expenses for the discounted professional services
during the pilot.”
• Customization: “If customization is necessary to achieve the desired results
during the pilot, then it must be included in the project plan. Otherwise, users
may say that the pilot phase is unsuccessful. In general, however, customization
should represent less than 20% of the projected cost of all subsequent
customization for the full rollout.”

• Leverage training sessions: Dorai Thodla of iMorph says seminars and training classes
can often lead to pilot projects and ultimately to major sales. “When people undergo
training courses for certification, or attend a bootcamp on a new technology like XML
or .NET, it’s a good time to offer a discount for a pilot to implement what they’ve
learned.” Since the company is sending people to learn about a technology topic,
Thodla adds, “it’s reasonable to suggest that a pilot will reduce the risk of
implementation.”

Dorai Thodla, iMorph, 18961 Sunnybrook Ct., Saratoga, Calif. 95070; 408/528-8888. E-mail:
dorait@msn.com

• Identify a “must-do” task: “The trick to selling a pilot project is to find something
that absolutely must be done,” says Kristin Zhivago. “For example, while a company’s
leaders may balk at the multi-million-dollar price and company-wide disruption
associated with a full-scale CRM project, they probably won’t mind the creation of a
single database for e-mail addresses. The ‘thing that must be done’ is for the company
to embark on opt-in e-mail marketing, which is impossible without a common database.
Meeting that need is the first step towards solving a bigger problem, but if you tried to
solve the bigger problem first, nothing would happen.”

Kristin Zhivago, principal, Zhivago Marketing Partners, 381 Seaside Dr., Jamestown, R.I. 02835;
401/423-2400. E-mail: kristin@zhivago.com
 • Keep the contract simple: Formal contracts are essential to define commitments on
both sides, says Gareth Jones of the Software Productivity Center, “but make them too
complex and you’ll be hung up with the customer’s lawyers for too long.” The essential
issues, he says, are “the scope and duration of the project, objectives and
expectations, and the pricing terms. In addition, make sure you include ‘get out’
clauses for both sides--you don’t want to get caught in a never-ending saga.”

Gareth Jones, vice president of sales and marketing, Software Productivity Center, 460-1122 Mainland
St., Vancouver, B.C. V6B 5L1; 604/662-8181. E-mail: gjones@spc.ca

• Avoid pricing surprises: The cost of a pilot project should be clearly defined up front,
argues marketing consultant Tracy Weatherby. “Don’t nickel and dime the customer.
Bundle all the core parts of the software and make sure that ‘options’ are truly
optional. If you cause the customer to go back to get more budget, you won’t get the
trial.”

Tracy Weatherby, principal, Active Ingredient, 1927 Wilson Court, Mountain View, Calif. 94040;
650/917-0360. E-mail: weatherby@activeingredient.com

• Give the customer an ongoing stake: “One successful tactic I’ve seen used in smaller
organizations that were looking for a foothold in certain vertical markets,” says
Adobe’s Mark Pieper, “is to pitch the initial customer on paying for development in
return for a 10% rebate on each additional sale of the customized solution. This
encourages referral selling, generates a great ROI for the customer, and establishes a
customer base for future updates and development.”

Mark Pieper, vice president of sales/ROW, Adobe Systems, 321 Park Ave., San Jose, Calif. 95110;
408/536-4668. E-mail: mpieper@adobe.com

• Limit the liability: “Contracts for pilot installations should always include some
language specifying that the software is provided ‘as is,’” says attorney Michael
Krieger. “You’re dealing with a real sales transaction and real customer data, so most
of the potential liabilities come into play that you’d have on a full-scale sale--maybe
more, since a pilot test is inherently risky. Of course,the sales guys won’t want to see
their deal killed by an ugly contract, so you’ll probably want to keep the liability
language as simple as possible. But you want some defense if the customer claims that
the software trashed their production data or didn’t perform as promised.”

Michael Krieger, partner, Krieger & Nunziato,10920 Wilshire Blvd., Los Angeles, Calif. 90024; 310/428-5208. E-
mail: mkrieger@mcimail.com

• Narrow the scope: “The goal of a first pilot should be a quick and solid win,” argues
Kristin Zhivago. “The pilot should have a minimum of customization--just enough to get
the job done and to prove out the concept.” Similarly, the trial period itself should be
fairly brief, probably no longer than three months. “By the end of the three months,
everyone will know where the project is headed. If it’s likely that the customer will
need further customization, that should be spelled out in the original agreement, with
well-defined project modules.”
Kristin Zhivago, principal, Zhivago Marketing Partners, 381 Seaside Dr., Jamestown, R.I. 02835; 401/423-2400.
E-mail: kristin@zhivago.com.

• Test specific points of vulnerability: A good pilot test shouldn’t just demonstrate “big
picture” success, says independent developer Will Fastie--it should also prove that “all
the little widgets” can be implemented reliably. “For each widget that isn’t well
understood by the customer, work up a proof of concept for just that widget that helps
give confidence about the whole project. Conversely, if a widget doesn’t prove out, an
area of risk has been identified and can be explored more fully. There’s no sense
building a house if individual bricks are going to crumble.”

Will Fastie, consultant, 7110 Sheffield Rd., Baltimore, Md. 21212; 410/340-1049. E-mail: will@fastie.net

• Watch for “customer experience” problems: Sometimes pilot projects prove that a
product works from a technology perspective, but users nevertheless resist full-scale
implementation because of clumsy design. Blake Wise of Vividence, a usability testing
firm, says it’s possible to spot these issues by “testing prototypes early and often.” To
simplify the prototyping process, he adds, it’s usually best to host test versions on the
Web and then “send target customers to review and interact with the prototypes.”

Blake Wise, senior marketing manager, Vividence, 1850 Gateway Dr., San Mateo, Calif. 94404; 650/645-5023. E-
mail: blakew@vividence.com.

• Develop deep relationships with users: Autodesk has been working with a group of
pilot sites to help fine-tune a new product, and senior design strategist Doug Look says
“one of the keys to success is the strong relationships we’ve developed with our pilot
participants. We gained valuable input from visiting sites in person and presenting
specific ideas about the project. We listened to user concerns, and we evolved the
application by constantly testing and reassessing as we go.” Developing strong
relationships with pilot testers doesn’t happen automatically, Look adds. “Sometimes it
was a struggle to get a sustained level of interest and feedback from our participants.”

Douglas Look, senior design strategist, Autodesk, 10 Brown Rd., Ithaca, N.Y. 14850; 607/266-7247. E-mail:
douglas.look@autodesk.com

• Try to profit from customization: Not all customization is bad, Carl Rogger points out.
“If it’s customization that junior developers can do and that ties the client to you more
profoundly, the more the better,” he says. “If it’s work that you were planning to do
anyway, that’s great also. It’s also great if you own or co-own the custom work and can
resell it. However, it’s not good if a one-time sale consumes your best development
resources or if you have to staff up with new, untested staff.”

Carl Rogger, president, KIS Information Systems, 20 Kerrian Rd., Oroville, Wash. 98844; 604/490-8117. E-mail:
crogger@kispayroll.com

• Monitor variable costs closely: “I’ve never seen a development project where all the
costs are understood in advance,” says pricing expert Jim Geisman. “A pilot may look
like a breakeven project early in its lifecycle, but a few miscalculations can make it
very expensive.” In particular, costs often get out of control when developers try to
meet customer performance specs. “Feature development can be estimated with high
 precision,” he says, “but usually no one has a clue what it will cost to deliver a certain
level of performance. That’s where you may want to negotiate some shared risk with
the customer.”

James Geisman, president, MarketShare, 35 Main St., Wayland, Mass. 01778; 508/647-0330. E-mail:
jgeis@moreshare.com

• Don’t over-promise on customization: “Set clear expectations that this is a pilot of a

general-market product, not custom development for one potential customer,” says
Todd Richman of Broad Reach Associates. “This trips up a lot of companies. Encourage
feedback, but don’t suggest you’ll incorporate it all before release. Some input may be
acted on immediately if it’s critical; other ideas can be slotted for future releases; and
for some you smile politely and put it on the ‘when hell freezes over’ list. I’ve had to
remind pilot users about the terms we agreed to up front, to get them back on track.
It’s in their interest as well as yours that you can bring a robust product to market as
quickly as possible.”

Todd Richman, president, Broad Reach Associates, 183 Willis Rd., Sudbury, Mass. 01776; 978/440-7600. E-mail:
tr@broadreachassociates.com

• Leave nothing to chance: James Mauch of Tenmast Software says his company
“makes sure that testers have a complete set of documentation before we install the
software, and we schedule training as well.” In addition, “the test itself requires a
specific set of test objectives that are included in the documentation and training. To
make sure these objectives are considered by the testers, we prepare an evaluation
form and give it to the testers during training.”

James Mauch, president, Tenmast Software, 132 Venture Ct., Lexington, Ky. 40511; 606/455-8061. E-
mail: jamesm@tenmast.com.

• Don’t over-sell: Because pilot projects “are priced attractively and have a short sales
cycle,” says Kurt Nguyen of Qnexis, sales reps sometimes end up neglecting bigger
deals that take more work. It’s important to keep an eye on the whole sales pipeline,
he says, and to make sure that easy-to-sell pilots don’t create “a state of
complacency” in the sales force.

Kurt Nguyen, president, Qnexis, 46548 Hollymead Pl., Potomac Falls, Vir. 20165; 703/421-6769. E-mail:
kurt.nguyen@qnexis.com.

• Manage the support transition carefully: “During pilot projects,” says support
consultant Rick Kilton, “customers usually have special access to engineers and high
priority for tech support. Users get attached to all this special effort and begin to feel
very special.” However, once the product goes into general release, he points out,
“pilot users are often dropped like hot potatoes.” The result can be badly bruised
feelings and, worst case, high-profile public complaints about how badly they’ve been
treated. One solution, says Kilton, is to promise users “some kind of reward for their
efforts,” such as free products or ongoing high support priority for a period of time.
“It’s important that the pilot customer knows very well what will happen, how it will
happen, when it will end, and what happens after that.”

Rick Kilton, principal, RWK Enterprises, Box 1809, Lyons, Co. 80540; 303/823-6448. E-mail:
rkilton@rwkenterprises.com.
 • Tie success to full implementation: Sami Jajeh of Abovo Marketing suggests that “if
possible,” the contract for the pilot should lead directly to a previously-negotiated full
rollout. “Determine upfront what the go/no-go criteria are, so that once you have met
the pilot objectives, the software license agreement automatically goes into effect,”
he says. “Worst case, you should have the exact criteria defined, so you can go back
and ask for the full order.”

Sami Jajeh, executive vice president, Abovo Marketing Group, 100 Ashford Center N., Atlanta, Ga. 30338;
678/597-3311. E-mail: sjajeh@abovomarketing.com

• Lay the groundwork for a full-price sale: “We settle the price issue right at the
beginning,” says John Scott of The Realm, a developer of real-estate management
applications. “We’re willing to give away a certain amount of consulting for free, which
our customers know is a meaningful contribution. But then we charge full price for the
product, and if at the end of the test the customer offers a price that isn’t palatable,
we’ll pass.” That doesn’t happen often, Scott adds, because his company also uses
pilots to quantify the customer’s return on investment and to cultivate internal
champions for the sale.