If Selling Out to Google Isn't a Good Business Model... |
| Amazon's Newest Product: Storage
March 13, 2006
Nucleus to SAP: Gotcha!
And we thought the war of words between software giants SAP and Oracle was bad. Now the German company that dominates the corporate application market is trading verbal jabs with a feisty little market research outfit, Nucleus Research, which challenges SAP’s advertising claim that its customers are much more profitable than others. Tech ROI-doubter Nicholas Carr has weighed in--demanding that SAP release the study it based its claim upon. The spat highlights the risks of making bold claims about technology’s effectiveness. As it stands right now, SAP’s stature may be hurt more than helped by its marketing campaign.
Bill Wohl, SAP spokesman: “We understand what’s going on here. They’re a third-tier analyst firm that’s out to get publicity.”
Ian Campbell, CEO of Nucleus: “All they have to do is publish their research and let everybody examine it. If I was them, I’d worry about the FTC, or the equivalent in Europe. But they’re saying, ‘You’ve just got to trust us.’”
Back to the beginning. Late last year, SAP was looking for a way to back up its advertising tag line, The Best-Run Businesses Run SAP, and decided to commission a study to see if it could come up with some proof points. It hired a Stratascope, a small Massachusetts research outfit, to crunch the numbers, and, ultimately, came out with the marketing claim that its customers are 32% more profitable than others. It was an eye-popper.
Nucleus didn’t react until last week. Partly it was because the company had a fire in its office, and just didn’t get around to it. “We have a lot of clients who are SAP customers. Some of them were asking us if this claim was true,” says Rebecca Wettemann, Nucleus’ vice-president of research.
Nucleus has made a name for itself by publishing skeptical reports about the claims of enterprise software companies. In the past, it has targeted Siebel Systems, Salesforce.com, and SAP. Its stock-in-trade is to vacuum the names of customers off software companies’ Web sites and then interview them to see if they’re thoroughly happy with their supplier. Often, they’re not. Ouch! In this case, Nucleus found 81 customers mentioned on SAP’s Web site and did calculations to determine their return on equity, comparing it to peers in their industry segments. Its findings: “SAP customers had an average ROE of 12.6% compared to an industry average ROE of 15.7%.” That meant, it claimed, that those customers had an average ROE that was 20% lower than their peers.
SAP’s reaction was swift and strong. “It’s junk science,” says Wohl. He says that basing a conclusion on the results of 81 customers out of SAP’s more than 30,000 total is not statistically valid, and he questions Nucleus’ methodology. “It’s an apples to rotten-oranges comparison,” he says.
Here’s how the two sides did their studies:
Stratascope started with the universe of 4,600 NYSE and NASDAQ companies. It took out financial services companies, which don’t measure performance the same way other companies do, and came up with 580 SAP customers and 2,800 non-SAP customers. It measured and compared operating profitability. The average operating profit margin for SAP customers was 10.3%, compared to 7.8% for non-SAP companies, or a 32% difference.
Nucleus didn’t have a list of SAP customers for easy comparison, so it just used those named on the SAP Web site. Using financial numbers from Bloomberg and peer segmentation from Hoovers, it compared the return on equity for those 81 customers with that of their industry peers. In addition to publishing an overall average, Nucleus published a table comparing each of the 81 with their peer group.
Each side criticizes the other’s study.
In addition to saying the 81 companies is too small a sampling to be accurate, SAP says it’s inaccurate to use ROE without taking out extraordinary charges that might distort a company’s true performance. Wohl says SAP won’t release the details of its own study because it doesn’t want competitors to get their hands on its customer list.
Nucleus’ Wettemann says that while the sample size she used is small, it’s still valid. It yielded a plus or minus accuracy range of 13%. So, even if the calculations are 13% off on the low side, SAP’s customers would still be 7% less profitable than average. “I can’t compare methodology, process, or data with them. I don’t know what mystery data they used,” she says.
Bruce Brien, CEO of Stratascope, defends his study, but has no comment on Wettemann’s. He says he’s comfortable with the way SAP used his conclusions. “They’re making an implication that my numbers can’t prove, but it’s a marketing message. Companies do that all the time,” he says.
On Nick Carr’s blog, Rough Type, he calls on SAP to make its research public. “We get the tagline, not the facts,” he writes.
I called up Bruce Richardson, a top enterprise software analyst at AMR Research, to get his reaction to the spat. He calls Nucleus’ approach “ambush research.” But, at the same time, he’s highly skeptical, in general, of software companies' claims when they tout performance results for their customers. He says, most often, the cost of technology is a tax that companies pay for being able to do business efficiently. He hadn’t made a close comparison of the methodologies of the two studies, but says measuring operating earnings seems like a more accurate way to go. “I don’t know if either study is misleading, but, I’ll say this, keeping vendors honest is something we all need. Rebecca’s right to raise questions.”
The dispute has been ping-ponging around the blogosphere and online Web sites for several days, and it’s not clear yet if it will become a big problem for SAP. If it heats up, the company might be forced to respond with more than just words. I understand its reluctance to hand over its detailed study and reveal its customers to the competition. One possibility: Give the study under non-disclosure to a reputable accounting or statistics professor and let them study it, and also study the Nucleus report, and issue a ruling on the validity of the two reports.
TrackBack URL for this entry:
Listed below are links to weblogs that reference Nucleus to SAP: Gotcha!:
SAP CUSTOMERS 20% LESS PROFITABLE ACCORDING TO NUCLEUS: JUNK SCIENCE AND ROTTEN ORANGES ACCORDING TO SAP... from Maneuver Marketing Communique
For the past several months SAP has been aggressively promoting a study showing that SAP customers were 32% more profitable [Read More]
Tracked on March 13, 2006 11:06 PM
It seems to me that both Jeff and Ian make good points. Perhaps a broader perspective is warranted. One problem with enterprise software is that, like air, its value is difficult to measure. Without air, life as we know it would not exist. That makes it both priceless and of no value at the same time. Without enterprise software, Citibank would not be able to accept credit cards, Qwest could not place calls, American Airlines could not book reservations. Clearly SAP applications adds great value to all those organizations - otherwise they would have stopped using them long ago. Yet the value each of the three organizations receives from SAP is different and difficult to aggregate. In our data driven times where attention spans are shorter than a New York minute (I am in New York today by the way) it is tempting to use stock price as a proxy for value. That can be very misleading. Since May ChevronTexaco shares have increased by about 14%. Both SAP and Hurricane Katrina were factors in that increase. To what extent did SAP contributed to that increase, to what extent did Hurricane Katrina? Perhaps the augment should not be about the measuring tools but the measurement itself.
Posted by: Bruce Daley at March 22, 2006 05:13 PM
Seemingly, it's an audition for Nucleaus' successors. The do's and don'ts of the internet are still being shaped as I write. 'You can't hit what you can't see' and staying in the loop by not doing things to attract attention ultimately keeps damage under control. Shame shame, they will all know your name the more you trip up in the writingsphere.
PS Love your Writing!
Posted by: Brian at April 12, 2006 02:07 PM
I don't know about all industries but for the airline and aerospace industry, SAP has been a desaster. The very public (in the Financial Tims) failure of SAP at British Airways where the solution failed to go live twice, went over two years late, cost over 250M GBP and eventually didn't implement the full solution is but one of several examples. At Lockheed they finally just killed the project and the GAO documented that the NAVY has wasted over $1B on SAP. At airlines like American, how can you say any ERP is adding value when you use it just for FI and HR and 80% of the rest of the technology footprint is best of breed solutions. JAL, SIA and ANZ who finally did get their SAP solutions live, usally after 4-5years refuese to endorse SAP. So when someone does a real scientific study I'd suggest two elements - include customers who never got SAP live and stratify the functional footprint - if the right hand side of the equation (benchmark) is operating profit then operating technology needs to be evaluated.
Posted by: michael denis at April 13, 2006 12:02 PM
I don't understand why SAP is afraid to let its competitors see their customer list. If SAP's customers are so profitable why would they switch to another's company's software ?
Posted by: Krell at July 21, 2006 12:58 PM