Nucleus to SAP: Gotcha!

Steve Hamm

And we thought the war of words between software giants SAP and Oracle was bad. Now the German company that dominates the corporate application market is trading verbal jabs with a feisty little market research outfit, Nucleus Research, which challenges SAP’s advertising claim that its customers are much more profitable than others. Tech ROI-doubter Nicholas Carr has weighed in--demanding that SAP release the study it based its claim upon. The spat highlights the risks of making bold claims about technology’s effectiveness. As it stands right now, SAP’s stature may be hurt more than helped by its marketing campaign.

A sampling:

Bill Wohl, SAP spokesman: “We understand what’s going on here. They’re a third-tier analyst firm that’s out to get publicity.”

Ian Campbell, CEO of Nucleus: “All they have to do is publish their research and let everybody examine it. If I was them, I’d worry about the FTC, or the equivalent in Europe. But they’re saying, ‘You’ve just got to trust us.’”

Back to the beginning. Late last year, SAP was looking for a way to back up its advertising tag line, The Best-Run Businesses Run SAP, and decided to commission a study to see if it could come up with some proof points. It hired a Stratascope, a small Massachusetts research outfit, to crunch the numbers, and, ultimately, came out with the marketing claim that its customers are 32% more profitable than others. It was an eye-popper.

Nucleus didn’t react until last week. Partly it was because the company had a fire in its office, and just didn’t get around to it. “We have a lot of clients who are SAP customers. Some of them were asking us if this claim was true,” says Rebecca Wettemann, Nucleus’ vice-president of research.

Nucleus has made a name for itself by publishing skeptical reports about the claims of enterprise software companies. In the past, it has targeted Siebel Systems,, and SAP. Its stock-in-trade is to vacuum the names of customers off software companies’ Web sites and then interview them to see if they’re thoroughly happy with their supplier. Often, they’re not. Ouch! In this case, Nucleus found 81 customers mentioned on SAP’s Web site and did calculations to determine their return on equity, comparing it to peers in their industry segments. Its findings: “SAP customers had an average ROE of 12.6% compared to an industry average ROE of 15.7%.” That meant, it claimed, that those customers had an average ROE that was 20% lower than their peers.

SAP’s reaction was swift and strong. “It’s junk science,” says Wohl. He says that basing a conclusion on the results of 81 customers out of SAP’s more than 30,000 total is not statistically valid, and he questions Nucleus’ methodology. “It’s an apples to rotten-oranges comparison,” he says.

Here’s how the two sides did their studies:

Stratascope started with the universe of 4,600 NYSE and NASDAQ companies. It took out financial services companies, which don’t measure performance the same way other companies do, and came up with 580 SAP customers and 2,800 non-SAP customers. It measured and compared operating profitability. The average operating profit margin for SAP customers was 10.3%, compared to 7.8% for non-SAP companies, or a 32% difference.

Nucleus didn’t have a list of SAP customers for easy comparison, so it just used those named on the SAP Web site. Using financial numbers from Bloomberg and peer segmentation from Hoovers, it compared the return on equity for those 81 customers with that of their industry peers. In addition to publishing an overall average, Nucleus published a table comparing each of the 81 with their peer group.

Each side criticizes the other’s study.

In addition to saying the 81 companies is too small a sampling to be accurate, SAP says it’s inaccurate to use ROE without taking out extraordinary charges that might distort a company’s true performance. Wohl says SAP won’t release the details of its own study because it doesn’t want competitors to get their hands on its customer list.

Nucleus’ Wettemann says that while the sample size she used is small, it’s still valid. It yielded a plus or minus accuracy range of 13%. So, even if the calculations are 13% off on the low side, SAP’s customers would still be 7% less profitable than average. “I can’t compare methodology, process, or data with them. I don’t know what mystery data they used,” she says.

Bruce Brien, CEO of Stratascope, defends his study, but has no comment on Wettemann’s. He says he’s comfortable with the way SAP used his conclusions. “They’re making an implication that my numbers can’t prove, but it’s a marketing message. Companies do that all the time,” he says.

On Nick Carr’s blog, Rough Type, he calls on SAP to make its research public. “We get the tagline, not the facts,” he writes.

I called up Bruce Richardson, a top enterprise software analyst at AMR Research, to get his reaction to the spat. He calls Nucleus’ approach “ambush research.” But, at the same time, he’s highly skeptical, in general, of software companies' claims when they tout performance results for their customers. He says, most often, the cost of technology is a tax that companies pay for being able to do business efficiently. He hadn’t made a close comparison of the methodologies of the two studies, but says measuring operating earnings seems like a more accurate way to go. “I don’t know if either study is misleading, but, I’ll say this, keeping vendors honest is something we all need. Rebecca’s right to raise questions.”

The dispute has been ping-ponging around the blogosphere and online Web sites for several days, and it’s not clear yet if it will become a big problem for SAP. If it heats up, the company might be forced to respond with more than just words. I understand its reluctance to hand over its detailed study and reveal its customers to the competition. One possibility: Give the study under non-disclosure to a reputable accounting or statistics professor and let them study it, and also study the Nucleus report, and issue a ruling on the validity of the two reports.

Before it's here, it's on the Bloomberg Terminal.