Silicon Valley has had no shortage of fast-talking salespeople trying to push the promises of big data over the years. It'll transform your business, let computers figure out the problems and make everyone's life easier, they say.
Until recently, good data-analysis software had been insanely expensive and practically required a warehouse full of pricey computers to get it to work. Now, big data is finally here at many big companies, but it hasn't yet turned into the revolution the tech industry predicted. Instead, big data is helping companies to do relatively small things, such as unlocking new ways to review customer information, highlight business inefficiencies that were once invisible and allow companies to do the things they were doing before at a faster pace.
For example, Allstate makes a device called Drivewise that plugs into a car's dashboard and reports vehicle information, such as speed, braking and when the car is in use. Customers can request the gizmo and let the insurance provider sift through their driving data to identify whether they stay under the speed limit, avoid braking abruptly and don't often drive at night, which is when more accidents occur. Allstate, the largest publicly traded auto and home insurer in the U.S., rewards good eggs with lower insurance rates.
More from the Buried in Big Data special report:
- The Big Data Dump: How Info-Hoarding Can Overwhelm Startups, Spy Agencies
- Retailers Use Big Data to Turn You Into a Big Spender
- 10 Surprising Ways Your Daily Life Is Feeding the Big Data Beast
- Saving the World? How Big Data Is Tackling Everything From Cancer to Slavery
This may not sound like a particularly impressive use of big data, but finding trends within minute-by-minute driving reports quickly, accurately and without spending a ton of money couldn't have been done before. The result is potentially a whole new way to structure insurance. Traditionally, rates were set based on the customer's demographics with penalties added after a collision.
"We've had a really good reception from it," says Mark Slusar, a fellow of big data research for Allstate. "All drivers can inherently be safe drivers. We reward those who are safe."
To manage the data load, Allstate adopted a system called Hadoop. The software, made by the Palo Alto, California-based startup Cloudera, helps the insurer to pull together information reported by customers and staff from across the U.S. into one database that's flexible and fast. Other large businesses are also turning to Hadoop for many of their database needs over pricier alternatives.
"In the old model, there's a big buy in," said Intel Vice President Jason Waxman. "You've got to buy a big, expensive vault and hope to fill it with valuable information."
Companies are swimming in so much data that they don't know what to do with them. By the end of next year, global Internet data traffic will pass a zettabyte (a billion terabytes) with more than half of it soon coming from devices that aren't personal computers, according to estimates by Cisco Systems. But big data has been slow to catch on in the business world, particularly for smaller companies that had gotten burned by complicated and expensive hardware that ended up gathering dust.
"Most of the data on the planet is generated by machines in the daily course of their operations," said Mike Olson, Cloudera's chief strategy officer. "Data is so much more ubiquitously available these days and is coming at such a faster pace. This wiring of the planet is going have a huge impact."
While Hadoop is helping many companies to crunch data in ways they previously weren't able to do, the system doesn't run on autopilot. Using it effectively requires technicians to install it, and statisticians to come up with experiments and analyze the results. Hadoop, said Waxman, "isn't a substitute for smart data science."
"It's not a magic fix," he said. Companies "are going to need people who understand what they're looking for."
The need to hire experts or bring on consulting firms has been one big barrier to attracting smaller businesses. So Cloudera is working on making Hadoop simple enough for anyone to set up and generate reports, Waxman said. In a vote of confidence, Intel invested $740 million into Cloudera in March, part of a funding round that valued the company at $4.1 billion.
In the meantime, large organizations with quants on staff are using the latest data-analysis software to bring in more types of information and crunch it faster.
The National Cancer Institute is using a version of Cloudera's Hadoop to cross-reference the relationship between 17,000 genes and five major cancer types by digging through 20 million write-ups from various biomedical publications. Medical researchers have been using mainframes to crunch numbers for decades, but Hadoop provides new ways to sift through the data.
Western Union is using Cloudera to analyze transaction data from customers sending money back and forth around the world to search for patterns. The findings help the financial giant minimize fraud and to sell customers on additional services based on their habits and location.
Sure, big data can reveal new potential revenue streams, but it's not always worth the investment when companies have other things to worry about. A couple years ago, Nokia touted how it had tracked phone usage patterns to predict which apps customers would call up next and organize icons on the home screen accordingly. That small insight, which could make a smartphone slightly smarter, didn't help to slow the Finnish phone maker's sales slide in the face of competition from Apple and Samsung Electronics.