Netflix, Reed Hastings Survive Missteps to Join Silicon Valley's Elite
On a normal weeknight, Netflix accounts for almost a third of all Internet traffic entering North American homes. That’s more than YouTube, Hulu, Amazon.com, HBO Go, iTunes, and BitTorrent combined. Traffic to Netflix usually peaks at around 10 p.m. in each time zone, at which point a chart of Internet consumption looks like a python that swallowed a cow. By midnight Pacific time, streaming volume falls off dramatically.
As prime time wound down on Jan. 31, though, there was an unusual amount of tension at Netflix. That was the night the company premièred House of Cards, its political thriller set in Washington. Before midnight about 40 engineers gathered in a conference room at Netflix’s headquarters. They sat before a collection of wall-mounted monitors that displayed the status of Netflix’s computing systems. On the conference table, a few dozen laptops, tablets, smartphones, and other devices had the Netflix app loaded and ready to stream.
When the clocks hit 12 a.m., the entire season of House of Cards started appearing on the devices, as well as in the recommendation lists of millions of customers chosen by an algorithm. The opening scene, a dog getting run over by an SUV, came and went. At 12:15 a.m., around the time Kevin Spacey’s character says “I’m livid,” everything was working fine. “That’s when the champagne comes,” says Yury Izrailevsky, the vice president in charge of cloud computing at Netflix, which has a history of self-inflicted catastrophes. Izrailevsky stayed until the wee hours of the morning—just in case—as thousands of customers binge-watched the show. The midnight ritual repeated itself on April 19, when Netflix premièred its werewolf horror series Hemlock Grove, and will again on May 26, when its revival of Arrested Development goes live.
Netflix has more than 36 million subscribers. They watch about 4 billion hours of programs every quarter on more than 1,000 different devices. To meet this demand, the company uses specialized video servers scattered around the world. When a subscriber clicks on a movie to stream, Netflix determines within a split second which server containing that movie is closest to the user, then picks from dozens of versions of the video file, depending on the device the viewer is using. At company headquarters in Los Gatos, Calif., teams of mathematicians and designers study what people watch and build algorithms and interfaces to present them with the collection of videos that will keep them watching.
Netflix is one of the world’s biggest users of cloud computing, which means running a data center on someone else’s equipment. The company rents server and storage systems by the hour, and it rents all this computing power from Amazon Web Services, the cloud division of Amazon.com, which runs its own video-streaming service that competes with Netflix.
It’s a mutually beneficial frenemy relationship. Over the years, Netflix has built an array of sophisticated tools to make its software perform well on Amazon’s cloud. Amazon has mimicked the advances and offered them to other business customers. President Barack Obama’s data-fueled reelection campaign, for example, was run almost entirely on Amazon with the help of code built by Netflix engineers.
While Netflix started out as a DVD-by-mail rental service, it’s now striving to become something far more complex: an entertainment power on par with HBO, if not HBO’s parent company, Time Warner. Netflix plans to lead the shift to delivering television-style programming over the Internet and has developed sophisticated technology to support that transition. The company has invested hundreds of millions of dollars in original series—House of Cards, Hemlock Grove, Arrested Development, Orange Is the New Black, a Ricky Gervais show called Derek, and Turbo: F.A.S.T., a kids show co-produced with DreamWorks Animation—to become a major player in Hollywood. “We think of the technology as a vehicle for creating a better, more modern experience for the content we have,” says Chief Executive Officer Reed Hastings. “What we’re really competing for quite broadly is people’s time.”
About 18 months ago, Netflix and Hastings were spending much of their time trying to save face. Netflix had awkwardly unveiled plans to raise prices and separate into two companies—a DVD mailer called Qwikster and a streaming entity still under the Netflix name—and lost millions of customers in the process. The share price fell from $298 to $52.81.
After issuing a flurry of apologies, Netflix has mounted one of the all-time great comebacks. House of Cards arrived to mostly spectacular reviews, while investors were equally enthusiastic about the company’s first-quarter results. Revenue rose 18 percent from the same period last year to $1.02 billion, while the company added 2 million subscribers in the U.S. alone, dispelling widespread fears that its growth had slowed. Shares of Netflix are back above $200. It’s one of the best-performing stocks of the year.
Hastings doesn’t have an office. He moves around headquarters meeting with people and plopping down at spare tables to deal with e-mail. When he needs a quiet place, he heads to his watchtower, a room-size glass square built on the roof of Netflix’s main building. To get there, you climb a staircase to the roof and walk along a narrow walkway past air conditioning units and other machinery. Usually someone turns on the A/C in advance. The room can get hot since it’s basically a greenhouse with a round conference table and spectacular views of the Santa Cruz Mountains.
As the watchtower’s A/C blasts on a late afternoon, Hastings, 52, sits in one of the suede chairs, exuding calm. He’s goateed, skinny, and sounds extremely Californian when he talks about Netflix’s prospects. “We are trying to set this up as a continuously learning organization,” he says between mouthfuls of granola. “My role is creating that learning atmosphere.”
There was much learning in 2011, during the Qwikster episode. In a Saturday Night Live skit, Jason Sudeikis played Hastings apologizing to consumers while at the same time unveiling increasingly complex businesses, culminating with Nutqwakflikster—a nut, insurance, and movie seller. “We know you hate us,” the faux Hastings concludes.
Qwikster was a fiasco, but far less threatening than a debacle that preceded it. In August 2008, Netflix’s technology infrastructure melted down. This was when the company was still known for DVDs-by-mail, and for three days it could not send discs because a crucial Oracle database kept malfunctioning. Reporters and customers took notice. Netflix traced the problem to an expensive, third-party storage system that went haywire after a software update. The incident still annoys Hastings. When the subject comes up in the watchtower, Chief Product Officer Neil Hunt, who’s also gathered at the table, suggests they not mention the storage-system vendor by name. Hastings responds, “Let IBM have it, baby.” (An IBM spokesman declined to comment.)
Hastings and Hunt have worked together on and off for 25 years, meeting when they were both in the research division of Schlumberger, the oil field-services contractor. According to Hastings, Hunt is responsible for Netflix’s technology. “It’s mostly listening to Neil,” he says. “He’s on a first principles basis with this stuff. It’s not like I saw some golden tablets and then went forward.” Still, engineers at Netflix recall Hastings presiding over a number of meetings following the 2008 shipping disaster. He warned that a similar technical issue on the streaming side of the business would be even more devastating. And Hastings began to see that as the streaming business grew, Netflix would need ever more computing horsepower. It could hire an elite team of data-center engineers and build its own computing centers—à la Google, Microsoft, and Amazon.com—or it could move everything to the cloud.
Netflix began to experiment with cloud services from Amazon and Microsoft, where Hastings served as a board member. In 2009 he bet his company’s future on Amazon. Up to that point, nothing the size of Netflix had placed so much of its crucial technology on Amazon’s systems. Hastings sent an e-mail to Amazon CEO Jeff Bezos, announcing his plans. “I asked him if he was comfortable with that idea,” Hastings says. “If not, there was no point going forward.” Bezos gave the go-ahead.
At any moment, Netflix draws upon 10,000 to 20,000 servers running in Amazon data centers somewhere. The computers handle customer information, video recommendations, digital rights management, encoding of video files into different formats, and monitoring the performance of the systems. When a new device like an upgraded Xbox or a Samsung smartphone comes along, Netflix uses thousands of extra servers to reformat movie files and deal with the new users. By day, some servers handle the grunt work tied to streaming video; by night, they’re repurposed to analyze data. The company has been pushing Amazon Web Services to its limits. “We’re using Amazon more efficiently than the retail arm of Amazon is,” says Adrian Cockcroft, Netflix’s cloud architect. “We’re pretty sure about that.”
Few relationships in the technology industry are as complex as Netflix and Amazon’s. Netflix’s status as Amazon’s biggest customer has earned it favorable pricing and direct lines of communication to Amazon’s top engineers. When Netflix wants a new software feature, Amazon is quick to deliver it, and other customers eventually benefit from that work. “There’s no question in my mind that our platform is stronger from a performance and functional standpoint because of the collaboration we have with Netflix,” says Andy Jassy, who heads Amazon’s cloud business.
Netflix has been forced to build from scratch much of the software it needs to survive. Since it relies on Amazon for data centers, its 700 engineers focus on coming up with tools for, say, automating the ways in which thousands of cloud servers get started and configured. In Silicon Valley, Netflix has become best known for its so-called Simian Army, a facetiously named set of applications that test the resilience of its systems. Chaos Monkey, for instance, simulates small outages by randomly turning services off, while Chaos Kong takes down an entire data center.
Companies such as EBay and Intel have started using these products with their own cloud computing systems, as did the Obama campaign during the last election. Scott VanDenPlas, who managed much of the campaign’s infrastructure, points to a Netflix software tool called Asgard. It’s a system management application that finds groups of servers and outfits them with all the software needed to do a specific job, performing work in seconds that would require a programmer hours or days. “It let us do things quicker and make decisions faster because you know you have this tool in your pocket,” says VanDenPlas. When Hurricane Sandy hit two weeks before the election, VanDenPlas shifted much of the Obama infrastructure from Amazon’s East Coast data center to its West Coast systems. “I don’t think we would have been able to do it without Asgard,” he says. “Our operational efficiency became this enormous strategic advantage.”
Hastings tends to exist in one of two emotional states: relaxed and attentive, or relaxed and dismissive. The things he cares about he’ll discuss animatedly; with everything else, he disengages. Architecture? He claims he never looked at the designs for Netflix headquarters, preferring to just walk in and get to work when someone said the facility was ready. “It was the symbolism of not having me focus on the building,” he says. But if you want to chat about contemporary computer science techniques such as distributed hashtag databases and the merits of key value stores, then, yes, Hastings does have an opinion.
His geeky side became fully apparent in December 2005. He was convinced that the star rating system provided all the information Netflix needed to predict accurately what people want to watch. Others at the company argued that more indicators—whether people started playing something and then stopped, or searched for a particular actor, etc.—were needed as well. Hastings spent two weeks over his Christmas vacation pounding away on an Excel spreadsheet with millions of customer ratings to build an algorithm that could beat the prediction system designed by his engineers.
He failed. Still, the attempt sparked the creation of the Netflix Prize, a $1 million bounty to the person or group that could improve its ratings-based algorithm the most. It was the rare meaningful publicity stunt: The winning team, a collection of independent engineers from around the world, built Netflix a better prediction engine. And a company that was famous for red DVD mailers and outmaneuvering Blockbuster started gaining attention as a place for creativity.
Netflix can now hire just about any engineer it wants. That’s a function of the computer science the company does and its reputation as the highest payer in Silicon Valley. Managers routinely survey salary trends in Silicon Valley and pay their employees 10 percent to 20 percent more than the going rate for a given skill. Fired employees also get ultragenerous severance packages; the idea is to remove guilt as an obstacle to management parting ways with subpar performers.
Netflix also tends to employ older people than its peers. “We hire fully formed adults,” says Cockcroft. “We let them do five years at Google before taking them on.” A walk around the company’s offices bears this out. Engineers work in proper cubicles with high walls that are very out of fashion in the Valley, which has embraced open floor plans. The only real flair: Bathroom doors are decorated to look like entertainment legends (Homer and Marge Simpson, for example), and the meeting rooms are named after TV shows and movies, with famous lines written on the glass walls.
One regular engineering meeting takes place in the Office Space room. (Quotation: “I wouldn’t say I’ve been missing it, Bob.”) To kick things off, everyone talks about what they’ve been watching. One guy says he saw The Hunger Games and started getting recommendations for Bachelorette and The Longshots. “I’m not sure how these have anything to do with Hunger Games,” he says. “They’re all cast as post-apocalyptic movies.”
The lead engineer brings up a Google Doc with meeting notes on a central screen. Netflix has moved all of its e-mail and collaboration tools to the cloud as well, and some of the limitations of that approach become apparent. No one can figure out how to enlarge the document and put it in presentation mode, so the roomful of engineers squints at barely legible text displayed via projector. No matter. The conversation moves on to a new feature built into the company’s Obiwan customer service system. The software interface has been changed to look something like Facebook’s News Feed, where every customer complaint has a trail of “stories” about the issue and what people have done to resolve it. The new system doesn’t seem to be working as well as the old one, so the engineers propose a battery of tests.
Netflix is always testing things. It will select a group of customers, typically by the tens of thousands, and use them as guinea pigs. One group has been given the ability to create avatars for each member of their families, who in turn get individualized recommendations. Others who watch Netflix via the Sony PlayStation have been greeted by a voice that asks what people in the room want to watch.
The most rigorous testing concerns recommendations. Netflix has a vast catalog of movies and shows, but much of its content is old and of limited appeal. To make its service feel valuable, Netflix tries to maximize the likability of the titles that get displayed on someone’s home page. One of Netflix’s mathematicians is known as 10-Foot User Interface Guy because the average person watching the service via TV sits 10 feet away. His job is to arrange the box art of videos in the most appealing way on a big screen. There’s also Two-Foot Guy, who deals with laptops, and 18-Inch Guy for tablets.
The master copies of all the shows and movies available to Netflix take up 3.14 petabytes of storage space. (In comparison, Facebook uses about 1.5 petabytes to store about 10 billion photos.) Hollywood studios used to send individual films and shows to Netflix on a disc or thumb drive; now they use a Netflix system called Backlot to send encrypted files via the Internet. Netflix then compresses the files and creates more than 100 different versions, each tuned for the varying bandwidth, device, and language needs of its customers. (An hour of video for the iPhone would be about 150 megabytes.) This compressed catalog comes to about 2.75 petabytes.
Each night, Netflix performs an analysis to see which shows were the most popular where. From 2 a.m. to 5 a.m. local time, it fills its servers with the appropriate programs. If Battlestar Galactica is popular in Houston on Tuesday, then servers in Texas will be loaded up with more episodes in time for Wednesday night. The most popular videos go on high-speed flash storage drives; everything else gets stored on cheaper, slower hard disks. “We use this predictive model to make sure the content is there before the user asks for it,” says Ken Florance, vice president for content delivery at Netflix.
The biggest bets Netflix is making now are on its original shows. The company won’t disclose how much it paid for two seasons of House of Cards, though the Hollywood blog Deadline.com says it was about $100 million. Rather than make it a weekly show, Netflix released all 13 episodes at once. That meant viewers could watch the whole season in one marathon sitting. It also meant the producers didn’t have to alter the plot to give every episode a cliffhanger ending. “If you give people a more creative format, then they can tell their stories better,” says Ted Sarandos, the company’s Beverly Hills-based content chief. He adds that Netflix’s goal is, in part, to become HBO before HBO can become Netflix. “They do great content that people love. What are the things we do well? It’s the delivery technology, the user interface stuff, the integration into computing devices, and the seamless streaming.”
While it got less hype, Hemlock Grove has attracted a bigger audience than House of Cards, and Arrested Development may well top both. Jenji Kohan, the creator of Showtime’s Weeds, plans to première her latest dark comedy Orange Is the New Black on Netflix this July. Kohan describes Netflix as supremely easy to work with and very hands off. She has yet to even meet Hastings. “I just hear his name, which, by the way, is a great character name,” she says. “It’s a romance novel name.”
Amazon is introducing its own series, too, and Hollywood is abuzz with hope that Netflix and Amazon will spend wild sums of money on even more shows. Sarandos, though, says Netflix is very calculating in its buys. The company puts actors’ names and the show type through its algorithms to determine the likely size of an audience. “I can justify the spend with our data and do so with a far greater degree of confidence than the television networks,” he says. Viewers have given House of Cards a rating of 4.5 stars out of five, which Sarandos says suggests the company is on the right track. “It’s definitely art and science mixing,” he says. Sarandos goes on to say that Netflix will use its data to help pick which actors should be in future shows and who should direct them.
Beau Willimon, the show runner for House of Cards, says the story and actors were all decided before Netflix bought the series. “Every single casting and story choice was made from the creative side,” he says. Even though Netflix has a rich trove of data about the House of Cards audience, Willimon has tried to avoid hearing about any viewership statistics. “That sort of data is a dangerous thing,” he says. “If you put too much thought into that stuff, you run the risk of trying to pander to people.”
Despite its stash of data, Netflix refuses to release any viewership numbers. Unlike the networks, it doesn’t have to prove the size of its audience to advertisers. “The networks have to spend tens of millions of dollars to promote must-see events and create an audience,” says Sarandos. “When we buy ads, it’s just to let people in the industry know that something different is happening.” Kohan says Netflix has gotten under the networks’ skin. “They’ve pissed a lot of people off,” she says. “There is rage because they won’t reveal their data. But I think they’re being brilliant because everyone is talking about them.”
Netflix’s critics argue that the company has sacrificed too much control over its technology in its effort to get ahead of rivals. “The problem with Netflix is that they are inextricably bound into Amazon for all eternity,” says Paul Maritz, a computing infrastructure veteran and CEO of the cloud computing startup Pivotal. “If they want to go somewhere else, it will be hard.” Hastings and his staff characterize such criticism as sour grapes.
Another concern is that the studios will stop licensing content now that Netflix is in the originals business. Hollywood is right to remain wary of letting any single entity get too powerful. As Netflix expands overseas, it intends to strike worldwide licensing deals instead of hammering them out country-by-country. From a studio perspective, that could give Netflix the ability to come up with lucrative terms that no regional competitor could match.
Over the past five years a chart of Netflix’s share price has shifted from a smooth, upward curve to more of a scribble. Hastings seems unperturbed. The experimentation is a goal in itself, whether it’s big things like corporate missions or little things like the personal technology he uses. For one month, he’ll only use products made by Apple, and the next he’s on to phones, tablets, and laptops running Windows. May is Google month, and Hastings has one of the company’s touchscreen Chromebook Pixel laptops. “I just keep it rotating,” he says.
The stock, which has been trading in the low 200s at a price-earnings ratio of 300, is just another side effect of learning, and Hastings doesn’t consider the ride all that rocky. “There was the Blockbuster battle and there were our own mistakes on Qwikster,” he says, ticking off a greatly abbreviated list of his company’s melodramas and near-death experiences. “Other than that, it’s not that volatile.”