Apple Inc.’s release of a software update that cut off people’s ability to make calls from their iPhones is linked to another snafu that’s still fresh in people’s minds: the 2012 introduction of a new maps program.
The similarities don’t end with the apologies Apple offered to disgruntled customers. The same person was in charge of catching problems before both products were released. Josh Williams, the director overseeing quality assurance for Apple’s iOS mobile-software group, was also in charge of quality control for maps, people familiar with Apple’s management said.
Williams was removed from the maps team after the software gave unreliable directions and mislabeled landmarks, though he remained in charge of iOS testing, said one person, who asked not to be identified since the information isn’t public.
Scores of customers took to social media to complain about losing the ability to make phone calls after installing the iOS 8.0.1 update, which Apple said it pulled back after more than an hour. The software glitches have undermined Apple’s mantra that its products “just work” and, at least temporarily, marred what Chief Executive Officer Tim Cook called the “best launch ever” for Apple’s newest iPhone models released last week.
Apple apologized again yesterday and issued another update, iOS 8.0.2, saying that less than 40,000 iPhone 6 and 6 Plus devices were affected by the previous version with the problem that caused people to lose cellular reception.
Trudy Muller, a spokeswoman for Cupertino, California-based Apple, declined to comment on the company’s quality assurance program. Williams didn’t respond to an e-mailed request for comment.
“Normally, things like this -- software updates -- are a strength for Apple and something that is easier for them than for almost any other device or operating system maker, because Apple has full control of the hardware and the software,” said Frank Gillett, an analyst at Forrester Research Inc.
Williams leads Apple’s choreographed process for catching bugs before they reach the public. He’s worked at Apple since 2000 and graduated from San Jose State University, according to his LinkedIn profile. An Eagle Scout known within the company for his tattoos and guitar playing, he’s been working on quality control for iPhone software since early iterations of the product.
Williams has a team of more than 100 people around the world who are responsible for putting new software through its paces and uncovering glitches that may eventually impact customers, according to one person. Apple relies more on people finding bugs than using automation-testing technology, according to former employees.
At a company of Apple’s size, no single person is solely responsible for large software problems, and Williams works with others tasked with catching issues before software is released. And while the public quickly knows about mistakes made by Williams’ team, it doesn’t see all the flaws his team catches. Before an early iOS release several years ago, Apple caught a mistake that would have made iPhones running e-mail accounts from Microsoft Corp.’s Exchange potentially unusable, according to an employee involved.
Apple has a committee to determine what bugs need to be fixed called the Bug Review Board, known internally as BRB. The panel is overseen by Kim Vorrath, a vice president in charge of product management for iOS and Mac software. The executive who oversees the entire software organization is Craig Federighi, Apple’s senior vice president for software engineering.
At the review board meetings, Vorrath, Williams and others from Apple’s software engineering group go through defects to determine what should be fixed immediately and what can be addressed later, after the software is sent to customers, according to engineers involved. The board designates rankings to different bugs, with “P1” being the highest, and meaning the company will stop production of a device if a glitch is discovered, one former manager said.
The meetings can become contentious because engineers will often argue for more time to fix a problem, while product managers will push to move forward to meet approaching deadlines, two people said.
For lower-priority fixes, known as “P2” or “P3” bugs, Apple usually starts work on a software update even before an initial version is released to the public, according to former employees who have been involved. That’s because Apple needs to complete a final version several weeks before the official release so that it can be installed on iPhones coming off the assembly line, two people said.
Another challenge is that the engineers who test the newest software versions often don’t get their hands on the latest iPhones until the same time that they arrive with customers, resulting in updates that may not get tested as much on the latest handsets. Cook has clamped down on the use of unreleased iPhones and only senior managers are allowed access to the products without special permission, two people said.
Internal turf battles also can impact quality testing, according to a former senior manager. Teams responsible for testing cellular and Wi-Fi connectivity will sometimes sign off on a product release, then Williams’ team will discover later that it’s not compatible with another feature, the person said.
The speed of issuing the 8.0.1 update was unusual, just days after the introduction of iOS 8, the latest major upgrade to the software that runs iPhones, iPads and iPod Touch devices.
In addition to testing iPhones manually, Apple’s quality control team also uses automated tools to assist, according to former employees. IPhones are placed on racks in a lab running automatic tests, one person involved said. Apple also relies heavily on outside developers to test early versions of the software to report glitches that can be fixed, the people said.
The 8.0.1 update was aimed at fixing issues from the initial release put out last week, and added Apple’s health and fitness-tracking application HealthKit. Instead, the update immediately disabled some people’s access to their mobile network so they couldn’t make phone calls.
With problems related to maps, Williams wasn’t the only person who worked on them. Richard Williamson, the vice president in charge of the product, was fired after the trouble-filled release. Scott Forstall, Apple’s senior vice president in charge of all mobile software, was also dismissed, partly because of the problems with the maps service.