Behind a small cafe in San Francisco’s Potrero Hill neighborhood stands an unmarked warehouse where the future of human-machine interaction is taking shape. Inside this sprawling maze of soundstages, machine shops, and computer labs, artists collaborate with engineers, cinematographers brainstorm with coders, and everyone has a collegial relationship with the small army of industrial robots stationed here. This is Bot & Dolly, a boutique design studio that specializes in combining massive mechanical arms with custom software for movies, architecture, digital fabrication, and entertainment installations. “We’re a culture of makers, of creators with open minds,” says Tobias Kinnebrew, Bot & Dolly’s director for product strategy. “We work on things that don’t seem possible and try to make them possible.”
One of Bot & Dolly’s first clients, Google (GOOG), bought into that vision quite literally. In 2012 it commissioned Bot & Dolly to create an attention-grabbing experience to promote its Nexus Q media-streaming device at the Google I/O conference. Bot & Dolly built an 8-foot-across, 300-pound Nexus Q mounted on a robot arm that passersby controlled via several Nexus gadgets working in tandem. The client was apparently quite pleased. Late last year, Google acquired Bot & Dolly for an undisclosed sum. A spokeswoman for the search giant declined to comment on the deal.
Bot & Dolly was started four years ago by Jeff Linnell and Randy Stowell, as a side project at their video production company, Autofuss. (The cafe at the front of their building, called Front, is the pair’s latest joint endeavor.) Still operating independently with around 20 full-timers, Bot & Dolly is best known for bringing weightlessness to the big screen for last year’s Oscar-winning film Gravity.
To do what director Alfonso Cuarón envisioned—but said “could not be done”—Bot & Dolly flipped a typical Hollywood approach on its head. The effect of weightlessness is usually achieved by filming actors flying around a soundstage on wires and adding computer graphics later. In Gravity’s case, Cuarón and visual effects company Framestore animated 3D digital storyboards of the entire film, a technique called previsualization, before live-action shooting with the actors even began. As a result, the live action had to perfectly match the preexisting digital choreography. After a few tests in San Francisco, Bot & Dolly loaded its 3,000-pound robots onto a cargo plane and headed for Cuarón’s set in London. Once there, the machines became part of an elaborate dance in the soundstage. A robot-mounted camera moved around Sandra Bullock in a way that matched the rough computer-generated sequences of a spacecraft interior or of a spacesuit hurtling toward earth. The focus was solely on the actors’ faces, though, as the rest of the film was created entirely inside a computer. Once the live action shots were in the can, that footage was digitally combined with the computer graphics. “Rather than moving an actor through a world, our robots moved the cameras, set pieces, and lights around the actor,” Kinnebrew says. According to Bot & Dolly Senior Producer Bill Galusha, the process was partly inspired by the classic Fred Astaire scene in Royal Wedding where he appears to be dancing on the walls and ceiling.
Just about every one of Bot & Dolly’s creative projects—whether private client work or an internal “exploration,” as Kinnebrew describes its projects of passion and public relations—involves a flow between the digital and the tangible, and back. “We capture something in the real world, consume it into a digital environment, and push it back out in the physical world,” he says. “It’s an iterative cycle.”
“We work on things that don’t seem possible and try to make them possible”
Consider Bot & Dolly’s Box, a combination art film and technical demonstration that at last count had almost 3 million views on YouTube. It’s a surreal and astonishing clip in which a human actor explores a computer-generated geometric universe. And yet it was made in live action, without green screens or imagery added in postproduction. High-definition projectors beamed angular computer graphics onto two large screens mounted on a pair of robotic arms. The shape-shifting 3D graphics give the illusion that the screens are portals into a psychedelic digital realm. Box is an example of projection mapping, where the virtual and the real are blended in form and motion with the precision of a prima ballerina.
Yes, timing was everything in Box, as it was on the Gravity set when matching CG sequences with live-action cinematography. But Kinnebrew says precision is only one of many characteristics that matter in the creative application of robotics. To illustrate, he describes another effort in which his colleagues are attempting to teach a robot to play cello. In this case, he says, there are nuances in the natural fluidity of a human cellist’s movements that make it highly difficult to translate into the language of robots.
How, then, can a smart and talented human teach these subtleties to a dumb-but-attentive machine? Basically, by telling the robot in your own native tongue to do what you do. “If you’re a basketball player, and you speak basketball, you need to speak to a robot in basketball,” says Kinnebrew, who previously spent eight years as a creative director for Microsoft. “If you’re a chef, and you speak cooking, you need to speak to a robot in cooking, and we need to build those kinds of tools.”
Bot & Dolly is designing software to help creative people from a broad spectrum of disciplines program robots in intuitive ways. For example, a cinematographer with a specific handheld camera shot in mind can just grab a robot-mounted camera and make the movement. Afterwards, the camera operator pushes a button and the robot plays back that physical movement exactly, every time. The result is indistinguishable from a real handheld shot with the benefit that the recorded motion can also be tweaked ad infinitum. Make a mistake? Hit undo. It’s like “command Z” for the real world.
Bot & Dolly’s latest internal project, called Stack, is a tool for parametric fabrication, an algorithmic approach to design in which architects use software to generate structures that look like they belong in a modern art museum. The designer feeds a series of values into the software, and Stack employs Bot & Dolly’s robots to build the objects. Inside a back room at Bot & Dolly, a large robot is diligently stacking and nailing wooden sticks into a futuristic curved tower nearly reaching the ceiling.
Is this how buildings will be made in our roboticized future? Kinnebrew isn’t willing to make that leap. Right now, he says, Bot & Dolly is delighted just to be developing another set of creative tools and learning to wield them. Only when we’re comfortable around robots “can we let the dots connect” between people and the machines, he says. Then the imagination will take over, applications will reveal themselves, and the real magic will emerge. “Computers have become a deeply valuable tool everywhere the creative process takes place in our world,” he says. “I think we’ll see the same thing happen with this technology as well.”