Photographer: Jesse Dittmar for The Washington Post via Getty Images

And the New Yorker Cartoon Contest Winner Is … a Computer

The magazine’s cartoon editor collaborated with Microsoft researchers on an artificial intelligence project that aims to teach machines what’s funny

Since 2005, the back page of the New Yorker has usually featured a wordless, black-and-white cartoon, and the funniest reader-submitted caption gets published in a following issue. The magazine’s caption contest has become a fan favorite over the last decade, and the cartoon department receives some 5,000 entries each week. This has become an overwhelming number of jokes to sift through—particularly for Bob Mankoff’s assistant. The 71-year-old cartoon editor for the New Yorker says the average tenure of his assistants is barely a couple of years because he keeps burning them out. “The process of looking at 5,000 caption entries a week usually destroys their mind in about two years, and then I get a new one,” Mankoff says. “It's a little bit daunting. It's like going snow blind; you go humor blind.”

Soon, Mankoff’s assistants could get relief in the form of an assistant of their own: an artificial intelligence system with a sense of humor. Mankoff collaborated with researchers at Microsoft on an artificial intelligence project that aims to teach a computer what’s funny. They’re feeding an archive of New Yorker cartoons and caption-contest entries into AI software to give machines some understanding of humor (the New Yorker’s brand of humor, at least). A Microsoft researcher plans to present the findings onstage on Aug. 13 at the KDD data conference in Sydney.

The idea for the project arose at a different convention about a year ago. Dafna Shahaf, a researcher at Microsoft, attended a speech by Mankoff about the cartoon archive, and she left feeling excited. Shahaf wondered whether she could teach a computer to accurately assess how funny a caption might prove to be—and in the process, crack one of the most difficult challenges in machine learning. Sarcasm, wordplay, and other tools of humor have perplexed AI systems for decades. At Microsoft, teaching machines and software to get the joke is important for things like the Skype Translator, which is designed to let users speak to each other in different languages and hear translations on the fly.

For the study, Shahaf fed cartoons and captions from the New Yorker’s database into the system and trained it to find the funniest choices among captions that make similar jokes. She relied partly on crowdsourced input from contract workers, using’s Mechanical Turk. Then she moved to the harder task of ranking jokes. Because typical computer vision software is designed for photos, not drawings, the researchers had to manually describe what was pictured in each cartoon. They organized this into two categories: the context and the anomalies.

New Yorker AI cartoon

A sample New Yorker cartoon, with the associated tags, that was fed into Microsoft's artificial intelligence software.

Source: Microsoft via Bloomberg

While Mankoff says a computer will probably never be able to beat his writers at being funny, Microsoft has successfully devised an AI system that could make cartoonists’ jobs easier. Mankoff is impressed by the software’s ability to winnow out lame submissions and narrow the list to the funnier ones. For example, when evaluating a cartoon depicting a car salesman hawking a hybrid animal-vehicle with hairy legs instead of wheels, the AI picked, “Just listen to that baby purr” as one of the top captions; “It runs on a 100 percent fuel efficient Paleo diet” was rated one of the worst. The winning caption in the New Yorker was, “Relax! It just smells the other car on you.”

The machine and the New Yorker editors don’t always align on shortlists. On average, though, all of the editors’ favorites appeared in the AI’s top 55.8 percent of choices, according to the study. That means the New Yorker could use the system to eliminate at least 2,200 submissions a week without missing the gems. “On average, we saved about 50 percent of his workload,” says Shahaf. It could also save Mankoff the time it takes to hire new assistants. “I do think the future is human-machine companionship,” Mankoff says. “Computers can be a great aid.”

New Yorker AI cartoon

New Yorker cartoon with the captions short list.

Source: Microsoft via Bloomberg

That’s an unexpected admission coming from Mankoff, who has frequently mined the concept of humans submitting to robot overlords in his cartoons. (In 2013, Mankoff published an article titled “How to Deal with the Coming Robot Apocalypse.”) He professes to be optimistic about AI’s ability to aid humorists, although he says Microsoft’s system needs to improve its accuracy somewhat before he’d employ it. Mankoff doubts whether machines can ever completely replicate his sense of humor. Riffing on Hamlet (“There are more things in heaven and earth, Horatio, than are dreamt of in your philosophy”), Mankoff opines, “There are more things in humor and human beings than are dreamt of in even Microsoft’s algorithms.”

That hasn’t lessened Microsoft’s ambitions. The researchers say they hope to one day train computers to come up their own jokes based on situations, which would make digital assistants such as Cortana and Siri more pleasant to interact with. Eric Horvitz, the managing director of Microsoft’s research group and a co-author on the study, jokes that your computer might try to lighten the mood when you accidentally delete an important file with a carefully timed “oy vey.” More broadly, understanding what we find funny and how we come up with jokes is important to the study of how the brain works, which itself is crucial to artificial intelligence research. A side benefit: With all the consternation over the evils of AI, Horvitz says it’s nice to think of these systems doing something more pleasant than destroying the universe.

Or stealing someone’s job. But that, too, is unlikely to happen when computer analysis remains split on even the basic principles of humor. While Microsoft found that positive messages play better in captions, a second study of the same database of New Yorker cartoons by a University of Michigan researcher came to the opposite conclusion. Mankoff, who authored one of the New Yorker’s most famous cartoons about a brutally direct businessman trying to avoid scheduling a meeting, doesn’t foresee a computer replacing him soon. “How about never,” Mankoff says. “Is never good for you?”