How did you decide that now’s the time to leave?
I had been in a very intense work situation for a very long time. And we have these natural junctures. Every two years is sort of when people decide to rebel. I’d thought that after the midterms in 2014 would probably be my last chance to go. Right after the midterms, I thought we were in a really bad place. I felt I couldn’t leave. I didn’t want to leave at one of the low points of the presidency. Then we had these really great four, five months that helped the president regain his political standing, grab hold of the narrative. And I thought if I was going to leave, this was the time. But it was a really hard thing to do.
What’s your assessment of Hillary Clinton’s e-mail scandal?
I think this is what David Axelrod referred to as a pimple on the ass of progress. Eighteen months from now or 20 months from now, when people go to the polls to vote in the presidential election, I can’t imagine anyone’s going to change their vote because of this. This is one of those things that’s happening because there’s a vacuum in our political discussion right now. Once there’s an up-and-running campaign, this will be a faint memory.
And give me your take on the House and the Senate and Iran and Netanyahu.
We’re actually at a very dangerous point in our polarization of foreign policy in this country. The speaker inviting the prime minister to try to politicize and make the U.S.-Israeli relationship partisan? That’s very dangerous. Then you have this news that 40-some Republican senators sent a letter to Iran to work with the hard-line elements. They’re trying everything to nullify the president’s win in 2012. He’s in charge of our foreign policy. Congress has a consultative role. And they’ll have a chance to vote, if they choose to bring a bill up to try to deal with this. But to actually write a letter to Iran to try to scuttle a U.S. foreign policy initiative … this is a very dangerous thing to do.
What was the best moment for you at President Obama’s side?
The best moment was probably the night we passed health care. To me, that was what validated that this was going to be a different sort of presidency. What really drove me in the  campaign was this idea that the president would do two things: He was going to take on fights other people have shied away from, and he would succeed where others had failed. I questioned that, when it looked like health care might fail. And that we were able to do it, well, there was just elation in the room. It’s one of the few times you can make a massive difference in millions of people’s lives.
Obama was in Selma last weekend. Does race have anything to do with the opposition he’s faced?
I’m incredibly hesitant to ascribe racist motivations to anyone. But there’s no question there are people who oppose the president because he’s African American and probably also people who support him because he is. I don’t think racism drives the overarching opposition to the president. I think it’s political, cultural—it’s policy.
Now that the president doesn’t face another election, does he feel more free to take action?
It was very important to him that we keep the Senate in 2014. That mattered to him, but it also held him back. It made it harder for him to go out and make his case to the country. With that behind him, he has more space. I also think he feels the ticking of the clock. He says to us all the time, “You will never have a better opportunity to do more good for more people than we have right here.” It’s like every single day matters.
What’s the most emotional you’ve ever seen Obama?
The most emotional, in terms of deeply affected, would be after Newtown. And I think the most angry I’ve seen him was after the Senate failed to pass background checks in 2013. There was probably no issue that better exemplified the dysfunctionality of Congress than an issue like background checks, with 90 percent support, being defeated only a few months after a tragedy like Newtown.
Watch Charlie Rose on Bloomberg TV weeknights at 7 p.m. and 10 p.m. ET.