It’s been nearly a decade since Dave Chappelle walked away from his Comedy Central show to pursue his true passions of bodybuilding and disappointing crowds. And in that time, the question of “why?” has remained a persistent one, with people often pestering him with it in the middle of his stand-up gigs, and occasionally on television. One of those people was David Letterman, who spent one of last night’s rare Chappelle sightings on the Late Show asking him, again, how he could just walk away like that—a query that, for Chappelle, probably ranks up there with “What did the five fingers say to the face?” on the list of stuff he’s tired of being asked. And yet, this is David Letterman, so Chappelle gamely played along and even opened up, sort of, in talking about how he could refuse so much money to keep doing a show he was already tired of (a decision that Letterman can’t quite comprehend, for obvious reasons).

You can watch the highlights of the interview below, but here are the salient points: Chappelle insists, “Technically, I never quit. I’m seven years late for work.” Ultimately, he sees no difference in his lifestyle between having $50 million and $10 million, unless you count that $40 million. “Of course I would like to have that money,” Chappelle says, adding that he thinks it would make him happier—though he also reminds that money isn’t “the end all, be all,” and not having the show (and that money) has allowed him to buy other things that are also valuable, like his time and freedom. Still, Chappelle also realizes that “there’s not too many people that don’t think I’m crazy,” and yes, he sometimes has his regrets about backing out on what that level of fame and fortune could have brought him—like so many of us, particularly whenever he hears a Jay Z song.