Paul Shaffer enjoyed a long career as David Letterman’s bandleader and sidekick, however he initially turned down the job.

“I got a call right as my five years on SNL were ending and everybody was leaving,” Shaffer recalled during an appearance on the Kenny Aronoff Sessions podcast. The keyboardist had been part of Saturday Night Live’s band and occasionally showcased his comedy chops in sketches. “I left with the original [SNL] casting. And I got a call at that time. David Letterman is having a morning show live in the morning, an hour live in the morning.”

Letterman was a rising young comedian at the time and NBC was giving him his first daily talk show. While the opportunity intrigued Shaffer, he had no interest in waking up early.

“Letterman still to this day, says, ‘You didn't do the morning show, couldn't get up that early,’” the bandleader quipped.

READ MORE: 15 Guests Who Were Banned From 'Saturday Night Live'

Ultimately, Letterman’s run on daytime television was short-lived. NBC decided he was better suited for late night an offered him a time slot after Johnny Carson’s Tonight Show.

“They called me again,” Shaffer noted. "’Well, now he's going to do a show after the Tonight Show, start at 12:30 to 1:30. What do you say?’ And I said, ‘Well, now that's more like it. Even later than Johnny Carson, now that's my kind of thing.’"

Letterman's Humor Quickly Won Over Shaffer

Before officially taking the bandleader position, Shaffer had to meet Letterman. The musician knew in their initial conversation that the chemistry was right.

“I went in for a meeting with Dave and we kind of hit it off,” he recalled. “He says he saw me on SNL, mentioned the things, the sketches I used to do with Bill Murray when he was the lounge singer. Dave mentioned that and stuff."

READ MORE: Rock's 60 Biggest 'Saturday Night Live' Moments

During their conversation, Shaffer said he wanted to have a band reminiscent of those in "the topless bars of Canada."

"Well, I do base it on the organ and do instrumental versions of R&B things," Shaffer explained to Letterman. "Well, I've always thought of myself as the Wayne Cochran of comedy," the host wryly responded.

"What a reference," Shaffer thought to himself. "A little known regional, white James Brown from Miami.’”

Given a glimpse at Letterman's esoteric humor, Shaffer was all in.

“I said, ‘I got to work for this guy.’ And I ended up working for him."

David Letterman's Most Memorable Late-Night Moments

A roundup of David Letterman's most memorable late-night moments.