There's a hush when Elvis Costello walks into the basement room of the London hotel where we're waiting to hear Wise Up Ghost, his new collaboration with The Roots. The small talk, gossiping and laughter drop a few notches as Costello, like the teacher of an unruly class, strides in with his hands behind his back. He nods at a few people and heads to the sound desk. When the master is ready, the music starts to play.

As you'd expect from a union of two of pop's most venerable marquee names, Wise Up Ghost is adventurous, ambitious and audacious. Led from the drum kit by Questlove, The Roots twist and turn and swing and sway the music in an abundance of directions.

Costello spikes the moody, sleazy, greasy funk with dazzling wordplay about power, lust, desire, fear and terror, as well as lyrical samples from his back catalogue. Glance to the back of the room during the playback and you can see Costello’s hat nodding along to the music and swivelling around as he makes sure everyone is paying attention.

The meeting of minds that produced Wise Up Ghost came about via Late Night With Jimmy Fallon, the TV show on which The Roots are the house band. "I made three appearances on the show," Costello says. "I also went and sang on one of the songs on the album they've done of Squeeze songs, which hasn't appeared yet. Right at the root of it was that we were getting to know one another and our working methods and the process by which they prepared for work on the TV show.

"When we did the last appearance, when we did the Bruce Springsteen song [Brilliant Disguise], we were on mutual ground. It wasn't my song, it wasn't their song, it was Bruce's song, and we were free to go about it in the way we wanted to. That kind of freedom really set us thinking about if we could do something together.

“The next thing, we were exchanging musical fragments and laying down sketches. It’s like Consequences, that game where you fold over the paper to reveal the word. We didn’t talk about it; we just did it.”

Wise Up Ghost is another collaboration from someone who feels he has been collaborating all his life. "My Aim Is True was a collaboration with Nick Lowe, but because I wasn't known – and because I didn't have any history – at that time, no one commented on that.

"Working with The Attractions over the initial period was a collaboration, with initial discoveries on the first couple of records followed by some tensions which created a work like Blood & Chocolate. Of course, there are more obvious cases, like the work with T-Bone Burnett and working with other musicians for King of America or Spike.

"It's different when you enter a collaboration with another songwriter, be it a star in their own right, like Burt Bacharach – someone I wouldn't even have associated myself with – or a move into areas of music where the performance aspect is different, like The Brodsky Quartet.

"But I've always carried the belief that it's all a form of collaboration from the very start. It's not my fault if I get all the credit. I always try to make sure people know Clive Langer wrote the music for Shipbuilding, for example. I don't say it every night, but it is important to say it, because I'd be up there reciting something if it wasn't for that great tune."

Musical nerds?
Costello and The Roots' paths were probably always destined to cross. Both parties are musical scholars of encyclopedic knowledge, and many have called them musical nerds, although Costello bristles at the description.

"I know Questlove and The Roots are quite self-deprecating about it, but I think 'nerd' is such a reductive word. Did they say Duke Ellington was a music nerd because of the range of music that he played? I don't think you would have said that of him.

“In a world where we have collective mentality and we have a history of every music, from hip hop to rock’n’roll to classical, things have significances that are extramusical in a cultural way, and I think that’s why it’s remarked upon.”

Wise Up Ghost fits neatly into Costello's back pages, because it's another record you never expected him to make. He likes the element of surprise – "I think this sounds like me a lot, in fact" – and he particularly likes the idea of not repeating himself.

"The one record I've steered away from doing, and have no intention of doing self-consciously, is the dreaded back-to-basics one. It's a terrible idea. Go into a garage and go back to real values? What are the real values? We were trying to make the best record we could when we did This Year's Model. If we couldn't play like the London Philharmonic Orchestra, that was because that was as good as we could play. And it was pretty f***ing good."

As always with Costello, Wise Up Ghost is just one plate spinning in the air. Plans are afoot to revisit Painted from Memory, his 1998 album with Bacharach. The American producer Chuck Lorre "has had a 10-year long ambition to turn Painted from Memory into a stage musical. I said to him, 'Really? That's fairly intense, melancholic stuff. What's it going to be like? Is it going to be Virginia Woolf?' Steven Sater, who wrote Spring Awakening, is collaborating," he says, referring to the lyricist who adapted Frank Wedekind's play for the stage, "and there's a very noble director we're hoping to work with."

A book is also in the works. “I started it because my dad got sick, and it came out of conversations I had with him as I watched him disappearing. Some of it felt therapeutic for him to talk about the past, because when people’s memories start to go they can gather their wits about old events. We had some wonderful conversations. We didn’t live together for large portions of my youth, so it was good to talk about a lot of things. There wasn’t any psychological stuff to sort out, just talk.

“I started to write and discovered after he passed away that there were lots more documents about my grandfather, who was also a musician. I thought there was a good story in examining why do we travel like this and our views of music. It’s funny doing the record with Quest and realising that both of our fathers were involved in music. It does give you a slightly different view of things. It takes away some of the mystery, but you can also see how it can be turned into a magical thing in a split second.”

All those plates keep spinning. “It’s difficult when you’re balancing all these things,” Costello says. “You’ve got the shows with the wheel, you’ve got this record, you’ve got Burt calling at night, looking for lyrics for the musical, and I think The Imposters are playing the best shows of their career.”

The work ethic that drives him, he says, comes back to family. "My mother's father was a solid Protestant working-class man from Liverpool. If you looked at 'Protestant work ethic' in the dictionary, Jim Ablett's picture is there. So I've a little bit of that in there. It's in the blood. I can't do anything about it."

Wise Up Ghost is on Blue Note