Last month, as I was being served at the counter of north London’s fashionable Rhythm Records, the assistant suddenly clasped his stomach and rushed out into Camden High Street. “It’s all right,” said a second assistant, who took over, “he had a heavy night. It isn’t a comment on your taste.” I had just bought a copy of Tennessee Moon, the new album by Neil Diamond.

I grew up in a house full of Neil Diamond music. My mum fell under his spell in 1973 after hearing the live double album Hot August Night while under local anaesthetic at the dentist. Subsequently, I realise I’ve learnt all the words to his first 10 albums without even trying. Talking to him is a bit like meeting Father Christmas.

For the uninitiated: Diamond is 55; he grew up in New York, got his first guitar at 16 and dropped out of a fencing scholarship at New York University to be a $50-a-week publishing-company songwriter. His hair grows in an arc around his head now, like a fur halo. He wears blue sequined jackets. He has sold more than 110m records. If he’d died, unknown, in 1972, he’d arguably be one of those lost 1960s figures that young hipsters lust to rediscover, a la Tim Hardin or Fred Neil. Instead, Diamond grew up to fill enormous stadiums, duet with Barbra Streisand and pursue some ill-advised pop-disco avenues. Still, that’s showbusiness.

For the teenage Diamond, seeing the leftish folk singer Pete Seeger was something of a pivotal moment. “He played a bunch of songs, including songs he’d written,” remembers Diamond. “This was 1956. There were maybe 30 or 40 people there. It was the first time I’d sat down and realised songs could be written by ordinary people.” Ten years later, Diamond was a songwriter at the Brill Building publishing house, alongside Gerry Goffin and Carole “Will You Still Love Me Tomorrow?” King, and the Monkees made his I’m a Believer famous. With hindsight, the Brill Building can be seen to have shaped American popular music, but for the young Diamond, it certainly didn’t feel like it was historic: “I was just trying to find a foothold, as a songwriter or a producer, I wasn’t too sure.” Soon Diamond became intent on carving out a solo career, a surprising move then for a paid hack. “It was unusual to put out a record yourself, a song that you’d written, but Bob Dylan proved it needn’t be such a novelty. And then the Beatles proved it could be commercial.”

For three decades since, Diamond’s career has been nothing if not commercial. He’s had mega- hits with Sweet Caroline (1969), Cracklin’ Rosie (1970) and Song Sung Blue (1972); his Greatest Hits, released in 1992, went platinum. He has just embarked on a 15-date British tour and is now filed in the same middle-of-the-road, stadium-fodder league as Michael Bolton.

But in the 1970s, Diamond was held in enough critical esteem to warrant a collaboration with producer Robbie Robertson, of Bob Dylan acolytes the Band, on his concept album Beautiful Noise. He played alongside such respected members of the rock hall of fame as Van Morrison and Neil Young in Scorsese’s film of the Band’s final gig, The Last Waltz, yet both these artists have an acclaim that Diamond lacks. But he doesn’t regret having chosen a different path. “I think we all chose the same thing: to make music. What pigeonhole you get put into comes later.”

Ironically, while the 1960s survivors all still wear rebel chic as comfortably as a suede-tasselled jacket, it’s the spangle-shirted Diamond who is least likely to be invited to the Rolling Stone magazine Christmas party. He’s the real outsider. “That’s true,” he agrees, “but don’t tell anyone, or I won’t get let back in.” That’s the essence of Neil Diamond, caught somewhere between earnest songwriter and snake-hipped Vegas cabaret star.

The signs are all there, as early as his first solo single, the brilliant Solitary Man from 1966. An earnest, brooding, writerly verse, knocked out over a minimal acoustic guitar, soon gives way to an upbeat, horn-augmented sing-along chorus that fades from a jaunty major key back into a menacing minor one, before returning to the verse the schizophrenic nature of Diamond in a nutshell. For every beautiful Girl, You’ll Be a Woman Soon, there is some awful 1980s pop effort like Headed for the Future, a song so gauche that Cliff Richard himself would be proud to sing it. But Diamond knows this. In 1984, he said he often wondered if he were Sammy Davis or Bob Dylan. “But now I am me,” he concludes, reassuringly. “All that self-evaluation and trying to be another artist is over. I’m not trying to be somebody else.”

The sleeve notes to Hot August Night describe, bizarrely, how Diamond’s audience “falls like plums at his feet” (sic), and predictably, most journalistic criticism of Diamond attacks his plums, rather than his music. The block vote of an army of middle-aged, middle-class housewives is apparently beneath consideration. But every ticket for Diamond bought out of a tight housekeeping budget is at least one less for an Andrew Lloyd Webber musical. That said, the audience we see in Diamond’s 1988 video release, Greatest Hits Live, has a bri-nylon ultra-normality so pronounced it makes them almost freakish, as if David Lynch had hand-picked every one. But Diamond won’t have any of it: “You have two choices. Leave it to the critics, or please yourself and the audience. My audience has a strong sense of loyalty that has got to be repaid. I try to put on the best show I can. I hold back nothing.”

Being a popular mainstream entertainer is a critically unrewarding job, but someone’s got to do it and Diamond at least does it with integrity. And with the release of his new album, the country-styled Tennessee Moon, it looks like he might at last be heading back on line. The only country songs I ever remember Diamond doing were the pastiches of the early 1970s, such as Soggy Pretzels and You’re so Sweet, Horseflies Keep Hanging Round Your Face, but he maintains that in Tennessee Moon he just made the record he wanted and that he wasn’t aware that country and western was enjoying a renaissance. Even if that’s true, nobody who has sold more than 110m records is allowed to make these kinds of decisions blind. There are men with ponytails who are paid handsomely to stop them.

For Diamond, Tennessee Moon, recorded in Nashville, represents something of a wish fulfilment. On the closing song, Blue Highway, he plays with his childhood hero, the 1950s country star Chet Atkins. How? “The way you get Chet Atkins to play on your record is to invite him over to watch the Super Bowl and tell him that you have to have him play, or your life just won’t be the same.”

I still think Diamond has a better album in him than Tennessee Moon. The title track itself keeps threatening to cut loose and shift up a gear, but never makes it, and the whole album has a kind of safe, homogenised sound that still doesn’t do Diamond justice. But at worst, Tennessee Moon shows that a producer with any kind of vision could easily sculpt a new, improved, unplugged Neil Diamond who could placate old fans, as well as win him new ones from the MTV generation. Whether that’s something Diamond himself would want is another matter entirely.

Earlier this year, Diamond told Q magazine: “The only time I’ll be happy is when I’m dead.” Asked to qualify this morbid statement, he explained: “I’m on a particular road and I don’t know why I’m on it, or why I’m so determined, but I am on it. It’s not my job to be happy, but to fulfil whatever potential I have. My goal is not happiness, but to be productive and to create beautiful noises.”

Back in the 1950s, a young Diamond helped out in his family’s lingerie shop. The radio was always on. “I was quite a shy kid, but when you have to take care of people in the shop it forces you out of yourself. Nothing’s changed. I still just try to keep the customer satisfied.”