“I’m not antisocial but I draw the line with pop groups,” says Robert Lloyd of the Nightingales. “People in bands aren’t generally my kind of people.” Pre-lockdown, Lloyd could instead be found at the pub or horse-racing track, but, despite his aversion to the fickle world of music, he’s been a fringe figure in it for more than 40 years. The band he fronts, the Nightingales, are a perpetual outsider outfit, deeply loved and revered by a cult following, but an unknown entity to most. Lloyd is also the subject of a new documentary, King Rocker, made by comedian Stewart Lee and director Michael Cumming.

Lloyd began his musical life in the Prefects, a Birmingham punk band that comedian Frank Skinner briefly fronted before Lloyd took his place. Skinner auditioned outside a pub by singing the Ramones’ Blitzkrieg Bop but Lloyd had done one better and actually been to see and meet the band. Plus, Skinner’s hair was apparently too long.

The band’s first gig at a party ended drunkenly with trashed equipment and a police raid. Their second gig was met with flying pint glasses after playing a track called Birmingham Is a Shithole. “When we started it was just a Sex Pistols rip-off thing,” Lloyd says. “I was a teenage pleb and didn’t have much in the way of original thought.”

Nevertheless, John Peel loved them and became an early supporter. Their 1979 track Going Through the Motions remains a truly brilliant single of the era. The dense, moody droning groove is the antithesis of rapid-fire punk and agitated post-punk discordance – the tone sitting more alongside fellow cult pioneers This Heat.

The song was a choleric response to punk, which Lloyd had tired of. “We were supporting Sham 69 and they had a bit of a skinhead following so we started opening our set with Going Through the Motions, this five-minute dirge,” he says. “It was to purposefully piss them off and see how much we could antagonise them. It worked a treat.”

Lloyd’s frustration set in a year earlier playing on the Clash’s White Riot tour, where disappointment and animosity quickly flourished. “It was the same old fucking cabbage,” recalls Lloyd. “Everyone wanting to be pop stars. In my stupid teenage mind I thought it was more revolutionary. I’m glad I got disillusioned so quickly so I didn’t waste my time thinking I was in a movement that didn’t exist.”

Lloyd’s band and the Clash came to blows. “Their roadies attacked a member of our band and put him in hospital and that was the icing on the cake,” he says. The Clash’s manager, Bernie Rhodes, exclaimed: “I’m a patron of the arts and you’re just a bunch of amateur wankers.” Years later, the Prefects released a compilation entitled Amateur Wankers.

The Nightingales were born from the Prefects, who broke up in 1979. For Lee and Cumming, the band not only represent a shared love, but make up their DNA. “We’re both made out of bits of the culture they came from,” Lee says. “We listened to John Peel’s shows and the comedian that made me want to do comedy was Ted Chippington, who was the Nightingales’ sort of in-house comedian. Going through Rob’s lyrics, and that wordplay of those regional post-punk bands, was one of the things that got me reading. This film is made by people that are partly as they are because of them.”

The Nightingales don’t sit comfortably within genre, often labelled post-punk or as some kind of Beefheartian avant-pop-rock, but no label really sticks. Nor do comparisons to the Fall that follow them around. “I never understood that comparison,” Lloyd says. “Only inasmuch as we’re both bands that you can’t compare to other bands – and us both being a couple of curmudgeonly old cunts who don’t know when to give up.”

Lloyd’s self-deprecating and vituperative humour is a constant, as is his hearty laugh and glugs of beer as we chat over Zoom. This humour, and relaxed conversational approach, is at the heart of King Rocker, which Lee and Cumming describe as an anti-rockumentary.

“I watched loads of rock documentaries with my son and asked him, ‘how would you describe these?’” Lee says. “He said: ‘Loads of old men remembering things, and then there’s black and white.’ The good thing about this is there weren’t that many people that remembered it and there’s hardly any footage, so it had to be something else.”

Chats in curry houses, a prolonged King Kong metaphor, and hours in the pub make up the film, which is as likable, funny and singular as Lloyd himself. Typical talking-head segments only pop up to question Lloyd’s occasionally unverifiable stories. “I like the idea of making a documentary in which quite a lot of people in it have never even heard of Rob,” says Cumming.

Lloyd has had an interesting career trajectory. The Nightingales’ 1982 debut Pigs on Purpose is a unique concoction of irregular time signatures, Lloyd’s distinct lyricism, taut post-punk and an undercurrent of wonky melody. By their third album, 1986’s In the Good Old Country Way, they integrated elements of blues, country and rockabilly to position themselves as a kind of British version of the cowpunk outfit the Gun Club, albeit with Lloyd’s delivery more refined and croon-like.

After years of making boundary-pushing music in the underground, Lloyd’s next step was unexpected. The Nightingales split in 1986 and he soon landed a solo deal with Virgin, moving to London to make a go of it. “In those properly produced pop videos he looks like the Jarvis Cocker that never happened,” says Lee. “I sort of lost the plot,” Lloyd says, looking back. “Although I didn’t think I had at the time. With pop videos, whether it’s Björk or Jason Donovan, it makes no difference: it’s still three minutes of some cunt showing off.”

The album Me and My Mouth didn’t do as well as hoped, and his second album never materialised beyond demos. Lloyd was living above a Chinese takeaway and went through a unique series of jobs. He wrote about food for GQ, getting Nigel Slater booted from his regular column in the process; co-wrote a TV script with music writer Steven Wells that imploded before production; then ended up a postman.

“I jacked in music altogether for a few years,” he says. “I got into a bit of a rut. I was fannying around having a good time. I thought, ‘I remember when you used to do stuff.’ That was what kicked me into gear and I moved out of London. I could see myself staying there and just boozing and doing coke and getting lazy – well, I was lazy.”

Lloyd returned to Birmingham and a new chapter emerged for the band with his renewed purpose and creative zeal. “It’s not like what I’m doing is something the world needs,” he says. “It’s not essential. But for my own personal wellbeing I needed to do it. It wasn’t sufficient for my mind to be just having a good time and not trying to do stuff.”

The Nightingales reformed and the opening track to their return 2006 album Out of True is the swaggering Born Again in Birmingham, which captures this transitional period and rebirth in Lloyd’s life over infectiously spiralling guitars, a flurry of drum work, and a band that truly do sound alive and vital again.

However, the band remained simmering under the surface, with five albums coming out on five different labels. Lloyd, exasperated, named their 2014 album For Fuck’s Sake. “Every record deal we got dropped after one album,” Lloyd says. “Which is why we eventually said, ‘Oh, for fuck’s sake.’”

They self-released the album to some acclaim and, with the steadiest lineup the band has ever had – Fliss Kitson, James Smith and Andreas Schmid – a period of consistency and potency followed over another four albums. “The band gets better and better,” Lloyd says. “The last couple of records are probably my favourites.”

They are now settled on the label Tiny Global, although it remains a committed passion project. “I’ve never made any money from a record, ever,” Lloyd says. “Frank Skinner says in the film: all successful artists want to be cult figures and all the cult figures want to be successful.”

So despite being an uncompromising and inimitable counter-culture figure for the last 40 years – a proud amateur wanker – does Lloyd deep down harbour aspirations to be genuinely popular?

“I don’t know this group at all, I’ve never heard a single song,” begins his response. “But I’d like to be like Grandaddy. If you want to tour Australia someone will book it, if you want to do an album someone will pay for it, if you play in Wolverhampton a thousand people will turn up but if you go in the local Wetherspoons then nobody will know who the fuck you are.” He breaks off into that laugh once more. “I couldn’t deal with fame and people knowing you down the chip shop. I mean, when was the last time Bono went to the chippy?”

• King Rocker airs on Sky Arts on 6 February.