James Palumbo: You don't like him? Do you think he cares?

An elevated glass office at the Ministry of Sound headquarters in south London provides James Palumbo with the perfect vantage point from which to keep an eye on his staff below. There is nothing unique about a boss who likes to know what is going on while maintaining his distance, but few make it quite as obvious as Palumbo, the 38-year-old music-industry mogul whose rise to wealth and influence is both one of the most improbable and most resonant of our times.

"There was something Big Brother-ish about it," says a former Ministry employee. "The only real contact he had with us was when he gave a speech at the Christmas party. And even then some of the things he'd say didn't square with what was going on at ground level."

In 10 years, Palumbo has shown many talents in turning the Ministry from a disused warehouse in Elephant & Castle into a multi-million-pound entertainment and media corporation, but the gift of social intercourse does not seem to be one of them. So when he said last week that "success is about people", Ministry employees could be forgiven for allowing themselves a wry smile.

Wary of the press to the point of reclusiveness, Palumbo was prompted into public utterance by the purchase of a 20 per cent stake in the Ministry by the venture capital group 3i. Its investment of £24m is seen as the prelude to the company's stock-market flotation and to its expansion into a US market where dance music's potential – at least on the Ministry's very British model of clubs, magazines, radio stations and record production – remains relatively untapped.

The official line was as upbeat as anything you might hear on the Ministry dance floor, but there were other, less favourable interpretations to be placed on a move which follows a period when business has faltered. The Ministry has two major divisions, of which recording is highly successful, media less so. Media covers magazines, radio and the internet, and it was clear that all was not well last winter when redundancies were announced, launches cancelled and operations merged in ways that destabilised the people who were supposed to be running them. There were the losses that often accompany rapid growth, and the Ministry could not escape the effects of the downturn in e-commerce. Palumbo realised that if he was to expand the business, he would need outside help.

How Palumbo handles the new set-up, and where he takes the Ministry from here, remain to be seen. The company retains a powerful brand image, Palumbo's avowedly anti-drugs stance having added to the authority it still enjoys on the youth scene, even if the club and its associated products are not quite as cool as they once were.

Some think he might soon decide to move on, exploring other media opportunities or perhaps building on the quasi-political role in which he cast himself as soon as he began cultivating Peter Mandelson around the time of the 1997 general election, even lending him a chauffeur-driven car. Palumbo scored Brownie points with New Labour for staging last year's New Year's Eve party at the Millennium Dome, and he is exactly the sort of metropolitan mover and shaker that the Government likes to be in touch with. Earlier this year, Palumbo and the former Dome chief, P-Y Gerbeau, linked up in an attempt to buy the attraction.

Others, like his friend Simon Napier-Bell, a leading light in the pop industry since the Sixties, think that, for Palumbo, it's only really about making the next £20m. "He could be the new Richard Branson if he wants to be," Napier-Bell says. "The difference is that he's a much better businessman than Branson, without the need to thrust himself forward."

What's not in doubt is that Palumbo is someone of immense personal resolve, who according to associates will seize on others' weaknesses in getting what he wants and freeze out colleagues he deems to have failed. Which is not to say that he doesn't also command loyalty. But even Napier-Bell says that Palumbo "never thought of business in terms of mutual benefit".

Palumbo was battle-hardened from an early age. Born into a property fortune built up by his Italian-immigrant grandfather, he gained notoriety at Eton, where he controversially acquired his House captaincy a year before the honour was normally bestowed, and had some boys expelled for possessing cannabis. Palumbo has admitted that some people at school saw him as "a monster", and contemporaries confirm his unpopularity. Someone who has known him for many years thinks that he actually thrived on the hostility, and that it prepared him for the much more rancorous conflict that was to come when he took legal action against his father, Peter, the property developer, peer, and former Arts Council chairman, over money in a family trust that James believed was owed to him.

The bitter court case, which James won, was merely the culmination of a feud between father and son that stretched back to the mid-Eighties. Nobody, apart from the two men involved, really seems to know what was at the root of it. A row occurred one Christmas at a time when Peter Palumbo and his ex-wife, James's mother, had been divorced for a few years but were back together at his home in Windsor Great Park as she struggled with cancer. Denia Palumbo died in 1986 and Peter later remarried, and although he and James made an attempt to patch things up in the early Nineties, it didn't work. Then came the court case. "It was just horrible," said someone closely involved at the time. "As it turned out, James was in the right. But it was just the way it was done."

The stand-off continues to this day. In 1997, a couple of years after the case ended, James Palumbo told an interviewer that he "never gives the situation a thought", but that appears no longer to be true. When Lord Palumbo had an operation for cancer last year, his son made enquiries about him through a third party.

Attempts to understand James Palumbo are liable to founder because he is so detached from other people. There have been girlfriends. Anna Pasternak, author of one of the more shameless books about Diana, Princess of Wales, was one. A relationship with a Middle Eastern woman 10 years ago produced a son. The boy's mother has brought him up, for much of the time in Dubai. One old friend who saw Palumbo's move into merchant banking after Oxford as all of a piece with a largely conventional exterior was amazed by the departure that saw him set up the Ministry of Sound. So the club might not have been Palumbo's idea, but the decision to put up the money for it showed that he could not be pigeon-holed.

He is good at talent-spotting and recognising that, on a business level at any rate, he needs other people. But he socialises little with colleagues, and the wealthy lifestyle extends no further than the suits, the immaculate grooming, and the sumptuous flat in Chelsea. No party animal, James Palumbo is the last person you would find enjoying what his own club has to offer. His taste is more for classical music; in particular, says Napier-Bell, Benjamin Britten. "He'd make a good Bond villain," says one former associate. "The chilly, clever loner who wants to take over the world." So that's what's really happening in the glass office up above.