Sir Simon Rattle and the Berlin Philarmonic: Is this great relationship ending on a sour note?

The rumour mill turning but Jessica Duchen finds out what really drove him to leave

The classical music world has been ablaze with speculation following the announcement that Sir Simon Rattle is to step down from his post as music director of the Berlin Philharmonic in five years' time. He will have spent 16 years at the head of what's usually considered the greatest orchestra in the world.

That span would normally be a "good innings" for such a partnership. Yet given the organisation's stature, it's hard to imagine that anybody might choose to leave. With the exception of Rattle's predecessor, Claudio Abbado, its chief conductors – including Wilhelm Furtwängler and Herbert von Karajan – have mostly kept the job until death. Rattle's tenure, though, has been a roller coaster from the beginning.

This orchestra is one of only a handful in which the players elect their music directors democratically, and after Abbado's resignation, the vote was a close contest between Rattle and Daniel Barenboim – with the latter thought to be the favourite.

Since his first season in 2002, Rattle has overseen a period of unprecedented change involving a veritable revolution in attitude. He instigated educational and outreach initiatives, tackled new, adventurous repertoire, fought for a pay rise for his players and was involved in the development of the high-quality Digital Concert Hall, which webcasts the orchestra's concerts.

Before he went to Berlin, the orchestra had no education department.

Change is always controversial in a field as mired in tradition as German orchestral music; there would have been a real tension between the orchestra's mighty past and its radicalised future. Under Karajan's leadership, the Berlin Philharmonic developed an unrivalled beauty of sound in 19th-century classics. Yet that repertoire has never been Rattle's first priority.

He prefers to experiment and innovate, whether working with "historically informed performance" specialists – a style antithetical to Karajan's – or championing the 20th-century works in which he has excelled. He has transformed the orchestra: today it is younger, fresher, eager to embrace the new. It is unlikely to look back.

Some of the players and staff are said to regard Rattle as the best thing that could have happened to them, and to be devastated by his decision. Other rumblings, though, suggest that after tensions in the past, the relationship had run its course and ennui was starting to encroach. Some commentators have alleged a "rebellion", but the fact is that musicians in orchestras everywhere are prone to severe griping about their conductors. I've never met an orchestra that has no dissenters.

Rattle, who turns 58 this week, is at a stage when many men in high-stress posts take stock of life and its progress; and he is certainly not someone to take such a seismic decision without long and thorough consideration. Friends praise his instinct for recognising the right time to make a change. It may be a cliché to say that he wants to spend more time with his family, but the likelihood is that he really does. He is married to the mezzo-soprano Magdalena Kozena and they have two small sons; work-life balance could scarcely help but be an issue.

Moreover, it is good for him to move on while his options for the future remain wide open. He will be nearly 64 in 2018: old enough to pick and choose as a freelancer, but by no means out of the running for a music directorship elsewhere, should he want one. Some interesting vacancies could emerge around that time – for instance, the New York Philharmonic after the 2016-17 season – and alternatively, don't forget that Rattle, today's pre-eminent British conductor, has never held a post in London. There could yet be a chance for him to come home in glory. With five years' notice, any organisations with their eye on him have plenty of time to mobilise their butterfly nets.

Indeed, five years is so much time that there are plenty of unknown quantities to deal with in London. It is unlikely that Rattle would want to follow his Berlin years with any organisation of lesser stature. And therefore, should London be in his sights at all, it would have to be either the London Symphony Orchestra or the Royal Opera House. The London Philharmonic and the Philharmonia, both resident at the Royal Festival Hall, are less well off financially than their Barbican-based rival, for a start; and when the time comes, they are more likely to replace their current magnificent chief conductors – respectively Vladimir Jurowski and Esa-Pekka Salonen – with younger (and possibly less expensive?) rising stars.

At the ROH, Sir Antonio Pappano's current contract will be up in 2015; as yet, the extremely popular maestro has given no indication of whether or not he plans to stay on. He will be a tough act to follow, too. Should he extend the contract for another three years, the timing could be perfect for a much-loved figure who becomes free at exactly that time. At the LSO, Valery Gergiev is on a "rolling contract" – again, it is difficult to predict how far it will roll as time goes by, and the Russian superconductor is only about two years older than Rattle. But once again, it might by then be time for a change; and the LSO, with its extra wing of activities at St Luke's, would be well-placed to attract a conductor who wants an orchestra that performs at world-class level yet also has a flourishing and creative education centre.

Meanwhile, everyone is wondering who Rattle's successor in Berlin could be. Will they choose a luxury traditionalist like Christian Thielemann, widely tipped as hot favourite; or another radical, eclectic explorer, such as Jurowski? The next great soap opera of classical music is underway.