Science: Phone-in frenzy made BT see red: Its futuristic nerve-centre can't always stop the network jamming. Susan Watts reports

WHEN hundreds of thousands of excited couch potatoes rang their favourite television talent show to place their votes, it paralysed British Telecom's network in Greater Manchester, Cleveland and parts of Scotland. It was BT's worst crash.

The company claims that within half an hour of the mass phone-in to the Granada Television show Stars in their Eyes three weeks ago, the network was back to normal - but last week it refused to say how many lines were out of service, and for how long. Most seriously, some customers of affected trunk exchanges would not have been able to get through to the emergency services. The company is conducting an urgent internal inquiry to find out exactly what went wrong.

BT had provided 10,000 lines for the event, but there were more calls than even Granada had anticipated. 'We had over 10 million call attempts, over 80 per cent of which were made during a 15-minute period,' a spokeswoman said. She added that the presenter of the show sought to raise the drama by encouraging people to wait until the last possible moment to call.

Most of the call attempts would have been the result of frantic fingers pressing recall buttons over and over again. BT had to introduce a procedure called 'call-gapping' to filter out all but a certain number of calls, at random, every second.

BT would not comment on whether the problems affected the result of the show. 'Granada is sticking with its result,' the spokeswoman said.

The timing of the incident was unfortunate for BT's image-makers, coming just four days after the company invited journalists to its high- security network management centre at Oswestry, Shropshire. From there, managers control every exchange on Britain's network - plus overseas links - changing the way they handle calls to overcome or avoid congestion. Network controllers maintain a 24- hour vigil in front of vast maps of the telephone system displayed on a wall of video screens 10ft high and 70ft long.

The television talent show, like charity appeals and disaster information lines, was exactly the type of situation the Oswestry centre is designed to handle.

The BT spokeswoman said it is still not clear exactly what went wrong. In a statement last week, BT said: 'There are technical limitations on the capacity of any network. Clearly there are also limitations on the network management centre's ability to implement call-gapping controls within such a short time-scale, given the volume of calls.'

The spokeswoman said that the wave of calls overloaded a number of local exchanges. Within seven minutes controllers moved to protect the rest of the network, stepping up call- gapping from a setting that allowed one call every fifth of a second to one just every four seconds.

Journalists shown around the Oswestry centre saw the result of a slick public relations operation. The women wear blue spotted blouses, men the corporate tie. John Davies, the centre's customer relations manager, said this 'image clothing' helps to show visiting customers that it is 'a clean and tidy operation'.

To the journalists intrigued by rumours that the Oswestry centre is the focus of MI5's telephone tapping activities, the visit yielded no clues. Mr Davies said: 'We do not carry out telephone tapping from here. There must be some confusion over the use of the word surveillance.'

Last month's crash was not the first time that Oswestry has been caught out. In July 1990, barely a month after the first batch of prestige customers was shown around the centre, Tina Turner fans caused a similar, more limited crash in the Bristol area.

An advertisement in a national newspaper had promised free tickets for a Tina Turner concert to fans who bought a bottle of cola, slotted two digits from its barcode into the gaps in a Bristol telephone number and rang between 6pm and 7pm. Shortly after 6pm on 24 July, the patch of 'video wall' at Oswestry covering the West Country burst into life. Red alarm lines sprouted around Bristol.

The Bristol-based company that had taken out the advert had laid on just 10 telephone lines. Within a minute, hundreds of thousands of devotees were trying to ring in.

As far as the company knew it was receiving about 50 calls every quarter of an hour. What it did not know was that in the same period, around 200,000 calls were failing. Exchange after exchange was hitting its maximum call capacity and giving up. Call-gapping again saved more of the network from disaster.

Since this experience, BT has been encouraging television companies to warn them if the network is likely to come under stress because of a programme. It coped with more than two million calls in the five minutes following this year's Song For Europe contest, when viewers rang to pick the British entry.

The Oswestry centre checks the state of each exchange or 'switch' and keeps track of how well it is transmitting to the others. It also monitors the volume of traffic that is ebbing and flowing through the arteries and veins of the network.

More than 80 per cent of BT's network consists of optical fibre. Each one-inch thick cable, carrying 96 fibres, can cope with up to 370,000 simultaneous calls. A minimum of two lines run from each digital local exchange to the trunk network, providing extra protection.

If a call fails to get through after a certain number of attempts, the exchanges (which are 'semi-intelligent') automatically send the call via a less congested route. Just as the local exchanges replace the rows of women telephone operators of Victorian times, Oswestry replaces their supervisor, guiding an exchange on how to redirect a call if an exchange's choice fails.

Computers at Oswestry fire off 'queries' to telephone exchanges around the country, asking each, in digital code, how it is performing. Over a two-minute period, the centre gathers performance data from more than 700 exchanges. During the following two minutes it analyses this data to see how traffic is faring. This data is displayed on the 'video wall' at Oswestry.

The video wall, made up of a patchwork of 140 video screens, is fed by red, blue and green rear projectors. These illuminate the branches of the network to draw the operators' attention to a problem. A red connection is the most severe warning, ranging through orange, yellow and green to blue, for the least serious fault or congestion. Lines linking exchanges appear coloured along only half their length, stretching from the end experiencing difficulties.

Congestion could mean three or three hundred calls are failing, so operators consult a database to find out the exact number of calls in difficulty. They then ring through to a local exchange for clues to what is going wrong and whether the problem is a technical fault or overloaded lines.

Most of the fault detecting and fixing takes place at three subsidiary network management units around the country, while Oswestry concentrates on overall management of the network and international links.

Procedures put in place after the Tina Turner incident were supposed to stop anything similar happening again. Yet it did, and dramatically. The company's internal inquiry into the talent-show crash should yield results in the next few weeks.

What is already clear is that liaison must be improved so that everyone is aware of the limitations of BT's network. And BT must be big enough to tell television producers that it simply cannot meet some requests - for the sake of the rest of its customers.

(Photograph omitted)

Start your day with The Independent, sign up for daily news emails
News
Ben Little, right, is a Labour supporter while Jonathan Rogers supports the Green Party
general election 2015
News
The 91st Hakone Ekiden Qualifier at Showa Kinen Park, Tokyo, 2014
news
Life and Style
Former helicopter pilot Major Tim Peake will become the first UK astronaut in space for over 20 years
food + drinkNothing but the best for British astronaut as chef Heston Blumenthal cooks up his rations
News
Kim Wilde began gardening in the 1990s when she moved to the countryside
peopleThe singer is leading an appeal for the charity Thrive, which uses the therapy of horticulture
Sport
Alexis Sanchez celebrates scoring a second for Arsenal against Reading
football
Life and Style
health
Voices
An easy-peel potato; Dave Hax has come up with an ingenious method in food preparation
voicesDave Hax's domestic tips are reminiscent of George Orwell's tea routine. The world might need revolution, but we like to sweat the small stuff, says DJ Taylor
News
i100
News
Japan's population is projected to fall dramatically in the next 50 years (Wikimedia)
news
Life and Style
Buyers of secondhand cars are searching out shades last seen in cop show ‘The Sweeney’
motoringFlares and flounce are back on catwalks but a revival in ’70s car paintjobs was a stack-heeled step too far – until now
ebooks
ebooksA special investigation by Andy McSmith
  • Get to the point
Latest stories from i100
Have you tried new the Independent Digital Edition apps?
Independent Dating
and  

By clicking 'Search' you
are agreeing to our
Terms of Use.

iJobs Job Widget
iJobs General

Recruitment Genius: Project Implementation Executive

£18000 - £23000 per annum: Recruitment Genius: They work with major vehicle ma...

Recruitment Genius: Chiropractic Assistant

£16500 per annum: Recruitment Genius: A Chiropractic Assistant is needed in a ...

Recruitment Genius: Digital Account Executive - Midlands

£18000 - £26000 per annum: Recruitment Genius: They work with major vehicle ma...

Recruitment Genius: Web Developer

£28000 - £30000 per annum: Recruitment Genius: This company provides coaching ...

Day In a Page

NHS struggling to monitor the safety and efficacy of its services outsourced to private providers

Who's monitoring the outsourced NHS services?

A report finds that private firms are not being properly assessed for their quality of care
Zac Goldsmith: 'I'll trigger a by-election over Heathrow'

Zac Goldsmith: 'I'll trigger a by-election over Heathrow'

The Tory MP said he did not want to stand again unless his party's manifesto ruled out a third runway. But he's doing so. Watch this space
How do Greek voters feel about Syriza's backtracking on its anti-austerity pledge?

How do Greeks feel about Syriza?

Five voters from different backgrounds tell us what they expect from Syriza's charismatic leader Alexis Tsipras
From Iraq to Libya and Syria: The wars that come back to haunt us

The wars that come back to haunt us

David Cameron should not escape blame for his role in conflicts that are still raging, argues Patrick Cockburn
Sam Baker and Lauren Laverne: Too busy to surf? Head to The Pool

Too busy to surf? Head to The Pool

A new website is trying to declutter the internet to help busy women. Holly Williams meets the founders
Heston Blumenthal to cook up a spice odyssey for British astronaut manning the International Space Station

UK's Major Tum to blast off on a spice odyssey

Nothing but the best for British astronaut as chef Heston Blumenthal cooks up his rations
John Harrison's 'longitude' clock sets new record - 300 years on

‘Longitude’ clock sets new record - 300 years on

Greenwich horologists celebrate as it keeps to within a second of real time over a 100-day test
Fears in the US of being outgunned in the vital propaganda wars by Russia, China - and even Isis - have prompted a rethink on overseas broadcasters

Let the propaganda wars begin - again

'Accurate, objective, comprehensive': that was Voice of America's creed, but now its masters want it to promote US policy, reports Rupert Cornwell
Why Japan's incredible long-distance runners will never win the London Marathon

Japan's incredible long-distance runners

Every year, Japanese long-distance runners post some of the world's fastest times – yet, come next weekend, not a single elite competitor from the country will be at the London Marathon
Why does Tom Drury remain the greatest writer you've never heard of?

Tom Drury: The quiet American

His debut was considered one of the finest novels of the past 50 years, and he is every bit the equal of his contemporaries, Jonathan Franzen, Dave Eggers and David Foster Wallace
You should judge a person by how they peel a potato

You should judge a person by how they peel a potato

Dave Hax's domestic tips are reminiscent of George Orwell's tea routine. The world might need revolution, but we like to sweat the small stuff, says DJ Taylor
Beige is back: The drab car colours of the 1970s are proving popular again

Beige to the future

Flares and flounce are back on catwalks but a revival in ’70s car paintjobs was a stack-heeled step too far – until now
Bill Granger recipes: Our chef's dishes highlight the delicate essence of fresh cheeses

Bill Granger cooks with fresh cheeses

More delicate on the palate, milder, fresh cheeses can also be kinder to the waistline
Aston Villa vs Liverpool: 'This FA Cup run has been wonderful,' says veteran Shay Given

Shay Given: 'This FA Cup run has been wonderful'

The Villa keeper has been overlooked for a long time and has unhappy memories of the national stadium – but he is savouring his chance to play at Wembley
Timeless drama of Championship race in league of its own - Michael Calvin

Michael Calvin's Last Word

Timeless drama of Championship race in league of its own