Another holy trinity

Merge Web servers, and random-process chunk everything on a supercomput er

EVERY SO often a couple of disparate ideas come together. Mind you, my ideas seem to be coming more and more slowly as the world speeds up and I slow down. So I loved it, after last fortnight's despatch about Beowulf supercomputers, when a couple of bits and pieces began clicking and fitting together.

In 1995, I sat in a Silicon Valley cafe and listened to a programmer called Chuck Shotton describe what he called an RAIC (Redundant Array of Inexpensive Computers). Shotton had just written WebStar, Web server software for the Apple Macintosh.

He thought that four or five inexpensive Macs could keep up with a UNIX Web server, and would be far less expensive to buy, set up and maintain. While WebStar went on to be successful, the RAIC concept never took off.

Fast forward to October 1998. I'm at the Advanced Imaging Conference in San Francisco, listening to a computer scientist, Eric Peters, describe a new way of serving streaming video. Peters was struggling to find a way to make video servers more reliable. High-end digital video editing systems offer great advantages in fast-moving operations such as TV news, but there's a problem on the server side. Serving multiple streams of full-motion video challenges even state-of-the-art equipment. One company, Avid, uses an SGI Origin 2000, a supercomputer, to move the data. Disk storage and retrieval speeds are a potential bottleneck, so the latest RAID technology is employed. RAID (Redundant Array of Individual Disks) divvies the data up into successive "stripes" that can be cued up and ready to go in quick sequence, improving "playback" performance. The problem is that disk drives are still mechanical devices subject to failure.

As the drive count goes up in massive RAID arrays, the reliability goes down. It only takes a single drive failure to bring playback to a halt. RAIDS can rebuild data from a crashed drive, but it takes time, and that's no consolation if the crash occurs while you're feeding live video to millions on the Six O'Clock News.

So Peters looked at mathematical Chaos and Complexity Theory. Both fields suggest that random processes are robust and highly reliable. While "random" may not seem useful to a video editor ("I don't want any old video, I want the bit I want!"), it can be. Incoming data are broken up into small chunks, and then randomly stored on an array of servers. Each chunk is then backed up to another server, also randomly chosen. The chunking software keeps a map of where all the pieces go.

The servers are connected by a network - 100 Base T Ethernet in the case cited by Peters. While it sounds like the RAID, there are differences. For one, a server failure doesn't bring things to a halt: the chunking software merely begins to point to the back-up chunks. Unlike the RAID, the random server array gets more robust as servers are added.

The random process insures that data will be very evenly distributed across all the servers. Peters said that, in tests, the machines could easily saturate the network, and that new servers could be added on the fly as extra capacity was needed. Peters' server cluster sounds exactly like the Beowulf supercomputer arrays described by Tom Sterling, a scientist at the California Institute of Technology. These supercomputers, like Los Alamos Labs' Avalon, consist of large arrays of high-end PCs connected by a fast Ethernet network.

Another link! So what happens if you merge all three ideas - clusters of Web servers and random-process chunking all running on a Beowulf-style supercomputer? Possibly, an answer to some of the most vexing problems in computing. Companies are trying to marry huge corporate databases to Web servers in order to do electronic commerce, and finding it's not easy. The systems get so complex that failures are frequent and maintenance is high.

A server cluster could replace a plethora of interconnected systems. A search engine on each server could provide indexing, like a mini Yahoo or Lycos. Indeed, Apple Computer recently started shipping an engine, called Sherlock, in Mac OS 8.5. The data maps could be set to mimic any kind of data structure, particularly if XML is its language, eliminating the need to replace officeloads of proprietary client software and data terminals.

Voila, Supercomputers 2. You read it here first!

cg@gulker.com

Suggested Topics
Arts and Entertainment
Matthew Healy of The 1975 performing on the Pyramid Stage at the Glastonbury Festival, at Worthy Farm in Somerset

music
Arts and Entertainment
booksThe Withnail and I creator, has a new theory about killer's identity
Have you tried new the Independent Digital Edition apps?
Arts and Entertainment

ebooksNow available in paperback
Arts and Entertainment

ebooks
Arts and Entertainment
tvDick Clement and Ian La Frenais are back for the first time in a decade
Arts and Entertainment
The Clangers: 1969-1974
tv
Arts and Entertainment
Rocky road: Dwayne Johnson and Carla Gugino play an estranged husband and wife in 'San Andreas'
film review
Arts and Entertainment
Nicole Kidman plays Grace Kelly in the film, which was criticised by Monaco’s royal family

film
Arts and Entertainment
Emilia Clarke could have been Anastasia Steele in Fifty Shades of Grey but passed it up because of the nude scenes

film
Arts and Entertainment
A$AP Rocky and Rita Ora pictured together in 2012

music
Arts and Entertainment
A case for Mulder and Scully? David Duchovny and Gillian Anderson in ‘The X-Files’

TV
Arts and Entertainment
Impressions of the Creative Community Courtyard within d3. The development is designed to 'inspire emerging designers and artists, and attract visitors'

architecture
Arts and Entertainment
Performers drink tea at the Glastonbury festival in 2010

GlastonburyWI to make debut appearance at Somerset festival

Arts and Entertainment
Lena Headey as Cersei Lannister

TV reviewIt has taken seven episodes for Game of Thrones season five to hit its stride

Arts and Entertainment
Jesuthasan Antonythasan as Dheepan

FilmPalme d'Or goes to radical and astonishing film that turns conventional thinking about immigrants on its head

Arts and Entertainment
Måns Zelmerlöw performing

Eurovision
Arts and Entertainment
Graham Norton was back in the commentating seat for Eurovision 2015

Eurovision
Arts and Entertainment
Richard Hammond, Jeremy Clarkson and James May on stage

TV
Arts and Entertainment
The light stuff: Britt Robertson and George Clooney in ‘Tomorrowland: a World Beyond’
film review
Arts and Entertainment
Reawakening: can Jon Hamm’s Don Draper find enlightenment in the final ‘Mad Men’?
tv reviewNot quite, but it's an enlightening finale for Don Draper spoiler alert
Arts and Entertainment
Breakfast Show’s Nick Grimshaw

Radio
Arts and Entertainment

Eurovision
Arts and Entertainment
'Youth' cast members Paul Dano, Jane Fonda, Harvey Keitel, Rachel Weisz, and Michael Caine pose for photographers at Cannes Film Festival
film
Arts and Entertainment
Adam West as Batman and Burt Ward and Robin in the 1960s Batman TV show

Comics
Latest stories from i100
Have you tried new the Independent Digital Edition apps?

ES Rentals

    Independent Dating
    and  

    By clicking 'Search' you
    are agreeing to our
    Terms of Use.

    Sepp Blatter resignation: The beginning of Fifa's long road to reform?

    Does Blatter's departure mean Fifa will automatically clean up its act?

    Don't bet on it, says Tom Peck
    Charles Kennedy: The baby of the House who grew into a Lib Dem giant

    The baby of the House who grew into a Lib Dem giant

    Charles Kennedy was consistently a man of the centre-left, dedicated to social justice, but was also a champion of liberty and an opponent of the nanny-state, says Baroness Williams
    Syria civil war: The harrowing testament of a five-year-old victim of this endless conflict

    The harrowing testament of a five-year-old victim of Syria's endless civil war

    Sahar Qanbar lost her mother and brother as civilians and government soldiers fought side by side after being surrounded by brutal Islamist fighters. Robert Fisk visited her
    The future of songwriting: How streaming is changing everything we know about making music

    The future of songwriting

    How streaming is changing everything we know about making music
    William Shemin and Henry Johnson: Jewish and black soldiers receive World War I Medal of Honor amid claims of discrimination

    Recognition at long last

    Jewish and black soldiers who fought in WWI finally receive medals after claims of discrimination
    Beating obesity: The new pacemaker which helps over-eaters

    Beating obesity

    The new pacemaker which helps over-eaters
    9 best women's festival waterproofs

    Ready for rain: 9 best women's festival waterproofs

    These are the macs to keep your denim dry and your hair frizz-free(ish)
    Cycling World Hour Record: Nervous Sir Bradley Wiggins ready for pain as he prepares to go distance

    Wiggins worried

    Nervous Sir Bradley ready for pain as he prepares to attempt cycling's World Hour Record
    Liverpool close in on Milner signing

    Liverpool close in on Milner signing

    Reds baulk at Christian Benteke £32.5m release clause
    On your feet! Spending at least two hours a day standing reduces the risk of heart attacks, cancer and diabetes, according to new research

    On your feet!

    Spending half the day standing 'reduces risk of heart attacks and cancer'
    With scores of surgeries closing, what hope is there for the David Cameron's promise of 5,000 more GPs and a 24/7 NHS?

    The big NHS question

    Why are there so few new GPs when so many want to study medicine?
    Big knickers are back: Thongs ain't what they used to be

    Thongs ain't what they used to be

    Big knickers are back
    Thurston Moore interview

    Thurston Moore interview

    On living in London, Sonic Youth and musical memoirs
    In full bloom

    In full bloom

    Floral print womenswear
    From leading man to Elephant Man, Bradley Cooper is terrific

    From leading man to Elephant Man

    Bradley Cooper is terrific