Another holy trinity

Merge Web servers, and random-process chunk everything on a supercomput er

EVERY SO often a couple of disparate ideas come together. Mind you, my ideas seem to be coming more and more slowly as the world speeds up and I slow down. So I loved it, after last fortnight's despatch about Beowulf supercomputers, when a couple of bits and pieces began clicking and fitting together.

In 1995, I sat in a Silicon Valley cafe and listened to a programmer called Chuck Shotton describe what he called an RAIC (Redundant Array of Inexpensive Computers). Shotton had just written WebStar, Web server software for the Apple Macintosh.

He thought that four or five inexpensive Macs could keep up with a UNIX Web server, and would be far less expensive to buy, set up and maintain. While WebStar went on to be successful, the RAIC concept never took off.

Fast forward to October 1998. I'm at the Advanced Imaging Conference in San Francisco, listening to a computer scientist, Eric Peters, describe a new way of serving streaming video. Peters was struggling to find a way to make video servers more reliable. High-end digital video editing systems offer great advantages in fast-moving operations such as TV news, but there's a problem on the server side. Serving multiple streams of full-motion video challenges even state-of-the-art equipment. One company, Avid, uses an SGI Origin 2000, a supercomputer, to move the data. Disk storage and retrieval speeds are a potential bottleneck, so the latest RAID technology is employed. RAID (Redundant Array of Individual Disks) divvies the data up into successive "stripes" that can be cued up and ready to go in quick sequence, improving "playback" performance. The problem is that disk drives are still mechanical devices subject to failure.

As the drive count goes up in massive RAID arrays, the reliability goes down. It only takes a single drive failure to bring playback to a halt. RAIDS can rebuild data from a crashed drive, but it takes time, and that's no consolation if the crash occurs while you're feeding live video to millions on the Six O'Clock News.

So Peters looked at mathematical Chaos and Complexity Theory. Both fields suggest that random processes are robust and highly reliable. While "random" may not seem useful to a video editor ("I don't want any old video, I want the bit I want!"), it can be. Incoming data are broken up into small chunks, and then randomly stored on an array of servers. Each chunk is then backed up to another server, also randomly chosen. The chunking software keeps a map of where all the pieces go.

The servers are connected by a network - 100 Base T Ethernet in the case cited by Peters. While it sounds like the RAID, there are differences. For one, a server failure doesn't bring things to a halt: the chunking software merely begins to point to the back-up chunks. Unlike the RAID, the random server array gets more robust as servers are added.

The random process insures that data will be very evenly distributed across all the servers. Peters said that, in tests, the machines could easily saturate the network, and that new servers could be added on the fly as extra capacity was needed. Peters' server cluster sounds exactly like the Beowulf supercomputer arrays described by Tom Sterling, a scientist at the California Institute of Technology. These supercomputers, like Los Alamos Labs' Avalon, consist of large arrays of high-end PCs connected by a fast Ethernet network.

Another link! So what happens if you merge all three ideas - clusters of Web servers and random-process chunking all running on a Beowulf-style supercomputer? Possibly, an answer to some of the most vexing problems in computing. Companies are trying to marry huge corporate databases to Web servers in order to do electronic commerce, and finding it's not easy. The systems get so complex that failures are frequent and maintenance is high.

A server cluster could replace a plethora of interconnected systems. A search engine on each server could provide indexing, like a mini Yahoo or Lycos. Indeed, Apple Computer recently started shipping an engine, called Sherlock, in Mac OS 8.5. The data maps could be set to mimic any kind of data structure, particularly if XML is its language, eliminating the need to replace officeloads of proprietary client software and data terminals.

Voila, Supercomputers 2. You read it here first!

cg@gulker.com

Suggested Topics
PROMOTED VIDEO
Arts and Entertainment
The frill of it all: Hattie Morahan in 'The Changeling'

theatre
Arts and Entertainment
Gillian Anderson and David Duchovny may reunite for The X Files

tv
Have you tried new the Independent Digital Edition apps?
Arts and Entertainment

ebooksNow available in paperback
Arts and Entertainment

ebooks
Arts and Entertainment
Jeremy Clarkson, left, and Richard Hammond upset the locals in South America
TV
News
A young woman punched a police officer after attending a gig by US rapper Snoop Dogg
people
Arts and Entertainment
Reese Witherspoon starring in 'Wild'

It's hard not to warm to Reese Witherspoon's heroismfilm
Arts and Entertainment
Word up: Robbie Coltrane as dictionary guru Doctor Johnson in the classic sitcom Blackadder the Third
books

Arts and Entertainment
The Oscar nominations are due to be announced today

Oscars 2015
Arts and Entertainment
Hacked off: Maisie Williams in ‘Cyberbully’

Maisie Williams single-handedly rises to the challenge

TV
Arts and Entertainment
Eddie Redmayne in The Theory of Everything and Benedict Cumberbatch in The Imitation Game are both nominated at the Bafta Film Awards
Arts and Entertainment

Academy criticised after no non-white actors nominated

Arts and Entertainment
Damian Lewis shooting a scene as Henry VIII in Wolf Hall
TV

Arts and Entertainment
A history of violence: ‘Angry, White and Proud’ looked at the rise of far-right groups

tv

An expose of hooliganism masquerading as an ideological battle

Arts and Entertainment

art

Lee Hadwin can't draw when he's awake, but by night he's an artist

Arts and Entertainment

TV

Arts and Entertainment
Michael Keaton in the 1998 Beetlejuice original

film

Arts and Entertainment

TV

Arts and Entertainment
Olivia Colman and David Tennant star in 'Broadchurch'

TV

Arts and Entertainment
Michael Kitchen plays Christopher Foyle in ITV's 'Foyle's War'

TV
Arts and Entertainment

TV
Arts and Entertainment
Downton Abbey star Joanne Froggatt will be starring in Dominic Savage's new BBC drama The Secrets

Arts and Entertainment
Vividly drawn: Timothy Spall in Mike Leigh’s ‘Mr Turner’
film
Latest stories from i100
Have you tried new the Independent Digital Edition apps?

ES Rentals

    Independent Dating
    and  

    By clicking 'Search' you
    are agreeing to our
    Terms of Use.

    Greece elections: In times like these, the EU has far more dangerous adversaries than Syriza

    Greece elections

    In times like these, the EU has far more dangerous adversaries than Syriza, says Patrick Cockburn
    Front National family feud? Marine Le Pen and her relatives clash over French far-right party's response to Paris terror attacks

    Front National family feud?

    Marine Le Pen and her relatives clash over French far-right party's response to Paris terror attacks
    Woman who was sent to three Nazi death camps describes how she escaped the gas chamber

    Auschwitz liberation 70th anniversary

    Woman sent to three Nazi death camps describes surviving gas chamber
    DSK, Dodo the Pimp, and the Carlton Hotel

    The inside track on France's trial of the year

    Dominique Strauss-Kahn, Dodo the Pimp, and the Carlton Hotel:
    As provocative now as they ever were

    Sarah Kane season

    Why her plays are as provocative now as when they were written
    Murder of Japanese hostage has grim echoes of a killing in Iraq 11 years ago

    Murder of Japanese hostage has grim echoes of another killing

    Japanese mood was against what was seen as irresponsible trips to a vicious war zone
    Syria crisis: Celebrities call on David Cameron to take more refugees as one young mother tells of torture by Assad regime

    Celebrities call on David Cameron to take more Syrian refugees

    One young mother tells of torture by Assad regime
    The enemy within: People who hear voices in their heads are being encouraged to talk back – with promising results

    The enemy within

    People who hear voices in their heads are being encouraged to talk back
    'In Auschwitz you got used to anything'

    'In Auschwitz you got used to anything'

    Survivors of the Nazi concentration camp remember its horror, 70 years on
    Autumn/winter menswear 2015: The uniforms that make up modern life come to the fore

    Autumn/winter menswear 2015

    The uniforms that make up modern life come to the fore
    'I'm gay, and plan to fight military homophobia'

    'I'm gay, and plan to fight military homophobia'

    Army general planning to come out
    Iraq invasion 2003: The bloody warnings six wise men gave to Tony Blair as he prepared to launch poorly planned campaign

    What the six wise men told Tony Blair

    Months before the invasion of Iraq in 2003, experts sought to warn the PM about his plans. Here, four of them recall that day
    25 years of The Independent on Sunday: The stories, the writers and the changes over the last quarter of a century

    25 years of The Independent on Sunday

    The stories, the writers and the changes over the last quarter of a century
    Homeless Veterans appeal: 'Really caring is a dangerous emotion in this kind of work'

    Homeless Veterans appeal

    As head of The Soldiers' Charity, Martin Rutledge has to temper compassion with realism. He tells Chris Green how his Army career prepared him
    Wu-Tang Clan and The Sexual Objects offer fans a chance to own the only copies of their latest albums

    Smash hit go under the hammer

    It's nice to pick up a new record once in a while, but the purchasers of two latest releases can go a step further - by buying the only copy