Another holy trinity

Merge Web servers, and random-process chunk everything on a supercomput er

EVERY SO often a couple of disparate ideas come together. Mind you, my ideas seem to be coming more and more slowly as the world speeds up and I slow down. So I loved it, after last fortnight's despatch about Beowulf supercomputers, when a couple of bits and pieces began clicking and fitting together.

In 1995, I sat in a Silicon Valley cafe and listened to a programmer called Chuck Shotton describe what he called an RAIC (Redundant Array of Inexpensive Computers). Shotton had just written WebStar, Web server software for the Apple Macintosh.

He thought that four or five inexpensive Macs could keep up with a UNIX Web server, and would be far less expensive to buy, set up and maintain. While WebStar went on to be successful, the RAIC concept never took off.

Fast forward to October 1998. I'm at the Advanced Imaging Conference in San Francisco, listening to a computer scientist, Eric Peters, describe a new way of serving streaming video. Peters was struggling to find a way to make video servers more reliable. High-end digital video editing systems offer great advantages in fast-moving operations such as TV news, but there's a problem on the server side. Serving multiple streams of full-motion video challenges even state-of-the-art equipment. One company, Avid, uses an SGI Origin 2000, a supercomputer, to move the data. Disk storage and retrieval speeds are a potential bottleneck, so the latest RAID technology is employed. RAID (Redundant Array of Individual Disks) divvies the data up into successive "stripes" that can be cued up and ready to go in quick sequence, improving "playback" performance. The problem is that disk drives are still mechanical devices subject to failure.

As the drive count goes up in massive RAID arrays, the reliability goes down. It only takes a single drive failure to bring playback to a halt. RAIDS can rebuild data from a crashed drive, but it takes time, and that's no consolation if the crash occurs while you're feeding live video to millions on the Six O'Clock News.

So Peters looked at mathematical Chaos and Complexity Theory. Both fields suggest that random processes are robust and highly reliable. While "random" may not seem useful to a video editor ("I don't want any old video, I want the bit I want!"), it can be. Incoming data are broken up into small chunks, and then randomly stored on an array of servers. Each chunk is then backed up to another server, also randomly chosen. The chunking software keeps a map of where all the pieces go.

The servers are connected by a network - 100 Base T Ethernet in the case cited by Peters. While it sounds like the RAID, there are differences. For one, a server failure doesn't bring things to a halt: the chunking software merely begins to point to the back-up chunks. Unlike the RAID, the random server array gets more robust as servers are added.

The random process insures that data will be very evenly distributed across all the servers. Peters said that, in tests, the machines could easily saturate the network, and that new servers could be added on the fly as extra capacity was needed. Peters' server cluster sounds exactly like the Beowulf supercomputer arrays described by Tom Sterling, a scientist at the California Institute of Technology. These supercomputers, like Los Alamos Labs' Avalon, consist of large arrays of high-end PCs connected by a fast Ethernet network.

Another link! So what happens if you merge all three ideas - clusters of Web servers and random-process chunking all running on a Beowulf-style supercomputer? Possibly, an answer to some of the most vexing problems in computing. Companies are trying to marry huge corporate databases to Web servers in order to do electronic commerce, and finding it's not easy. The systems get so complex that failures are frequent and maintenance is high.

A server cluster could replace a plethora of interconnected systems. A search engine on each server could provide indexing, like a mini Yahoo or Lycos. Indeed, Apple Computer recently started shipping an engine, called Sherlock, in Mac OS 8.5. The data maps could be set to mimic any kind of data structure, particularly if XML is its language, eliminating the need to replace officeloads of proprietary client software and data terminals.

Voila, Supercomputers 2. You read it here first!

Suggested Topics
Arts and Entertainment

Filming to begin on two new series due to be aired on Dave from next year


Arts and Entertainment
Kit Harington plays MI5 agent Will Holloway in Spooks: The Greater Good

'You can't count on anyone making it out alive'film
Arts and Entertainment
War veteran and father of Peter and Laust Thoger Jensen played by Lars Mikkelson

TVBBC hopes latest Danish import will spell success

Arts and Entertainment
Carey Mulligan in Far From The Madding Crowd
FilmCarey Mulligan’s Bathsheba would fit in better in The Hunger Games
Arts and Entertainment
Pandas-on-heat: Mary Ramsden's contribution is intended to evoke the compound the beasts smear around their habitat
Iart'm Here But You've Gone exhibition has invited artists to produce perfumes
Have you tried new the Independent Digital Edition apps?
Arts and Entertainment

ebooksNow available in paperback
Arts and Entertainment

  • Get to the point
Latest stories from i100
Have you tried new the Independent Digital Edition apps?

ES Rentals

    Independent Dating

    By clicking 'Search' you
    are agreeing to our
    Terms of Use.

    'It was first time I had ever tasted chocolate. I kept a piece, and when Amsterdam was liberated, I gave it to the first Allied soldier I saw'

    Bread from heaven

    Dutch survivors thank RAF for World War II drop that saved millions
    Britain will be 'run for the wealthy and powerful' if Tories retain power - Labour

    How 'the Axe' helped Labour

    UK will be 'run for the wealthy and powerful' if Tories retain power
    Rare and exclusive video shows the horrific price paid by activists for challenging the rule of jihadist extremists in Syria

    The price to be paid for challenging the rule of extremists

    A revolution now 'consuming its own children'
    Welcome to the world of Megagames

    Welcome to the world of Megagames

    300 players take part in Watch the Skies! board game in London
    'Nymphomaniac' actress reveals what it was really like to star in one of the most explicit films ever

    Charlotte Gainsbourg on 'Nymphomaniac'

    Starring in one of the most explicit films ever
    Robert Fisk in Abu Dhabi: The Emirates' out-of-sight migrant workers helping to build the dream projects of its rulers

    Robert Fisk in Abu Dhabi

    The Emirates' out-of-sight migrant workers helping to build the dream projects of its rulers
    Vince Cable interview: Charging fees for employment tribunals was 'a very bad move'

    Vince Cable exclusive interview

    Charging fees for employment tribunals was 'a very bad move'
    Iwan Rheon interview: Game of Thrones star returns to his Welsh roots to record debut album

    Iwan Rheon is returning to his Welsh roots

    Rheon is best known for his role as the Bastard of Bolton. It's gruelling playing a sadistic torturer, he tells Craig McLean, but it hasn't stopped him recording an album of Welsh psychedelia
    Russell Brand's interview with Ed Miliband has got everyone talking about The Trews

    Everyone is talking about The Trews

    Russell Brand's 'true news' videos attract millions of viewers. But today's 'Milibrand' interview introduced his resolutely amateurish style to a whole new crowd
    Morne Hardenberg interview: Cameraman for BBC's upcoming show Shark on filming the ocean's most dangerous predator

    It's time for my close-up

    Meet the man who films great whites for a living
    Increasing numbers of homeless people in America keep their mobile phones on the streets

    Homeless people keep mobile phones

    A homeless person with a smartphone is a common sight in the US. And that's creating a network where the 'hobo' community can share information - and fight stigma - like never before
    'Queer saint' Peter Watson left his mark on British culture by bankrolling artworld giants

    'Queer saint' who bankrolled artworld giants

    British culture owes a huge debt to Peter Watson, says Michael Prodger
    Pushkin Prizes: Unusual exchange programme aims to bring countries together through culture

    Pushkin Prizes brings countries together

    Ten Scottish schoolchildren and their Russian peers attended a creative writing workshop in the Highlands this week
    14 best kids' hoodies

    14 best kids' hoodies

    Don't get caught out by that wind on the beach. Zip them up in a lightweight top to see them through summer to autumn
    Robert Fisk in Abu Dhabi: The acceptable face of the Emirates

    The acceptable face of the Emirates

    Has Abu Dhabi found a way to blend petrodollars with principles, asks Robert Fisk