The DIY guide to supercomputers

EARLIER THIS year I wrote about how I built a computer from the parts bins at Fry's, a local Silicon Valley institution. I installed the Linux operating system on it, and said computer is still chugging away as a Web server, telling the tale of how I put it together (at magellan.gulker.com).

Seems as though my idea caught on.

Mike Warren, an astrophysicist at the Los Alamos National Laboratory in New Mexico, recently put the final touches to Avalon, a do-it-yourself supercomputer that his lab team put together using off-the-shelf, high-end DEC Alpha PCs. Avalon recently became the 88th fastest computer on the planet after some new CPUs were hooked up,

Avalon's price tag - $300,000 - is a bit more than the $700 (pounds 425) I spent, but a fraction of what most supercomputers cost. IBM recently installed "Pacific Blue" at the Lawrence Livermore Laboratory for $94m. True, it's more than 80 times faster than Avalon, but it's also 300 times more expensive.

And, Avalon is not alone. There are lots of home-built supercomputers. Thomas Sterling, a computer scientist at the California Institute of Technology and Nasa, stirred things up by publishing a specification for low-cost clusters of PCs called Beowulf. Avalon is just one Beowulf-class supercomputer.

The idea is simple. You set up an arbitrary number of PCs, network them, typically using fast Ethernet, and then send them problems that can be divided up among the machines' processors. One machine acts as a server that syncs up all its clients. Beowulf specs software like the Message Passing Interface (MPI) written under Linux, that lets the machines communicate. And since Linux - the brainchild of the computer science student Linus Torvalds - is free, it keeps the cost down.

One group, at Australia's University of South Queensland, built Topcat, a Pentium-powered, Beowulf-class computer, for just A$8,000 (pounds 3,137). Other Beowulf machines are called Loki (pre-Avalon), Grendel and Megalon and have various price tags, all modest for their power.

The Linux OS has had a pretty phenomenal year, but an abstract on the Argonne National Laboratory website describes a machine called Chicago Pile 6 that runs under Windows NT, and Microsoft offers the "Wolfpack", clusters of Windows NT machines that can be harnessed together for big problems.

And, to round things out, a group at the UCLA Physics Department's Plasma Simulation Group has a $28,000 parallel computer called Appleseed - eight Power Macintosh G3s - which outperforms a Cray Y-MP. UCLA's statistical mathematics department even plans to link up Apple's new iMacs in their computer lab to make a "very pretty supercomputer".

One advantage of the design of most of these machines is that other machines can be roped in as needed for big jobs. The UCLA physics department home page mentions using other Macs in the department at nights and weekends. If the machines are on the network, they're fair game for becoming part of the supercomputer (as long as you can persuade their regular users). UCLA's Macs even use MacMPI, so that problems can be run on Linux or Mac or a "real" supercomputer without rewriting code.

In fact earlier this year, a group called Distributed.net linked thousands of computers of all kinds around the world via the Internet, and cracked a 56-bit encryption code in 40 days. It had previously been thought that such heavyweight ciphers would take hundreds of years to crack, even on fast computers. One version of the Distributed.net program ran as a screen saver that kicked in, and began cracking code, when the machine was idle for more than a few minutes.

You don't normally think about a screen-saver being a powerful program, but when thousands of them act in concert, some serious computing can get done. Distributed.net bills itself as the "Fastest Computer on Earth", even though its hardware bill is effectively zero.

To be sure, the megabucks hardware offers some advantages that the home- brew machines can't match. For one, the expensive machines tend to be more general-purpose, and have an advantage when huge volumes of data have to be loaded.

Beowulf clusters tend to do best on problems that are easily broken up into independent pieces, such as particle simulations, with each machine representing a particle interacting with its neighbours. Distributed.net can crack ciphers only because its client software just knows code-breaking: running a different task would mean reinstalling tens of thousands of client software for every new job.

Some problems just don't chop up easily, and run better on the "big iron". But there aren't many supercomputers around (not at those prices!) and scientists have to queue for time on them. Massive data sets can take so long to load that only a little computing is done in any one window of opportunity.

In fact, it was just this dilemma that prompted Mike Warren and others to build their own supercomputers. They can apply their machines widely, and to classes of problems that would only rarely get a chance on that expensive supercomputer time.

For example, here at gulker.com, our "single-node supercomputer" is churning furiously, trying to figure out how to come up with another $700.

cg@gulker.com

Arts and Entertainment
'Silent Night' last topped Classic FM's favourite Christmas carol poll in 2002
classical
Arts and Entertainment
Caroline Flack became the tenth winner of Strictly Come Dancing
tvReview: 'Absolutely phenomenal' Xtra Factor presenter wins Strictly Come Dancing final
Arts and Entertainment
J Jefferson Farjeon at home in 1953
booksBooksellers say readers are turning away from modern thrillers and back to golden age of crime writing
Arts and Entertainment
Female fans want more explicit male sex in Game of Thrones, George R R Martin says

film George RR Martin owns a cinema in Santa Fe

Arts and Entertainment
Clued up: John Lynch and Gillian Anderson in ‘The Fall’

TV review

PROMOTED VIDEO
Arts and Entertainment
Rhys says: 'I'm not playing it for laughs, but I have learnt that if you fall over on stage, people can enjoy that as much as an amazing guitar solo'
musicGruff Rhys on his rock odyssey, and the trouble with independence
Arts and Entertainment
Krysia and Daniel (Hand out press photograph provided by Sally Richardson)
How do today's composers answer the challenge of the classical giant?
Have you tried new the Independent Digital Edition apps?
Arts and Entertainment

ebooksNow available in paperback
Arts and Entertainment

ebooks
News
Shenaz Treasurywala
film
News
tv
Arts and Entertainment
Jason Watkins as Christopher Jefferies
tv
Arts and Entertainment
Star Wars Director JJ Abrams: key character's names have been revealed
film
Arts and Entertainment
Pharrell Williams won two BBC Music Awards for Best Song and International Artist
music
Arts and Entertainment
Mark, Katie and Sanjay in The Apprentice boardroom
TV
Arts and Entertainment

Film The critics but sneer but these unfashionable festive films are our favourites

Arts and Entertainment
Frances O'Connor and James Nesbitt in 'The Missing'

TV We're so close to knowing what happened to Oliver Hughes, but a last-minute bluff crushes expectations

Arts and Entertainment
Joey Essex will be hitting the slopes for series two of The Jump

TV

Who is taking the plunge?
Arts and Entertainment
Katy Perry as an Ancient Egyptian princess in her latest music video for 'Dark Horse'

music
Arts and Entertainment
Dame Judi Dench, as M in Skyfall

film
Arts and Entertainment
Morrissey, 1988

TV
Latest stories from i100
Have you tried new the Independent Digital Edition apps?

ES Rentals

    Independent Dating
    and  

    By clicking 'Search' you
    are agreeing to our
    Terms of Use.

    Surrounded by high-rise flats is a little house filled with Lebanon’s history - clocks, rifles, frogmen’s uniforms and colonial helmets

    Clocks, rifles, swords, frogmen’s uniforms

    Surrounded by high-rise flats is a little house filled with Lebanon’s history
    Return to Gaza: Four months on, the wounds left by Israel's bombardment have not yet healed

    Four months after the bombardment, Gaza’s wounds are yet to heal

    Kim Sengupta is reunited with a man whose plight mirrors the suffering of the Palestinian people
    Gastric surgery: Is it really the answer to the UK's obesity epidemic?

    Is gastric surgery really the answer to the UK's obesity epidemic?

    Critics argue that it’s crazy to operate on healthy people just to stop them eating
    Homeless Veterans appeal: Christmas charity auction Part 2 - now LIVE

    Homeless Veterans appeal: Christmas charity auction

    Bid on original art, or trips of a lifetime to Africa or the 'Corrie' set, and help Homeless Veterans
    Pantomime rings the changes to welcome autistic theatre-goers

    Autism-friendly theatre

    Pantomime leads the pack in quest to welcome all
    The week Hollywood got scared and had to grow up a bit

    The week Hollywood got scared and had to grow up a bit

    Sony suffered a chorus of disapproval after it withdrew 'The Interview', but it's not too late for it to take a stand, says Joan Smith
    From Widow Twankey to Mother Goose, how do the men who play panto dames get themselves ready for the performance of a lifetime?

    Panto dames: before and after

    From Widow Twankey to Mother Goose, how do the men who play panto dames get themselves ready for the performance of a lifetime?
    Thirties murder mystery novel is surprise runaway Christmas hit

    Thirties murder mystery novel is surprise runaway Christmas hit

    Booksellers say readers are turning away from dark modern thrillers and back to the golden age of crime writing
    Anne-Marie Huby: 'Charities deserve the best,' says founder of JustGiving

    Anne-Marie Huby: 'Charities deserve the best'

    Ten million of us have used the JustGiving website to donate to good causes. Its co-founder says that being dynamic is as important as being kind
    The botanist who hunts for giant trees at Kew Gardens

    The man who hunts giants

    A Kew Gardens botanist has found 25 new large tree species - and he's sure there are more out there
    The 12 ways of Christmas: Spare a thought for those who will be working to keep others safe during the festive season

    The 12 ways of Christmas

    We speak to a dozen people who will be working to keep others safe, happy and healthy over the holidays
    Birdwatching men have a lot in common with their feathered friends, new study shows

    The male exhibits strange behaviour

    A new study shows that birdwatching men have a lot in common with their feathered friends...
    Diaries of Evelyn Waugh, Virginia Woolf and Noël Coward reveal how they coped with the December blues

    Famous diaries: Christmas week in history

    Noël Coward parties into the night, Alan Clark bemoans the cost of servants, Evelyn Waugh ponders his drinking…
    From noble to narky, the fall of the open letter

    From noble to narky, the fall of the open letter

    The great tradition of St Paul and Zola reached its nadir with a hungry worker's rant to Russell Brand, says DJ Taylor
    A Christmas ghost story by Alison Moore: A prodigal daughter has a breakthrough

    A Christmas ghost story by Alison Moore

    The story was published earlier this month in 'Poor Souls' Light: Seven Curious Tales'