The DIY guide to supercomputers

EARLIER THIS year I wrote about how I built a computer from the parts bins at Fry's, a local Silicon Valley institution. I installed the Linux operating system on it, and said computer is still chugging away as a Web server, telling the tale of how I put it together (at magellan.gulker.com).

Seems as though my idea caught on.

Mike Warren, an astrophysicist at the Los Alamos National Laboratory in New Mexico, recently put the final touches to Avalon, a do-it-yourself supercomputer that his lab team put together using off-the-shelf, high-end DEC Alpha PCs. Avalon recently became the 88th fastest computer on the planet after some new CPUs were hooked up,

Avalon's price tag - $300,000 - is a bit more than the $700 (pounds 425) I spent, but a fraction of what most supercomputers cost. IBM recently installed "Pacific Blue" at the Lawrence Livermore Laboratory for $94m. True, it's more than 80 times faster than Avalon, but it's also 300 times more expensive.

And, Avalon is not alone. There are lots of home-built supercomputers. Thomas Sterling, a computer scientist at the California Institute of Technology and Nasa, stirred things up by publishing a specification for low-cost clusters of PCs called Beowulf. Avalon is just one Beowulf-class supercomputer.

The idea is simple. You set up an arbitrary number of PCs, network them, typically using fast Ethernet, and then send them problems that can be divided up among the machines' processors. One machine acts as a server that syncs up all its clients. Beowulf specs software like the Message Passing Interface (MPI) written under Linux, that lets the machines communicate. And since Linux - the brainchild of the computer science student Linus Torvalds - is free, it keeps the cost down.

One group, at Australia's University of South Queensland, built Topcat, a Pentium-powered, Beowulf-class computer, for just A$8,000 (pounds 3,137). Other Beowulf machines are called Loki (pre-Avalon), Grendel and Megalon and have various price tags, all modest for their power.

The Linux OS has had a pretty phenomenal year, but an abstract on the Argonne National Laboratory website describes a machine called Chicago Pile 6 that runs under Windows NT, and Microsoft offers the "Wolfpack", clusters of Windows NT machines that can be harnessed together for big problems.

And, to round things out, a group at the UCLA Physics Department's Plasma Simulation Group has a $28,000 parallel computer called Appleseed - eight Power Macintosh G3s - which outperforms a Cray Y-MP. UCLA's statistical mathematics department even plans to link up Apple's new iMacs in their computer lab to make a "very pretty supercomputer".

One advantage of the design of most of these machines is that other machines can be roped in as needed for big jobs. The UCLA physics department home page mentions using other Macs in the department at nights and weekends. If the machines are on the network, they're fair game for becoming part of the supercomputer (as long as you can persuade their regular users). UCLA's Macs even use MacMPI, so that problems can be run on Linux or Mac or a "real" supercomputer without rewriting code.

In fact earlier this year, a group called Distributed.net linked thousands of computers of all kinds around the world via the Internet, and cracked a 56-bit encryption code in 40 days. It had previously been thought that such heavyweight ciphers would take hundreds of years to crack, even on fast computers. One version of the Distributed.net program ran as a screen saver that kicked in, and began cracking code, when the machine was idle for more than a few minutes.

You don't normally think about a screen-saver being a powerful program, but when thousands of them act in concert, some serious computing can get done. Distributed.net bills itself as the "Fastest Computer on Earth", even though its hardware bill is effectively zero.

To be sure, the megabucks hardware offers some advantages that the home- brew machines can't match. For one, the expensive machines tend to be more general-purpose, and have an advantage when huge volumes of data have to be loaded.

Beowulf clusters tend to do best on problems that are easily broken up into independent pieces, such as particle simulations, with each machine representing a particle interacting with its neighbours. Distributed.net can crack ciphers only because its client software just knows code-breaking: running a different task would mean reinstalling tens of thousands of client software for every new job.

Some problems just don't chop up easily, and run better on the "big iron". But there aren't many supercomputers around (not at those prices!) and scientists have to queue for time on them. Massive data sets can take so long to load that only a little computing is done in any one window of opportunity.

In fact, it was just this dilemma that prompted Mike Warren and others to build their own supercomputers. They can apply their machines widely, and to classes of problems that would only rarely get a chance on that expensive supercomputer time.

For example, here at gulker.com, our "single-node supercomputer" is churning furiously, trying to figure out how to come up with another $700.

cg@gulker.com

Arts and Entertainment
Reawakening: can Jon Hamm’s Don Draper find enlightenment in the final ‘Mad Men’?
tv reviewNot quite, but it's an enlightening finale for Don Draper spoiler alert
Arts and Entertainment
Breakfast Show’s Nick Grimshaw

Radio
Have you tried new the Independent Digital Edition apps?
Arts and Entertainment

ebooksNow available in paperback
Arts and Entertainment

ebooks
Arts and Entertainment

Eurovision
Arts and Entertainment
'Youth' cast members Paul Dano, Jane Fonda, Harvey Keitel, Rachel Weisz, and Michael Caine pose for photographers at Cannes Film Festival
film
Arts and Entertainment
Adam West as Batman and Burt Ward and Robin in the 1960s Batman TV show

Comics
Arts and Entertainment
I am flute: Azeem Ward and his now-famous instrument
music
Arts and Entertainment
A glass act: Dr Chris van Tulleken (left) and twin Xand get set for their drinking challenge
TV review
Arts and Entertainment
MIA perform at Lovebox 2014 in London Fields, Hackney

music
Arts and Entertainment
Finnish punk band PKN hope to enter Eurovision 2015 and raise awareness for Down's Syndrome

eurovision
Arts and Entertainment
William Shakespeare on the cover of John Gerard's The Herball or Generall Historie of Plantes

books
Arts and Entertainment

Game of Thrones review
Arts and Entertainment
Grayson Perry dedicates his Essex home to Julie

Potter's attempt to create an Essex Taj Mahal was a lovely treat

tv
Arts and Entertainment
A scene from the original Swedish version of the sci-fi TV drama ‘Real Humans’
tv
Arts and Entertainment
Hugh Keays-Byrne plays Immortan Joe, the terrifying gang leader, in the new film
filmActor who played Toecutter returns - but as a different villain in reboot
Arts and Entertainment
Charlize Theron as Imperator Furiosa in Mad Max: Fury Road
film
Arts and Entertainment
Jessica Hynes in W1A
tvReview: Perhaps the creators of W1A should lay off the copy and paste function spoiler alert
Arts and Entertainment
Power play: Mitsuko Uchida in concert

classical
Arts and Entertainment
Dangerous liaisons: Dominic West, Jake Richard Siciliano, Maura Tierney and Leya Catlett in ‘The Affair’ – a contradictory drama but one which is sure to reel the viewers in
TV review
Arts and Entertainment
Richard Herring, pictured performing at the Edinburgh Fringe Festival two years ago
comedy
Arts and Entertainment
Music freak: Max Runham in the funfair band
theatre
Arts and Entertainment
film 'I felt under-used by Hollywood'
Latest stories from i100
Have you tried new the Independent Digital Edition apps?

ES Rentals

    Independent Dating
    and  

    By clicking 'Search' you
    are agreeing to our
    Terms of Use.

    Abuse - and the hell that came afterwards

    Abuse - and the hell that follows

    James Rhodes on the extraordinary legal battle to publish his memoir
    Why we need a 'tranquility map' of England, according to campaigners

    It's oh so quiet!

    The case for a 'tranquility map' of England
    'Timeless fashion': It may be a paradox, but the industry loves it

    'Timeless fashion'

    It may be a paradox, but the industry loves it
    If the West needs a bridge to the 'moderates' inside Isis, maybe we could have done with Osama bin Laden staying alive after all

    Could have done with Osama bin Laden staying alive?

    Robert Fisk on the Fountainheads of World Evil in 2011 - and 2015
    New exhibition celebrates the evolution of swimwear

    Evolution of swimwear

    From bathing dresses in the twenties to modern bikinis
    Sun, sex and an anthropological study: One British academic's summer of hell in Magaluf

    Sun, sex and an anthropological study

    One academic’s summer of hell in Magaluf
    From Shakespeare to Rising Damp... to Vicious

    Frances de la Tour's 50-year triumph

    'Rising Damp' brought De la Tour such recognition that she could be forgiven if she'd never been able to move on. But at 70, she continues to flourish - and to beguile
    'That Whitsun, I was late getting away...'

    Ian McMillan on the Whitsun Weddings

    This weekend is Whitsun, and while the festival may no longer resonate, Larkin's best-loved poem, lives on - along with the train journey at the heart of it
    Kathryn Williams explores the works and influences of Sylvia Plath in a new light

    Songs from the bell jar

    Kathryn Williams explores the works and influences of Sylvia Plath
    How one man's day in high heels showed him that Cannes must change its 'no flats' policy

    One man's day in high heels

    ...showed him that Cannes must change its 'flats' policy
    Is a quiet crusade to reform executive pay bearing fruit?

    Is a quiet crusade to reform executive pay bearing fruit?

    Dominic Rossi of Fidelity says his pressure on business to control rewards is working. But why aren’t other fund managers helping?
    The King David Hotel gives precious work to Palestinians - unless peace talks are on

    King David Hotel: Palestinians not included

    The King David is special to Jerusalem. Nick Kochan checked in and discovered it has some special arrangements, too
    More people moving from Australia to New Zealand than in the other direction for first time in 24 years

    End of the Aussie brain drain

    More people moving from Australia to New Zealand than in the other direction for first time in 24 years
    Meditation is touted as a cure for mental instability but can it actually be bad for you?

    Can meditation be bad for you?

    Researching a mass murder, Dr Miguel Farias discovered that, far from bringing inner peace, meditation can leave devotees in pieces
    Eurovision 2015: Australians will be cheering on their first-ever entrant this Saturday

    Australia's first-ever Eurovision entrant

    Australia, a nation of kitsch-worshippers, has always loved the Eurovision Song Contest. Maggie Alderson says it'll fit in fine