The DIY guide to supercomputers

EARLIER THIS year I wrote about how I built a computer from the parts bins at Fry's, a local Silicon Valley institution. I installed the Linux operating system on it, and said computer is still chugging away as a Web server, telling the tale of how I put it together (at magellan.gulker.com).

Seems as though my idea caught on.

Mike Warren, an astrophysicist at the Los Alamos National Laboratory in New Mexico, recently put the final touches to Avalon, a do-it-yourself supercomputer that his lab team put together using off-the-shelf, high-end DEC Alpha PCs. Avalon recently became the 88th fastest computer on the planet after some new CPUs were hooked up,

Avalon's price tag - $300,000 - is a bit more than the $700 (pounds 425) I spent, but a fraction of what most supercomputers cost. IBM recently installed "Pacific Blue" at the Lawrence Livermore Laboratory for $94m. True, it's more than 80 times faster than Avalon, but it's also 300 times more expensive.

And, Avalon is not alone. There are lots of home-built supercomputers. Thomas Sterling, a computer scientist at the California Institute of Technology and Nasa, stirred things up by publishing a specification for low-cost clusters of PCs called Beowulf. Avalon is just one Beowulf-class supercomputer.

The idea is simple. You set up an arbitrary number of PCs, network them, typically using fast Ethernet, and then send them problems that can be divided up among the machines' processors. One machine acts as a server that syncs up all its clients. Beowulf specs software like the Message Passing Interface (MPI) written under Linux, that lets the machines communicate. And since Linux - the brainchild of the computer science student Linus Torvalds - is free, it keeps the cost down.

One group, at Australia's University of South Queensland, built Topcat, a Pentium-powered, Beowulf-class computer, for just A$8,000 (pounds 3,137). Other Beowulf machines are called Loki (pre-Avalon), Grendel and Megalon and have various price tags, all modest for their power.

The Linux OS has had a pretty phenomenal year, but an abstract on the Argonne National Laboratory website describes a machine called Chicago Pile 6 that runs under Windows NT, and Microsoft offers the "Wolfpack", clusters of Windows NT machines that can be harnessed together for big problems.

And, to round things out, a group at the UCLA Physics Department's Plasma Simulation Group has a $28,000 parallel computer called Appleseed - eight Power Macintosh G3s - which outperforms a Cray Y-MP. UCLA's statistical mathematics department even plans to link up Apple's new iMacs in their computer lab to make a "very pretty supercomputer".

One advantage of the design of most of these machines is that other machines can be roped in as needed for big jobs. The UCLA physics department home page mentions using other Macs in the department at nights and weekends. If the machines are on the network, they're fair game for becoming part of the supercomputer (as long as you can persuade their regular users). UCLA's Macs even use MacMPI, so that problems can be run on Linux or Mac or a "real" supercomputer without rewriting code.

In fact earlier this year, a group called Distributed.net linked thousands of computers of all kinds around the world via the Internet, and cracked a 56-bit encryption code in 40 days. It had previously been thought that such heavyweight ciphers would take hundreds of years to crack, even on fast computers. One version of the Distributed.net program ran as a screen saver that kicked in, and began cracking code, when the machine was idle for more than a few minutes.

You don't normally think about a screen-saver being a powerful program, but when thousands of them act in concert, some serious computing can get done. Distributed.net bills itself as the "Fastest Computer on Earth", even though its hardware bill is effectively zero.

To be sure, the megabucks hardware offers some advantages that the home- brew machines can't match. For one, the expensive machines tend to be more general-purpose, and have an advantage when huge volumes of data have to be loaded.

Beowulf clusters tend to do best on problems that are easily broken up into independent pieces, such as particle simulations, with each machine representing a particle interacting with its neighbours. Distributed.net can crack ciphers only because its client software just knows code-breaking: running a different task would mean reinstalling tens of thousands of client software for every new job.

Some problems just don't chop up easily, and run better on the "big iron". But there aren't many supercomputers around (not at those prices!) and scientists have to queue for time on them. Massive data sets can take so long to load that only a little computing is done in any one window of opportunity.

In fact, it was just this dilemma that prompted Mike Warren and others to build their own supercomputers. They can apply their machines widely, and to classes of problems that would only rarely get a chance on that expensive supercomputer time.

For example, here at gulker.com, our "single-node supercomputer" is churning furiously, trying to figure out how to come up with another $700.

cg@gulker.com

Arts and Entertainment
Former Communards frontman Jimmy Somerville
music
Arts and Entertainment
Secrets of JK Rowling's Harry Potter workings have been revealed in a new bibliography
arts + ents
Arts and Entertainment
Fearne Cotton is leaving Radio 1 after a decade
radio The popular DJ is leaving for 'family and new adventures'
Arts and Entertainment
arts + ents
Arts and Entertainment
Public Service Broadcasting are going it alone
music
Arts and Entertainment

Oscars 2015 Mexican filmmaker uses speech to urge 'respect' for immigrants

Arts and Entertainment
The Oscar nominations are due to be announced today

Oscars 2015 Bringing you all the news from the 87th Academy Awards

Have you tried new the Independent Digital Edition apps?
Arts and Entertainment

ebooksNow available in paperback
Arts and Entertainment

ebooks
Arts and Entertainment
Lloyd-Hughes takes the leading role as Ralph Whelan in Channel 4's epic new 10-part drama, Indian Summers

TV Review

The intrigue deepens as we delve further but don't expect any answers just yet
Arts and Entertainment
Jason Segal and Cameron Diaz star in Sex Tape

Razzies 2015 Golden Raspberry Awards 'honours' Cameron Diaz and Kirk Cameron

Arts and Entertainment
The Oscars ceremony 2015 will take place at the Dolby Theatre in Los Angeles
Oscars 2015A quiz to whet your appetite for tonight’s 87th Academy Awards
Arts and Entertainment
Sigourney Weaver, as Ripley, in Alien; critics have branded the naming of action movie network Movies4Men as “offensive” and “demographic box-ticking gone mad”.
TVNaming of action movie network Movies4Men sparks outrage
Arts and Entertainment
Sleater Kinney perform at the 6 Music Festival at the O2 Academy, Newcastle
musicReview: 6 Music Festival
Arts and Entertainment
Sleater Kinney perform at the 6 Music Festival at the O2 Academy, Newcastle
musicReview: 6 Music Festival
News
Kristen Stewart reacts after receiving the Best Actress in a Supporting Role award for her role in 'Sils Maria' at the 40th annual Cesar awards
people
News
A lost Sherlock Holmes story has been unearthed
arts + ents Walter Elliot, an 80-year-old historian, found it in his attic,
Arts and Entertainment
Margot Robbie rose to fame starring alongside Leonardo DiCaprio in The Wolf of Wall Street

Film Hollywood's new leading lady talks about her Ramsay Street days

Arts and Entertainment
Right note: Sam Haywood with Simon Usborne page turning
musicSimon Usborne discovers it is under threat from the accursed iPad
Arts and Entertainment
A life-size sculpture by Nick Reynolds depicting singer Pete Doherty on a crucifix hangs in St Marylebone church
arts + ents
Arts and Entertainment
Escalating tension: Tang Wei and Chris Hemsworth in ‘Blackhat’
filmReview: Chris Hemsworth stars as a convicted hacker in Blackhat
Arts and Entertainment

Oscar voter speaks out

film
Arts and Entertainment
The Oscars race for Best Picture will be the battle between Boyhood and Birdman

Oscars
Arts and Entertainment
Anne Boleyn (Claire Foy), Thomas Cromwell (Mark Rylance)
tvReview: Wolf Hall
Arts and Entertainment
Tom Meighan of Kasabian collects the Best Album Award
music
Arts and Entertainment
Best supporting stylist: the late L’Wren Scott dressed Nicole Kidman in 1997
film
Arts and Entertainment
Dakota Johnson and Jamie Dornan as Anastasia Steele and Christian Grey in Fifty Shades of Grey

Film

Arts and Entertainment
Mick Carter (Danny Dyer) and Peggy Mitchell (Barbara Windsor)
tv occurred in the crucial final scene
Arts and Entertainment
Glasgow wanted to demolish its Red Road flats last year
architecture
Latest stories from i100
Have you tried new the Independent Digital Edition apps?

ES Rentals

    Independent Dating
    and  

    By clicking 'Search' you
    are agreeing to our
    Terms of Use.

    The Last Word: For the good of the game: why on earth don’t we leave Fifa?

    Michael Calvin's Last Word

    For the good of the game: why on earth don’t we leave Fifa?
    HIV pill: Scientists hail discovery of 'game-changer' that cuts the risk of infection among gay men by 86%

    Scientists hail daily pill that protects against HIV infection

    Breakthrough in battle against global scourge – but will the NHS pay for it?
    How we must adjust our lifestyles to nature: Welcome to the 'Anthropocene', the human epoch

    Time to play God

    Welcome to the 'Anthropocene', the human epoch where we may need to redefine nature itself
    MacGyver returns, but with a difference: Handyman hero of classic 1980s TV series to be recast as a woman

    MacGyver returns, but with a difference

    Handyman hero of classic 1980s TV series to be recast as a woman
    Tunnel renaissance: Why cities are hiding roads down in the ground

    Tunnel renaissance

    Why cities are hiding roads underground
    'Backstreet Boys - Show 'Em What You're Made Of': An affectionate look at five middle-aged men

    Boys to men

    The Backstreet Boys might be middle-aged, married and have dodgy knees, but a heartfelt documentary reveals they’re not going gently into pop’s good night
    Crufts 2015: Should foreign dogs be allowed to compete?

    Crufts 2015

    Should foreign dogs be allowed to compete?
    10 best projectors

    How to make your home cinema more cinematic: 10 best projectors

    Want to recreate the big-screen experience in your sitting room? IndyBest sizes up gadgets to form your film-watching
    Manchester City 1 Barcelona 2 player ratings: Luis Suarez? Lionel Messi? Joe Hart? Who was the star man?

    Manchester City vs Barcelona player ratings

    Luis Suarez? Lionel Messi? Joe Hart? Who was the star man at the Etihad?
    Arsenal vs Monaco: Monaco - the making of Gunners' manager Arsene Wenger

    Monaco: the making of Wenger

    Jack Pitt-Brooke speaks to former players and learns the Frenchman’s man-management has always been one of his best skills
    Cricket World Cup 2015: Chris Gayle - the West Indies' enigma lives up to his reputation

    Chris Gayle: The West Indies' enigma

    Some said the game's eternal rebel was washed up. As ever, he proved he writes the scripts by producing a blistering World Cup innings
    In Ukraine a dark world of hybrid warfare and murky loyalties prevails

    In Ukraine a dark world of hybrid warfare

    This war in the shadows has been going on since the fall of Mr Yanukovych
    'Birdman' and 'Bullets Over Broadway': Homage or plagiarism?

    Homage or plagiarism?

    'Birdman' shares much DNA with Woody Allen's 'Bullets Over Broadway'
    Broadchurch ends as damp squib not even David Tennant can revive

    A damp squib not even David Tennant can revive

    Broadchurch, Series 2 finale, review
    A Koi carp breeding pond, wall-mounted iPads and a bathroom with a 'wellness' shower: inside the mansion of Germany's 'Bishop of Bling'

    Inside the mansion of Germany's 'Bishop of Bling'

    A Koi carp breeding pond, wall-mounted iPads and a bathroom with a 'wellness' shower