The DIY guide to supercomputers

EARLIER THIS year I wrote about how I built a computer from the parts bins at Fry's, a local Silicon Valley institution. I installed the Linux operating system on it, and said computer is still chugging away as a Web server, telling the tale of how I put it together (at magellan.gulker.com).

Seems as though my idea caught on.

Mike Warren, an astrophysicist at the Los Alamos National Laboratory in New Mexico, recently put the final touches to Avalon, a do-it-yourself supercomputer that his lab team put together using off-the-shelf, high-end DEC Alpha PCs. Avalon recently became the 88th fastest computer on the planet after some new CPUs were hooked up,

Avalon's price tag - $300,000 - is a bit more than the $700 (pounds 425) I spent, but a fraction of what most supercomputers cost. IBM recently installed "Pacific Blue" at the Lawrence Livermore Laboratory for $94m. True, it's more than 80 times faster than Avalon, but it's also 300 times more expensive.

And, Avalon is not alone. There are lots of home-built supercomputers. Thomas Sterling, a computer scientist at the California Institute of Technology and Nasa, stirred things up by publishing a specification for low-cost clusters of PCs called Beowulf. Avalon is just one Beowulf-class supercomputer.

The idea is simple. You set up an arbitrary number of PCs, network them, typically using fast Ethernet, and then send them problems that can be divided up among the machines' processors. One machine acts as a server that syncs up all its clients. Beowulf specs software like the Message Passing Interface (MPI) written under Linux, that lets the machines communicate. And since Linux - the brainchild of the computer science student Linus Torvalds - is free, it keeps the cost down.

One group, at Australia's University of South Queensland, built Topcat, a Pentium-powered, Beowulf-class computer, for just A$8,000 (pounds 3,137). Other Beowulf machines are called Loki (pre-Avalon), Grendel and Megalon and have various price tags, all modest for their power.

The Linux OS has had a pretty phenomenal year, but an abstract on the Argonne National Laboratory website describes a machine called Chicago Pile 6 that runs under Windows NT, and Microsoft offers the "Wolfpack", clusters of Windows NT machines that can be harnessed together for big problems.

And, to round things out, a group at the UCLA Physics Department's Plasma Simulation Group has a $28,000 parallel computer called Appleseed - eight Power Macintosh G3s - which outperforms a Cray Y-MP. UCLA's statistical mathematics department even plans to link up Apple's new iMacs in their computer lab to make a "very pretty supercomputer".

One advantage of the design of most of these machines is that other machines can be roped in as needed for big jobs. The UCLA physics department home page mentions using other Macs in the department at nights and weekends. If the machines are on the network, they're fair game for becoming part of the supercomputer (as long as you can persuade their regular users). UCLA's Macs even use MacMPI, so that problems can be run on Linux or Mac or a "real" supercomputer without rewriting code.

In fact earlier this year, a group called Distributed.net linked thousands of computers of all kinds around the world via the Internet, and cracked a 56-bit encryption code in 40 days. It had previously been thought that such heavyweight ciphers would take hundreds of years to crack, even on fast computers. One version of the Distributed.net program ran as a screen saver that kicked in, and began cracking code, when the machine was idle for more than a few minutes.

You don't normally think about a screen-saver being a powerful program, but when thousands of them act in concert, some serious computing can get done. Distributed.net bills itself as the "Fastest Computer on Earth", even though its hardware bill is effectively zero.

To be sure, the megabucks hardware offers some advantages that the home- brew machines can't match. For one, the expensive machines tend to be more general-purpose, and have an advantage when huge volumes of data have to be loaded.

Beowulf clusters tend to do best on problems that are easily broken up into independent pieces, such as particle simulations, with each machine representing a particle interacting with its neighbours. Distributed.net can crack ciphers only because its client software just knows code-breaking: running a different task would mean reinstalling tens of thousands of client software for every new job.

Some problems just don't chop up easily, and run better on the "big iron". But there aren't many supercomputers around (not at those prices!) and scientists have to queue for time on them. Massive data sets can take so long to load that only a little computing is done in any one window of opportunity.

In fact, it was just this dilemma that prompted Mike Warren and others to build their own supercomputers. They can apply their machines widely, and to classes of problems that would only rarely get a chance on that expensive supercomputer time.

For example, here at gulker.com, our "single-node supercomputer" is churning furiously, trying to figure out how to come up with another $700.

cg@gulker.com

Have you tried new the Independent Digital Edition apps?
Arts and Entertainment

ebooksNow available in paperback
Arts and Entertainment

ebooks
Arts and Entertainment
Christopher Eccleston (centre) plays an ex-policeman in this cliché-riddled thriller

TV review
Arts and Entertainment
Lena Headey looks very serious as Cersei Lannister in Game of Thrones

TV This TV review contains spoilers
Arts and Entertainment

film
Arts and Entertainment
Wiz Khalifa performs on stage during day one of the Wireless Festival at Perry Park in Birmingham

music
Arts and Entertainment
Festival-goers soak up the atmosphere at Glastonbury

music

Arts and Entertainment
Star Wars creator George Lucas

film

Arts and Entertainment

music

Arts and Entertainment
A shot from the forthcoming Fast and Furious 7

film

Arts and Entertainment
The new-look Top of the Pops could see Fearne Cotton returns as a host alongside Dermot O'Leary

TV

Arts and Entertainment
The leader of the Church of Scientology David Miscavige

TV

Arts and Entertainment
No half measures: ‘The Secret Life of the Pub’

Grace Dent on TV The Secret Life of the Pub is sexist, ageist and a breath of fresh air

Arts and Entertainment
Art on their sleeves: before downloads and streaming, enthusiasts used to flick through racks of albums in their local record shops
musicFor Lois Pryce, working in a record shop was a dream job - until the bean counters ruined it
Arts and Entertainment
Serial suspect: the property heir charged with first-degree murder, Robert Durst
TV review
Arts and Entertainment
Igarashi in her

Art Megumi Igarashi criticises Japan's 'backwards' attitude to women's sexual expression

Arts and Entertainment
Could Ed Sheeran conquer the Seven Kingdoms? He could easily pass for a Greyjoy like Alfie Allen's character (right)

tv Singer could become the most unlikely star of Westeros

Arts and Entertainment
Beyonce, Boris Johnson, Putin, Nigel Farage, Russell Brand and Andy Murray all get the Spitting Image treatment from Newzoids
tvReview: The sketches need to be very short and very sharp as puppets are not intrinsically funny
Arts and Entertainment
Despite the controversy it caused, Mile Cyrus' 'Wrecking Ball' video won multiple awards
musicPoll reveals over 70% of the British public believe sexually explicit music videos should get ratings
Arts and Entertainment
Lena Headey as Cersei Lannister and Ian Beattie as Meryn Trant in the fifth season of Game of Thrones

TV
Arts and Entertainment

book review
Arts and Entertainment
It's all in the genes: John Simm working in tandem with David Threlfall in 'Code of a Killer'

TV review
  • Get to the point
Latest stories from i100
Have you tried new the Independent Digital Edition apps?

ES Rentals

    Independent Dating
    and  

    By clicking 'Search' you
    are agreeing to our
    Terms of Use.

    General Election 2015: Chuka Umunna on the benefits of immigration, humility – and his leader Ed Miliband

    Chuka Umunna: A virus of racism runs through Ukip

    The shadow business secretary on the benefits of immigration, humility – and his leader Ed Miliband
    Yemen crisis: This exotic war will soon become Europe's problem

    Yemen's exotic war will soon affect Europe

    Terrorism and boatloads of desperate migrants will be the outcome of the Saudi air campaign, says Patrick Cockburn
    Marginal Streets project aims to document voters in the run-up to the General Election

    Marginal Streets project documents voters

    Independent photographers Joseph Fox and Orlando Gili are uploading two portraits of constituents to their website for each day of the campaign
    Game of Thrones: Visit the real-life kingdom of Westeros to see where violent history ends and telly tourism begins

    The real-life kingdom of Westeros

    Is there something a little uncomfortable about Game of Thrones shooting in Northern Ireland?
    How to survive a social-media mauling, by the tough women of Twitter

    How to survive a Twitter mauling

    Mary Beard, Caroline Criado-Perez, Louise Mensch, Bunny La Roche and Courtney Barrasford reveal how to trounce the trolls
    Gallipoli centenary: At dawn, the young remember the young who perished in one of the First World War's bloodiest battles

    At dawn, the young remember the young

    A century ago, soldiers of the Empire – many no more than boys – spilt on to Gallipoli’s beaches. On this 100th Anzac Day, there are personal, poetic tributes to their sacrifice
    Dissent is slowly building against the billions spent on presidential campaigns – even among politicians themselves

    Follow the money as never before

    Dissent is slowly building against the billions spent on presidential campaigns – even among politicians themselves, reports Rupert Cornwell
    Samuel West interview: The actor and director on austerity, unionisation, and not mentioning his famous parents

    Samuel West interview

    The actor and director on austerity, unionisation, and not mentioning his famous parents
    General Election 2015: Imagine if the leading political parties were fashion labels

    Imagine if the leading political parties were fashion labels

    Fashion editor, Alexander Fury, on what the leaders' appearances tell us about them
    Phumzile Mlambo-Ngcuka: Home can be the unsafest place for women

    Phumzile Mlambo-Ngcuka: Home can be the unsafest place for women

    The architect of the HeForShe movement and head of UN Women on the world's failure to combat domestic violence
    Public relations as 'art'? Surely not

    Confessions of a former PR man

    The 'art' of public relations is being celebrated by the V&A museum, triggering some happy memories for DJ Taylor
    Bill Granger recipes: Our chef succumbs to his sugar cravings with super-luxurious sweet treats

    Bill Granger's luxurious sweet treats

    Our chef loves to stop for 30 minutes to catch up on the day's gossip, while nibbling on something sweet
    London Marathon 2015: Paula Radcliffe and the mother of all goodbyes

    The mother of all goodbyes

    Paula Radcliffe's farewell to the London Marathon will be a family affair
    Everton vs Manchester United: Steven Naismith demands 'better' if Toffees are to upset the odds against United

    Steven Naismith: 'We know we must do better'

    The Everton forward explains the reasons behind club's decline this season
    Arsenal vs Chelsea: Praise to Arsene Wenger for having the courage of his convictions

    Michael Calvin's Last Word

    Praise to Wenger for having the courage of his convictions