The key to spam-free inboxes

Efforts to cut junk e-mail aren't working. Will a 'sender authentication' scheme halt the flood?

At the start of this year I forecast that spam would get worse before it got better, despite (or perhaps because of) the United States's new "CAN-SPAM" Act, intended to put the fear of fines and jail into spammers, but which seems to have passed them by completely.

According to Postini (www.postini.com), 84 per cent of e-mail is spam. According to Brightmail, last month the figure was 62 per cent. That's a big range, but neither thinks spam is going down: Brightmail's numbers (at www.brightmail.com/spamstats.html) show a trend that, if anything, has accelerated in the past few months.

But now there's a chance that things will get better, after an initiative from Sendmail (whose package of the same name runs a lot of the internet's mail servers) and industry giants including Microsoft and Yahoo!. It's by no means a complete solution, but it is a key step.

The companies have said that this month they will start testing a system called "Domain Keys", which will in effect make sure that when messages claim to be from hotmail.com or yahoo.com, they really are. It will do so by a cryptographic process: as each message leaves Hotmail's or Yahoo!'s Sendmail-driven servers, it will have an extra header added, calculated using a "private key" (a large prime number) secret to the company.

When the e-mail arrives at a receiver's mail server, it can extract the header and use Hotmail's or Yahoo!'s "public key" (a very large number produced by multiplying the private key with another large prime) to calculate whether the apparent mail server really did send it. If not, it could be a forgery, and so can be rejected.

The desired end-point is that every internet domain will have its own Domain Key, and one can know for certain whether any e-mail claiming to come from there really does, and accept or reject it on that basis. (And, of course, on the basis of whether you want mail from that domain, which can be a more important consideration.) That could have drawbacks, as we'll see, but it could have a huge effect.

However, that effect won't be to stop spam, says Steve Linford of Spamhaus, a commercial organisation that provides spam-blocking services. "It won't have an effect on the volume of spam, because it isn't designed to reduce spam. But it will reduce the amount of spam claiming to come from Hotmail or Yahoo!, because it will reduce the ability of spammers to claim fraudulently that they're sending a message from them." So many e-mails now sent out make that claim - or otherwise fake their "From:" field (a technique charmingly called "joe-jobbing") - that the average internet user thinks that the two giants of the free e-mail world are the source of much of its spam.

This isn't so: while it used to be easy for spammers to set up dozens of accounts an hour on those systems, new authentication methods mean each account must be created by a human.

So the spammers have given up on Hotmail and Yahoo! and moved to other mail servers - often, for the large commercial spammers, based in China or Asia-Pacific. (This is why, if you're operating a spam-filtering system, it makes sense to weigh its filters so that messages coming from domains registered with the Asia-Pacific Network Information Centre, Apnic, are suspected as being spam.) But, for the moment, they haven't given up pretending to be "from" Yahoo! or Hotmail.

Similar "authentication" techniques are planned for e-mail. They go under various names - Domain Keys and SPF ("Sender Permitted From") being the principal ones. If their use becomes more widespread, there could be some unexpected results. One could imagine that some large companies might begin to refuse to receive mail from any domain that did not provide a Domain Keys or SPF look-up. That would have a serious effect on smaller businesses and internet service providers unable to implement the changes themselves.

There would also be problems for travellers using laptops to send e-mail, and those who run their own domains. I sometimes send e-mails using my own address, but it's not likely I'll get SPF any time soon. Similarly, roaming users can install their own mail servers (it's only software) on their laptops; but if they sent them from those machines to an address hedged round by SPF or Domain Keys, they'd fail the tests because they wouldn't have the private key needed to generate the correct header. (Providing it would be a huge security risk for their company.)

The result? Their missives to "protected" servers would disappear into the bit bucket, their absence unnoted until someone needed them urgently. We would discover that the barriers of SPF and Domain Keys had, in effect, created a class structure within internet e-mail.

What else, then, might stem the tide of junk? "It needs a change in the SMTP [Simple Mail Transfer Protocol] system used to send e-mail on the net," Linford says. "The problem is that it was invented by scientists on a network who all trusted each other. They never imagined it would be out in public being used by people who would tell lies." SMTP trusts you to tell the truth about who you are. Spam doesn't.

Changing SMTP isn't a trivial matter. It would be like changing the UK's TV broadcasting from the European PAL system to the US's NTSC - and doing it overnight. "It will take about two years before people realise it, but changing SMTP is the only way we're going to solve spam," Linford says.

Until then, keep watching those Brightmail and Postini stats, and remember: every e-mail you send helps to stop spam taking over completely.

network@independent.co.uk