Fast and easy way to cut a long story short

Summarising a text is a key human skill. But can software do it? Roderick Neil Kay looks at progress

Automatic summarising has long been considered one of the most prized goals in artificial intelligence, or AI. Working summarisers have now finally appeared, although they are based on far more superficial techniques.

The ability to summarise a text is founded on core intellectual skills, so much so that the trials in the task still appear in numerous IQ and recruitment tests, including those run for the elite of the Civil Service.

Like many other products involving natural language, the new summarising technology doesn't quite live up to its billing. Summarisers have now been developed by a host of companies, including Oracle, InXight and BT, all based on similar statistical techniques. The simplest such method runs as follows. First, find the most frequently occurring words in a text, excluding trivial words. Second, locate the sentences in which groups of these words occur. Third, extract these sentences and compile them in chronological order. Then describe the compilation as a summary, without blinking.

One of the most surprising things about the statistical techniques at the centre of the new software is that the ideas have been around for a long time, almost before the dawn of AI, in fact. As early as 1958, while working for IBM, Luhn ran some experiments on a corpus of technical articles, using the algorithm just described. He was enthusiastic about the results: "The auto-abstract is perhaps the first example of a machine- generated equivalent of a completely intellectual task in the field of literature evaluation."

But at the time, text retrieval wasn't the hot topic it is today, and his idea was never marketed in the form of software. The view within the AI community, which has always aimed at getting computers to understand language, has been that statistical techniques are OK as far they go, but if you consider what a human can do, that isn't very far. The emergence at this point of the new summarisers probably owes as much to bumped-up demand as it does to advancement in the field.

While the recent crop of summarisers fall reassuringly short of human performance, their wide availability should generate the kind of interest which leads to improvement. And anyway, enough of human condescension; let us allow the computer to speak - or rather to summarise - for itself. The following extract is a 20 per cent summary of this article, produced by BT's Netsumm.

"Automatic summarising has long been considered one of the most prized goals in artificial intelligence, but working summarisers have now finally appeared based on far more superficial techniques. Summarisers have now been developed by a host of companies not normally associated with text processing: Oracle, InXight and British Telecom, all based on similar statistical heuristics. The view within the AI community, which has always aimed at getting computers to understand language, has been that statistical techniques are OK as far they go, but if you consider what a human can do, it isn't very far"n

News
people'It can last and it's terrifying'
Sport
Danny Welbeck's Manchester United future is in doubt
footballGunners confirm signing from Manchester United
Sport
footballStriker has moved on loan for the remainder of the season
Sport
footballFeaturing Bart Simpson
PROMOTED VIDEO
Have you tried new the Independent Digital Edition apps?
Life and Style
ebooksA superb mix of recipes serving up the freshest of local produce in a delicious range of styles
Life and Style
ebooksFrom the lifespan of a slug to the distance to the Sun: answers to 500 questions from readers
New Articles
Olivia Colman topped the list of the 30 most influential females in broadcasting
tv
News
Kelly Brook
peopleA spokesperson said the support group was 'extremely disappointed'
News
The five geckos were launched into space to find out about the effects of weightlessness on the creatures’ sex lives
i100
Life and Style
techIf those brochure kitchens look a little too perfect to be true, well, that’s probably because they are
Sport
Andy Murray celebrates a shot while playing Jo-Wilfried Tsonga
TennisWin sets up blockbuster US Open quarter-final against Djokovic
Arts and Entertainment
Hare’s a riddle: Kit Williams with the treasure linked to Masquerade
booksRiddling trilogy could net you $3m
Arts and Entertainment
Alex Kapranos of Franz Ferdinand performs live
music Pro-independence show to take place four days before vote
News
news Video - hailed as 'most original' since Benedict Cumberbatch's
News
i100
Life and Style
The longer David Sedaris had his Fitbit, the further afield his walks took him through the West Sussex countryside
lifeDavid Sedaris: What I learnt from my fitness tracker about the world
Arts and Entertainment
Word master: Self holds up a copy of his novel ‘Umbrella’
booksUnlike 'talented mediocrity' George Orwell, you must approach this writer dictionary in hand
News
i100
Latest stories from i100
Have you tried new the Independent Digital Edition apps?
Independent Dating
and  

By clicking 'Search' you
are agreeing to our
Terms of Use.

ES Rentals

    iJobs Job Widget
    iJobs General

    SQL Implementation Consultant (VB,C#, SQL, Java, Eclipse, integ

    £40000 - £50000 per annum + benefits+bonus+package: Harrington Starr: SQL Impl...

    SQL Technical Implementation Consultant (Java, BA, Oracle, VBA)

    £45000 - £55000 per annum + Benefits + Bonus: Harrington Starr: SQL Technical ...

    Head of IT (Windows, Server, VMware, SAN, Fidessa, Equities)

    £85000 per annum: Harrington Starr: Head of IT (Windows, Server, VMware, SAN, ...

    Lead C# Developer (.Net, nHibernate, MVC, SQL) Surrey

    £55000 - £60000 per annum + Benefits + Bonus: Harrington Starr: Lead C# Develo...

    Day In a Page

    'I’ll tell you what I would not serve - lamb and potatoes': US ambassador hits out at stodgy British food served at diplomatic dinners

    'I’ll tell you what I would not serve - lamb and potatoes'

    US ambassador hits out at stodgy British food
    Radio Times female powerlist: A 'revolution' in TV gender roles

    A 'revolution' in TV gender roles

    Inside the Radio Times female powerlist
    Endgame: James Frey's literary treasure hunt

    James Frey's literary treasure hunt

    Riddling trilogy could net you $3m
    Fitbit: Because the tingle feels so good

    Fitbit: Because the tingle feels so good

    What David Sedaris learnt about the world from his fitness tracker
    Saudis risk new Muslim division with proposal to move Mohamed’s tomb

    Saudis risk new Muslim division with proposal to move Mohamed’s tomb

    Second-holiest site in Islam attracts millions of pilgrims each year
    Alexander Fury: The designer names to look for at fashion week this season

    The big names to look for this fashion week

    This week, designers begin to show their spring 2015 collections in New York
    Will Self: 'I like Orwell's writing as much as the next talented mediocrity'

    'I like Orwell's writing as much as the next talented mediocrity'

    Will Self takes aim at Orwell's rules for writing plain English
    Meet Afghanistan's middle-class paint-ballers

    Meet Afghanistan's middle-class paint-ballers

    Toy guns proving a popular diversion in a country flooded with the real thing
    Al Pacino wows Venice

    Al Pacino wows Venice

    Ham among the brilliance as actor premieres two films at festival
    Neil Lawson Baker interview: ‘I’ve gained so much from art. It’s only right to give something back’.

    Neil Lawson Baker interview

    ‘I’ve gained so much from art. It’s only right to give something back’.
    The other Mugabe who is lining up for the Zimbabwean presidency

    The other Mugabe who is lining up for the Zimbabwean presidency

    Wife of President Robert Mugabe appears to have her sights set on succeeding her husband
    The model of a gadget launch: Cultivate an atmosphere of mystery and excitement to sell stuff people didn't realise they needed

    The model for a gadget launch

    Cultivate an atmosphere of mystery and excitement to sell stuff people didn't realise they needed
    Alice Roberts: She's done pretty well, for a boffin without a beard

    She's done pretty well, for a boffin without a beard

    Alice Roberts talks about her new book on evolution - and why her early TV work drew flak from (mostly male) colleagues
    Get well soon, Joan Rivers - an inspiration, whether she likes it or not

    Get well soon, Joan Rivers

    She is awful. But she's also wonderful, not in spite of but because of the fact she's forever saying appalling things, argues Ellen E Jones
    Doctor Who Into the Dalek review: A classic sci-fi adventure with all the spectacle of a blockbuster

    A fresh take on an old foe

    Doctor Who Into the Dalek more than compensated for last week's nonsensical offering