Fast and easy way to cut a long story short

Summarising a text is a key human skill. But can software do it? Roderick Neil Kay looks at progress

Automatic summarising has long been considered one of the most prized goals in artificial intelligence, or AI. Working summarisers have now finally appeared, although they are based on far more superficial techniques.

The ability to summarise a text is founded on core intellectual skills, so much so that the trials in the task still appear in numerous IQ and recruitment tests, including those run for the elite of the Civil Service.

Like many other products involving natural language, the new summarising technology doesn't quite live up to its billing. Summarisers have now been developed by a host of companies, including Oracle, InXight and BT, all based on similar statistical techniques. The simplest such method runs as follows. First, find the most frequently occurring words in a text, excluding trivial words. Second, locate the sentences in which groups of these words occur. Third, extract these sentences and compile them in chronological order. Then describe the compilation as a summary, without blinking.

One of the most surprising things about the statistical techniques at the centre of the new software is that the ideas have been around for a long time, almost before the dawn of AI, in fact. As early as 1958, while working for IBM, Luhn ran some experiments on a corpus of technical articles, using the algorithm just described. He was enthusiastic about the results: "The auto-abstract is perhaps the first example of a machine- generated equivalent of a completely intellectual task in the field of literature evaluation."

But at the time, text retrieval wasn't the hot topic it is today, and his idea was never marketed in the form of software. The view within the AI community, which has always aimed at getting computers to understand language, has been that statistical techniques are OK as far they go, but if you consider what a human can do, that isn't very far. The emergence at this point of the new summarisers probably owes as much to bumped-up demand as it does to advancement in the field.

While the recent crop of summarisers fall reassuringly short of human performance, their wide availability should generate the kind of interest which leads to improvement. And anyway, enough of human condescension; let us allow the computer to speak - or rather to summarise - for itself. The following extract is a 20 per cent summary of this article, produced by BT's Netsumm.

"Automatic summarising has long been considered one of the most prized goals in artificial intelligence, but working summarisers have now finally appeared based on far more superficial techniques. Summarisers have now been developed by a host of companies not normally associated with text processing: Oracle, InXight and British Telecom, all based on similar statistical heuristics. The view within the AI community, which has always aimed at getting computers to understand language, has been that statistical techniques are OK as far they go, but if you consider what a human can do, it isn't very far"n

PROMOTED VIDEO
Have you tried new the Independent Digital Edition apps?
Life and Style
ebookNow available in paperback
ebooks
ebookPart of The Independent’s new eBook series The Great Composers
Arts and Entertainment
Peter Mayhew as Chewbacca alongside Harrison Ford's Han Solo in 'Star Wars'
film
News
i100
Arts and Entertainment
British actor Idris Elba is also a DJ and rapper who played Ibiza last summer
filmIdris Elba responds to James Bond rumours on Twitter
News
Hackers revealed Oscar-winning actress Lawrence was paid less than her male co-stars in American Hustle
people
Arts and Entertainment
Ellie Levenson’s The Election book demystifies politics for children
bookNew children's book primes the next generation for politics
News
Outspoken: Alexander Fury, John Rentoul, Ellen E Jones and Katy Guest
newsFrom the Scottish referendum to the Ice Bucket Challenge, our writers voice their opinions
Sport
Adnan Januzaj and Gareth Bale
footballManchester United set to loan out Januzaj to make room for Bale - if a move for the Welshman firms up
Arts and Entertainment
Lena Headey as Cersei Lannister in Game of Thrones
film
News
i100
Sport
Yaya Sanogo, Mats Hummels, Troy Deeney and Adnan Januzaj
footballMost Premier League sides are after a striker, but here's a full run down of the ins and outs that could happen over the next month
News
Nigel Farage celebrates with a pint after early local election results in the Hoy and Helmet pub in South Benfleet in Essex
peopleHe has shaped British politics 'for good or ill'
Latest stories from i100
Have you tried new the Independent Digital Edition apps?
Independent Dating
and  

By clicking 'Search' you
are agreeing to our
Terms of Use.

ES Rentals

    iJobs Job Widget
    iJobs General

    Recruitment Genius: Panel & Cabinet Wireman

    £20000 per annum: Recruitment Genius: Panel Wireman required for small electro...

    Recruitment Genius: Electronics Test Engineer

    £25000 - £27000 per annum: Recruitment Genius: An SME based in East Cheshire, ...

    Recruitment Genius: Marketing Assistant

    £18000 - £22000 per annum: Recruitment Genius: Do you have previous experience...

    Recruitment Genius: Accounts Administrator

    £16000 - £18000 per annum: Recruitment Genius: This is an exciting opportunity...

    Day In a Page

    War with Isis: The West needs more than a White Knight

    The West needs more than a White Knight

    Despite billions spent on weapons, the US has not been able to counter Isis's gruesome tactics, says Patrick Cockburn
    Return to Helmand: Private Davey Graham recalls the day he was shot by the Taliban

    'The day I was shot by the Taliban'

    Private Davey Graham was shot five times during an ambush in 2007 - it was the first, controversial photograph to show the dangers our soldiers faced in Helmand province
    Revealed: the best and worst airlines for delays

    Revealed: the best and worst airlines for delays

    Many flyers are failing to claim compensation to which they are entitled, a new survey has found
    The stories that defined 2014: From the Scottish independence referendum to the Ice Bucket Challenge, our writers voice their opinions

    The stories that defined 2014

    From the Scottish independence referendum to the Ice Bucket Challenge, our writers voice their opinions
    Stoke-on-Trent becomes first British city to be classified as 'disaster resilient' by the United Nations

    Disaster looming? Now you know where to head...

    Which British city has become the first to be awarded special 'resilience' status by the UN?
    Finally, a diet that works: Californian pastor's wildly popular Daniel Plan has seen his congregation greatly reduced

    Finally, a diet that works

    Californian pastor's wildly popular Daniel Plan has seen his congregation greatly reduced
    Say it with... lyrics: The power of song was never greater, according to our internet searches

    Say it with... lyrics

    The power of song was never greater, according to our internet searches
    Professor Danielle George: On a mission to bring back the art of 'thinkering'

    The joys of 'thinkering'

    Professor Danielle George on why we have to nurture tomorrow's scientists today
    Monique Roffey: The author on father figures, the nation's narcissism and New Year reflections

    Monique Roffey interview

    The author on father figures, the nation's narcissism and New Year reflections
    Introducing my anti-heroes of 2014

    Introducing my anti-heroes of 2014

    Their outrageousness and originality makes the world a bit more interesting, says Ellen E Jones
    DJ Taylor: Good taste? It's all a matter of timing...

    Good taste? It's all a matter of timing...

    It has been hard to form generally accepted cultural standards since the middle of the 19th century – and the disintegration is only going to accelerate, says DJ Taylor
    Olivia Jacobs & Ben Caplan: 'Ben thought the play was called 'Christian Love'. It was 'Christie in Love' - about a necrophiliac serial killer'

    How we met

    Olivia Jacobs and Ben Caplan
    Bill Granger recipes: Our chef's breakfasts will revitalise you in time for the New Year

    Bill Granger's healthy breakfasts

    Our chef's healthy recipes are perfect if you've overindulged during the festive season
    Transfer guide: From Arsenal to West Ham - what does your club need in the January transfer window?

    Who does your club need in the transfer window?

    Most Premier League sides are after a striker, but here's a full run down of the ins and outs that could happen over the next month
    The Last Word: From aliens at FA to yak’s milk in the Tour, here’s to 2015

    Michael Calvin's Last Word

    From aliens at FA to yak’s milk in the Tour, here’s to 2015