Parts of the internet are disappearing around us – and they might be lost forever

We see the internet as storing vast swathes of information forever – but parts of it are under more threat than we realise, writes Andrew Griffin

Billions of sites have already disappeared from the web (Reuters)

The Internet Archive – the most capacious library ever made – is home to 835 billion webpages. A single backup of its library collection requires more than 145 petabytes of space. By comparison, the world’s largest physical library is the US Library of Congress, and is home to about 175 million items, according to Guinness World Records.

And yet even as the world generates more data than ever, much of it is falling away. Some 38 per cent of webpages that existed in 2013 are no longer there, according to new research from the Pew Research Center, and even 8 per cent of those that existed in 2023 are now gone.

Increasingly, the web is made up of content that is being both produced and consumed by automated systems: a report last month from cybersecurity company Imperva said that almost exactly half of all internet traffic came from bots.