Ministers want to farm out a Big Brother database of everyone's emails, phone calls and internet use to private companies who will be given the job of storing the data on behalf of the state.
The £2bn cost of the plans could add millions of pounds to phone and internet bills to help pay for new systems to collect and sort private information.
Jacqui Smith, the Home Secretary, said the Government had rejected the idea of a centralised database because it would impinge on privacy. She favoured a "middle way" in which primary communication companies, such as BT or Virgin, and leading internet service providers would have the job of collating phone, email and web use.
The Home Office wants communications companies to extend the range of information they currently hold on subscribers and organise it so that it can be better used by the police, MI5 and other public bodies investigating serious crime and terrorism.
Ministers estimate that the project will cost £2bn to set up, which includes some compensation to the communications industry for the work it may be asked to do.
"Communications data is an essential tool for law enforcement agencies to track murderers, paedophiles, save lives and tackle crime," Ms Smith said.
"It is essential that the police and other crime-fighting agencies have the tools they need to do their job. However, to be clear, there are absolutely no plans for a single central store."
The primary service providers would have the additional responsibility of collecting information from internet and other communications services which cross their networks.
Under the proposals an individual or household will be given a user ID so the company would be able to organise all the data linked to that ID.
Ms Smith said the Government had no interest in content of emails or phone calls and was only interested in logs showing who was communicating with whom.
The Information Commissioner, Richard Thomas, welcomed the decision not to go ahead with a giant centralised database, but called on the Government to publish details about how ministers intended to protect privacy. He said: "You can tell an awful lot about some people's personal circumstances from the people they are talking to and the websites they visit. It is important that the proposals are tightly defined and minimise the level of intrusion with appropriate safeguards in place."
Shami Chakrabarti, director of Liberty, described the decision as a "Home Office climb-down" on a super Big Brother database. "It is a clear signal that the public interest in personal privacy can no longer be ignored, she said. "However, if companies are to be required to hold even more information than they do at present, concerns about access and use become even more important."
Communications service providers already hold large amounts of communications data and an EU directive that came into force this month requires data to be retained for a year.
The Internet Services Providers' Association (ISPA) welcomed the government consultation. Nicholas Lansman, ISPA secretary general, said: "To ensure that any updated law enforcement requirements do not place extra financial burdens on internet service providers, ISPA stresses the importance of cost recovery."
Big Brother: How you're being watched
*Emails: Under European Union rules, communication service providers have to keep details of all email and other internet traffic for 12 months. Ministers now want them to also store details of other internet services that cross their networks.
*Telephone calls: These should be easier to trace and store as the primary communications company will route internet use and emails through a telephone broadband line. Mobile phone companies tend to retain this information for billing purposes.
*Web browsing: An individual's internet history can help law enforcement and security services paint a picture of suspicious online behaviour. There will be no right to see the content of messages left on websites.
*Social networking sites: Ministers have said they want to monitor sites such as Facebook and Bebo. Intelligence gleaned by law enforcement agencies suggests suspects believe they can disguise their identities by leaving messages for co-conspirators on these private access sites. Named public bodies and the police will only be able to see this information if they have made a case under the Regulation of Investigatory Powers Act.
*Internet telephonic use: Web-phone call software like Skype has become popular because it cuts the cost of making telephone calls. It is the kind of rapid development in new technology which law enforcement agencies want to be able to monitor. More and more telephone calls are being routed over the web, meaning that police are losing the ability to track who has called whom, from where and for how long.
Join our commenting forum
Join thought-provoking conversations, follow other Independent readers and see their replies