The basic picture is simple enough. We have known for 70 years, since the work of Edwin Hubble, that the Universe is expanding (galaxies such as our Milky Way are moving apart as the space between them stretches). This implies that there was a beginning, a little less than 15 billion years ago, from which all the matter, energy and space in the Universe emerged in a Big Bang. We know how long it has been since the Big Bang because we can measure how quickly galaxies are moving apart, and so how long it has taken them to get to where they are now.
But we should also like to know whether the expansion will continue for ever, or will one day be brought to a halt by the gravity of everything in the Universe pulling on everything else in the Universe. Think of the Big Bang as giving a great initial push to the Universe, which then coasts along as gravity tries to pull everything together - it's a bit like a cricket ball hit upwards, coasting along after the impact of the bat, slowing down even though it is still rising. If hit hard enough, it would escape the Earth's gravity. But will the Universe escape from its own gravity? This is where the new discoveries are completing the puzzle.
All the bright stars in all the bright galaxies are nowhere near enough to stop the expansion. A galaxy like the Milky Way contains a few hundred billion stars, and there are a few hundred billion galaxies within range of our telescopes. But all this visible matter adds up to maybe 1 per cent of the total needed (within that volume of space) just to halt the universal expansion (this total is sometimes known as the critical density). From the way galaxies move in groups, and the way they are distributed across the sky, astronomers can also detect the gravitational influence of a vast amount of unseen dark matter. This adds up to about a third of the critical density, implying that the bright stuff is only one-thirtieth of the Universe.
No problem, you might think. The implication is that the Universe was born in a Big Bang, where it was given a great outward push, and will expand for ever, even though the expansion will slow down gradually as gravity does its work. But there is a problem. In the early Eighties, theorists came up with a beautiful idea about how the Big Bang could have happened, invoking the relationships between the fundamental forces of physics and details of quantum theory. The idea is called inflation, and it explains just about everything about the observed nature of the Universe, subject to one proviso. Inflation specifies how hard the cricket ball was hit, and forces the density of the Universe close to the critical value. But that doesn't agree with what we see from the behaviour of galaxies.
You could say that the inflation theory is wrong. But studies of the famous cosmic microwave background radiation, the echo of the Big Bang itself, strongly support the idea of inflation, and provide a direct indication of the smoothed out density of the Universe on the very largest scales. The density measured in this way is, indeed, the critical value. But only one-third can be detected by its gravitational influence on galaxies. Where is the rest?
So, as far back as the mid-Eighties, cosmologists revived an idea known as the cosmological constant, originally introduced into the equations of cosmology by Albert Einstein. At the time, more than 80 years ago, Einstein didn't know that the Universe was expanding. Like his contemporaries, he thought that the Milky Way was the Universe. And the Milky Way as a whole is neither expanding nor contracting. But when Einstein used his equations of the general theory of relativity to provide a mathematical description of space as a whole, the equations insisted that it must be either expanding or contracting. He introduced the cosmological constant to hold the equations still. When Hubble found space really is expanding, there was no need for the constant, which Einstein discarded, commenting that it had been the biggest blunder of his career. But was it?
Choosing one value of the constant could indeed, as it were, hold the Universe still. But choosing a different value of the constant would make space "springy", and encourage the Universe to expand. There is even a reason for space to be springy - according to quantum physics, empty space is in fact a seething foam of what are called virtual particles, which pop in and out of existence all the time, in line with the notorious Heisenberg Uncertainty Principle. These particles should, in theory, contribute an energy that makes space expand.
But hold on a minute. Einstein's most familiar equation tells us that energy and matter are equivalent. With the right choice of numbers, the energy of empty space associated with the cosmological constant could be just two-thirds of the critical density. Add that to the one-third detected by conventional means, and you have the overall value predicted by inflation and measured by the background radiation.
So, some 15 years ago, theoretical cosmologists were predicting the existence of a cosmological constant equivalent to the presence of dark energy in the Universe, totalling two-thirds of the critical density. This is exactly what the new discoveries have found. The observers found it by looking at supernovae, huge stellar explosions that are visible far away across the Universe. In a supernova, a single star briefly shines as brightly as all the rest of the stars in its parent galaxy put together. Happily, in one particular kind of supernova explosion the exploding stars each have the same brightness, so the apparent brightness (or faintness, in fact) tells you how far away such a supernova is. This distance can then be compared with measurements of how fast the supernova and its parent galaxy are moving away, using the famous redshift effect - the faster they are moving, the redder they appear through a telescope.
The results show that the expansion of the Universe is getting faster, because the springiness of the dark energy is starting to overwhelm the inward tug of gravity. And it tells you the amount of springiness required to explain the observations - dark energy that is equivalent to two-thirds of the critical density.
The effect gets bigger the more space there is, so as the Universe grows, it expands faster. In the far future, dark energy will dominate, and galaxies will move apart faster and faster, as the Universe runs away with itself. But in the past, when there was less space between the galaxies, the cosmological constant was less, and gravity was relatively far more important. We live at a highly unusual moment in the history of the Universe, when the dark energy and the mass energy components are roughly the same size. Why? Nobody knows - so there are still puzzles for cosmologists to worry about in the next millennium. Meanwhile, though, give proper credit to cosmologists today - important though it is, the dark energy was not a surprise, but was predicted by theory, a theory now tested by experiment and observation in the best scientific tradition. It suggests that cosmology is real science, and that we really do know how the Universe works.
The writer is a visiting fellow in astronomy at the University of Sussex. His latest book is `The Birth of Time' (Weidenfeld & Nicolson)Reuse content