What lies within: Why we are fascinated by caves

They inspire artists, hold ancient secrets and, in the popular imagination, they're the refuge of bandits and terrorists. Caves remain unmapped in our know-everything age.

Last month I went to a particularly cave-like cinema (the basement walls, outside the bathrooms, were covered with synthetic stone) to see Cave of Forgotten Dreams, Werner Herzog's 3D-documentary about cave paintings. The film is a rapturous tour of the Chauvet cave in southern France, a dramatic pocket of rock that was discovered, in 1994, to contain the oldest known paintings in the history of the world: portraits, up to 32,000 years old, of mammoths, reindeer, lions, bears and woolly rhinoceroses – the improbable megafauna of southern France during the last ice age.

Aside from the art covering its undulating walls, Chauvet is a totally alien landscape: barren, inhospitable, forbidden. It is, like many caves, both rugged and delicate. Herzog's access was severely restricted. He was allowed only three crew members and four hours of filming at a time; he had to assemble his cameras inside the cave and shoot everything from a narrow metal walkway. The landscape he records is terrifying and magical. Glittering stalactites and stalagmites stretch toward one another, through the darkness, in epochal slow motion. The space is dominated by the remnants of cave bears: scratches on the walls, tracks on the floor, bones everywhere. The back of the cave is full of poisonous gas. It is not, in other words, a place for humans.

The miracle, however, is that humans were here. Watching the film in the dark cavern of a cinema in New York City, on a 21st-century Monday, surrounded by contemporary Homo sapiens I knew nothing about, enduring Herzogian blasts of New Age music and heavily accented spoken-word poetry (''Are we today the crocodiles who look back into the abyss of time?''), I found myself most fascinated not by the paintings themselves – although these were gorgeous and strange – but by the more mundane evidence of human life in the cave. There's the footprint of an eight-year-old boy next to the footprint of a wolf. There are black stripes where, 28,000 years ago, visitors scraped their torches against the cave wall to rekindle them. Near the entrance there's a cluster of red dots that turn out to be palm prints, made roughly 30,000 years ago, by a 6ft-tall person with a crooked little finger.

That crooked little finger, in particular, stuck in my mind. It's an incidental detail that conjures, uncannily, a whole person. It creates an intimacy that, given the timescale involved, is hard to believe. In the face of the truly sublime, the mundane often takes on an extraordinary power.

I saw Herzog's film just after the death of Osama bin Laden, and like everything that happened then, the movie seemed to bend around that event. The mythology of Bin Laden, of course, had much to do with caves. He was supposed to have been living in them since 2001, foiling history's most technologically advanced superpower with the most archaic tool possible: a hole in a rock. He was often portrayed as a Stone Age super-villain, plotting our destruction from the depths of his mountain lair. (It's interesting that Bin Laden did not see himself that way. In his book The Bin Ladens, Steve Coll writes that he ''mocked his Western adversaries for misunderstanding him as a pre-modern fanatic, a bearded loner in a faraway cave; he saw himself, instead, as a master of global technology and change''.)

Caves, in this mythology, were deeply anti-American: beyond surveillance, devoid of information, immune to cultivation or colonisation or commercialism. Tora Bora was a negative image of Manhattan: a city built into the ground instead of rising up dramatically from it. And Bin Laden was Tora Bora. The elaborate network of reinforced caves – falsely rumoured to include its own hospital – had been built at his instigation, largely with his family's construction equipment, even occasionally with him at the wheel of a bulldozer, personally hollowing out the perilous terrain. It was fascinating, then, to learn that Bin Laden's actual habitation, at least for the last five or six years, was a different kind of cave: a walled, three-storey concrete bunker largely disconnected from the world.

Reading the newspaper coverage after his death, I found myself ravenous for details of Bin Laden's secret life. I recognised, in this, an echo: it was the same feeling I had watching Herzog's film. Once again, the most trivial details were the most thrilling: that he took a walk every day in the courtyard; that one of his couriers regularly brought back Coca-Cola; that he cured his kidney ailments with ''homemade remedies, including watermelon''; that some of the 100 memory sticks the Seal team confiscated were loaded with pornography; that Bin Laden left behind handwritten notebooks about a major attack. (I found myself wondering less about the details of the attack than about what Bin Laden's handwriting looked like.) And then there was that mesmerising silent video of Bin Laden sitting on his living room floor, channel-surfing.

It occurred to me, watching this, that seeing Bin Laden adjust the volume on his television set carries the same uncanny weight as seeing 28,000-year-old torch-scrapes on a cave wall. It's someone who is gone, impossibly remote, beyond our understanding – and yet doing something that we've all done, or surely would have done in his place: an action so trivial it leaps over all the fences of ideology and geology and time within which we like to define ourselves. It's an eerie kind of time travel, an existential transfusion. It made me think that a 3D-tour of Bin Laden's compound would probably be a blockbuster.

The appeal of caves is, obviously, primal. They offer, in their darkness, both an instant physical reward – shelter – and something more metaphysical. For as many millenniums as there have been humans, caves seem to have been considered a contact zone with the magical, the otherworldly, the irrational, the unconscious. Prehistoric people used them as burial grounds and ritualistic art galleries. The Greeks built shrines and oracles in them and populated them with monsters. (Odysseus's Cyclops lived in a cave.) Ancient Buddhists dug out caves everywhere – 30 at the base of an Indian waterfall, 500 in a mountain at the edge of the Gobi desert – and stuffed them with their most elaborate art. Christ was entombed in, and then resurrected from, a cave. The Dead Sea Scrolls were found buried in 11 different caves. It's no accident that walking into a great cathedral or mosque feels like entering a giant above-ground cavern.

But it doesn't take religion to sanctify a cave. Caves challenge any common-sensical division between secular and sacred. A cave is a paradox: a place defined by its absence. It operates on a timescale that we can't even begin to comprehend – a timescale that is obscene to any species that cares about life and tends to measure things in minutes and years and decades. The formation of a cave is appallingly incremental. Most often it happens when water, trickling down through the air and the ground, picks up carbon dioxide, creating a very weak acid. This acid finds its way into the tiniest of cracks in the rock and begins, very weakly, to dissolve it. After a million years or so, this nibbling forms a nice-size cave. Stalactites and stalagmites, created by minuscule mineral deposits left by single drops of water, form at a rate of roughly one cubic inch per 100 years. The tallest known stalagmite is 220ft high.

A cave, in other words, is time showing off. Most geological features form slowly, of course, but caves seem extra miraculous because of the intricacy, the beauty and the delicacy of the structures – all created not by plate tectonics or giant rivers but by individual drops of water. It's like painting the Sistine Chapel with an eyelash.

Today, in the omnipresent data storm of the 21st century, the primal appeal of caves takes on a new dimension. The Earth, including the ocean floor, is now comprehensively mapped. Caves are not. Google's camera cars have yet to drive inside them. They remain blank spaces. In a world of instant access, caves are the very definition of slow. In a world of constant presence, caves are aggressively absent. In a world of superficiality, they are profound – literally profound, in the original sense of ''deep.'' (Latin profundus: ''before the bottom.'') This means that we're even more drawn to them because they preserve something precious that's becoming hard to find: ignorance, blankness, the integrity of total silence. Today, given that we can know just about anything, a cave is even more of a cave.

My parents divorced when I was four, and one of my dad's favourite activities, when he had us on weekends or during holidays, was to take us to caves. He took us to Mercer Caverns, a three-million-year-old hole in the ground that was once a sacred burial site (prehistoric people apparently used to roll dead bodies in) but was turned into a tourist destination by a gold-hunting miner in the 1880s. He took us to Moaning Cavern, which, we were told, could swallow the Statue of Liberty. We went to Oregon Caves, where I have a memory of a bat flying at my face. We saw not only stalactites and stalagmites but also ''angel wings'' and ''soda straws'' and ''chocolate waterfalls'' and crystals that looked like sea anemones or carrots or pipe organs. We explored chambers called the Meatmarket and Godzilla's Nostril.

Recently, probably in an attempt to recapture the lost prehistory of my childhood, I went caving again. A friend told me about a cave he discovered in the woods not far from where we live: a horrifying black hole that, when he saw it last winter, had a giant icicle snaggling down from the top of its mouth. He had only peeked into the cave but had clearly been thinking about it ever since; as soon as I mentioned the subject to him, he insisted that we go explore it. We were looking, I think, for all the things people are usually looking for in caves: adventure, exhilaration, wonder, ''ourselves.'' (As Yoda, in The Empire Strikes Back, puts it when he tests Luke Skywalker by sending him alone into a cavern, the cave contains ''only what you take with you.'')

Google had little to say about the cave. The owner of the hiking shop had never heard of it. From the history of the area, we could infer that it was part of a network of 19th-century iron mines. My friend told me, ominously, that a dead body had turned up in the vicinity a few years back, and people had been trying to hide the cave's existence ever since.

We assembled a team of novice adventurers and, on a drab, grey, drizzly day, hiked out (past a couple of beaver lakes, down a muddy slope) to the mouth of the cave. It was, as promised, horrifying: big, deep, savage, black, violent, aggressive. A cold wind, complete with gothic mist, blew out of the opening, as if the earth were breathing on us. (This, by the way, is how most caves are discovered: people notice an inexplicable wind coming out of the rocks.)

We turned on our headlamps and entered. A steep descent soon levelled out into a floor of small rocks and logs, all under maybe a foot of water. The cavern was huge: probably 70ft tall and 25ft wide. I understood, immediately, why caves have been sacred spaces to so many cultures. You can just feel it. As we got deeper, I kept looking back at the opening –that savage black mouth –and from inside, it looked like the exact opposite: a big burst of light.

We had plenty of adventures, it turned out, in our couple of subterranean hours. First we had to cross through a long, bottomless pool of freezing cave water by hanging on to an old rope stretched across one wall. We hiked past a block of ice the size of a mini-fridge. We passed under a rectangular hole high up in the ceiling, from which a shaft of light spilled that seemed to create its own microclimate – 10 degrees warmer, with a carpet of wet leaves that supported (once we looked closely) all kinds of life: frogs, orange newts, centipedes, mushrooms. We found a cave within the cave, a crevice that seemed to go nowhere until one of us managed to squeeze, claustrophobically, through it. The ceiling rained fat drops of water on us. And just about everywhere, we passed evidence of our predecessors: beer cans, rusty old tools, a broken torch, a garden hose.

Then, before we were ready for it, we came to the end. The cave narrowed to a flat circle of rock, around which bloomed a gallery of graffiti – the names, in pink and blue and yellow, of previous explorers: Eric, Kevin, Ozarco, Joey, KRobot. We turned off our headlamps and stood for a minute in dark silence. Then we headed back out, past the frogs and the ice, through the freezing pool, over the rocks and logs and finally up, once again, through the big shining mouth, which immediately turned dark again. Outside, it was raining, and the green that I'd hardly noticed on our hike in – ferns, moss, bushes, trees – now seemed like the greenest possible green, the original invention of green. The world, it seemed, had woken up while we had been inside the cave.