Dom Joly: 'How are his wife and children?' Funny, that didn't come up

I am seriously outnumbered up here in the Canadian "wilderness" (I'm four miles from the nearest latte, so suck on that, Bear Grylls). It's not bears or beavers or moose that have the numbers – it's women. My four-year-old son, Jackson, and I are the only men in a lakeside cottage rammed to the pine beams with the opposite sex. There are cousins, aunts and grandmothers – a formidable group when all together. It's actually interesting to be able to observe them from up close, in their natural habitat.

I'm often criticised by Stacey for non-communication. I'll go and have a drink with a friend and come back to a Guantanamo-style grilling: "How's his wife? What are the children up to? How is his brother who is dying from some flesh-eating disease?" I'll look nonplussed and be forced to admit that none of these things really came up. When pushed, I struggle to remember what we talked about – just stuff and lots to drink... it's what blokes do.

Women, on the other hand, talk a lot about people they don't know. There are endless discussions about whether Reese Witherspoon is happy with Jake Gyllenhaal or whether Madonna has implants in her arms. The place is littered with US gossip magazines that are pored over again and again, as though there is some hidden message within.

Another favourite topic of conversation is all things morbid. Top of the list are missing children, and they will all go into hushed tones and each take a turn in describing the horrible fate that might await the poor things. They almost compete to come up with the most horrific scenarios, holding the floor with graphic descriptions of depravity while their audience get more and more upset.

If anyone within any of their interconnecting universes is even remotely ill, then the discussion turns to potential outcomes (mostly fatal) and the subsequent fall-out. Then the whole group gets depressed. Then someone remembers there is a new tub of ice cream in the freezer and everyone gets a sugar rush and the whole rollercoaster starts over again. It's very tiring.

Jackson and I often head off together down to the rocks above the lake and do a bit of fishing. I'm starting to really understand the appeal of fishing, which is mostly about escape and very little to do with fish. It's like golf, except there's not even any walking involved.

All too aware of my overall inertia this holiday, I've entered a regatta on our lake. I've signed up for the one-metre diving (over-17s), the T-shirt swim relay (with Stacey and Parker) and log-rolling. It all sounds fun, but log-rolling is the big one. Stacey and I have to stand on a huge floating log and try to stay on top of it by running in sync. We'll give it our best shot but, deep down, I'm sure she'd prefer her current heart-throb, actor James MacAvoy, up there with her.

Sadly, my holiday is coming to an end. I'm off directly from here to Beijing to cover the Olympics for 'The Independent'. All my life I've dreamt of being a foreign correspondent and I've finally achieved it – sort of. I'll be wandering around the place, seeing what it's all about. What is the Olympic Village? How dull is the five-metre air pistol? Is this strange-tasting hotdog really dog?

I'm staying in Hotel Number Three and have the feeling that I shall miss my lady-packed cottage over the next couple of weeks. Fortunately, I will have a village full of drug-fuelled, über-competitive, adrenalin junkies for company – and that's just my fellow journalists.