I have lived in France for 10 years and travel to Belgium, Germany and Spain. It is untrue that "an identity card or passport must be carried at all times" and "I also have to carry a residence permit ...."
Some form of identification is useful if you want to pay by cheque in a shop where you are not known, because cheque guarantee cards don't exist here. A residence card, not "permit", is needed if you want to claim some state subsidy such as housing benefit. You can work, travel and pay income tax without a residence card.
The only time I would need to carry a passport is if I wanted to come home to Britain.There's the rub!