Bernard Oliver, from Blackley in Manchester, had worked as a driver for the charity full-time for over four years when his employers cut his hours to two and half days a week.
He alleges that officials thought he should be putting his feet up and relaxing because of his age.
Mr Oliver had managed a shop and worked as a driver for the charity, taking home pounds 540 a month. He resigned when he was ordered to share his job with a volunteer, and now plans to take the organisation to an industrial tribunal claiming constructive dismissal.
"I am not too old and I certainly don't feel it and I am not ready for retiring. It all boils down to discrimination, age discrimination," said Mr Oliver.
"I'm fit and healthy and there is no reason why I can't carry on doing the job that I have loved for four years."
Mr Oliver, who spent 10 years working as a carer at a psychiatric hospital before moving to Age Concern, said: "I felt terrible when they sat me down and said I was too old to do the driving job because I was over 65. They said I could still do voluntary work for them but it felt like an insult after the years I had spent working for them.
"Now I'm left with my P60 after all the years I devoted to them, and have just my pension to live on."
Margaret Burford, assistant director for Age Concern said: "Retirement for employees has to be taken at 65 but people can stay on with the approval of charity's trustees. Mr Oliver resigned after being offered the two- and-a-half days a week work option. It was, however, the only post available to him."