Ivan Massow, the ICA's 31-year-old new chairman, confirmed the details of the plan yesterday after a letter outlining his proposal was leaked.
The 18 celebrity patrons, who include singers Jarvis Cocker, David Byrne, Madonna, Laurie Anderson and film-maker Jane Campion, lend their name to the ICA's masthead and appear at occasional functions. But many of them never have any contact with the institute.
"One of the problems with the ICA is, how do you keep its image fresh and contemporary?" said Mr Massow. "Its work is and remains really exciting, but we have to make people aware of that from the outside. There's a perception of the ICA that it is not cutting edge. It has an undeserved reputation for stuffiness. Yet if you look at exhibitions like Die Young Stay Pretty it is very hard-hitting stuff. That exhibition shocked me.
"We have to look at all strands of how to keep the ICA fresh and so we are considering a review of how the patrons and presidents are made up. Some may want to get more involved. Those who don't, or who we never hear from, may stop being patrons."
Mr Massow added: "I wanted to look at it as not an insult or blow to people who have supported us in the past, but simply to consider who should be on there. The replacements would be selected because of their contribution to the contemporary art world."
Charles Saatchi, who sometimes acts as an unofficial advisor to the ICA, is not one of those on Mr Masson's wish list of patrons. "After all he already has his own gallery," he said.
"We would also consider if they should be selected by a panel, or whether they should change every year. It seemed like an innocent enough suggestion to me. But obviously someone is opposed to the plan, because a letter that was not meant for public consumption has been leaked. No decision has yet been made, and I am just one member of the council. It is wrong to suggest that I can somehow push changes through."
In recent years there has been increasing public interest in cutting- edge contemporary art following high profile events such as the Royal Academy's Sensation exhibition two years ago and the annual Turner Prize. Some art critics believe the ICA has been missing out on the contemporary art boom.
Mr Massow became a millionaire by selling financial services targeted at the gay community. He recently withdrew from the race to be the Tory candidate for Mayor of London to concentrate on becoming a Conservative MP.