Lord Patten, who became chancellor of Oxford last year, lambasted public school yobs "with more money than sense" who shamed the university.
Oxford has been dogged by stories of upper-class students misbehaving. Last year four undergraduates belonging to the Bullingdon Club, a secret drinking society with some of the university's wealthiest students among its members, were arrested after trashing a pub.
Lord Patten, in a speech at Green College, Oxford, to mark its 25th anniversary, admitted that the university had "a problem" with attracting talented students from poorer backgrounds if they read about the bad behaviour of its privileged undergraduates.
Research from the Sutton Trust suggests at least 3,000 students every year from modest backgrounds achieve outstanding A-level grades but choose not to apply to the country's best universities.
Lord Patten said Oxford had to suffer "occasional `Brideshead' coverage", referring to Evelyn Waugh's classic novel of upper-class life, Brideshead Revisited.
"It cannot be very good for recruitment of able sixth-formers in Yorkshire comprehensives when a bunch of yobs with more money than sense do what young yobs have always done and always will do, that is to behave in ways that shame the families and schools that have lavished advantages on them."
Lord Patten also said that he hoped fees would be uncapped by 2030. The present plan will allow universities to charge fees of up to pounds 3,000 a year from 2006. Almost all universities have indicated they will charge the full amount.
The 100-year-old Bullingdon club, whose previous members have included Darius Guppy - Earl Spencer's best man - and Lord Bath has a history of drunken vandalism. It was depicted as the "Bollinger Club" in Evelyn Waugh's Decline and Fall.
Oxford has been chosen as a guinea-pig town for a new "shop-a-yob" scheme encouraging people to report acts of hooliganism to a helpline.
Lord Patten, the former Conservative Party chairman who became the last governor of Hong Kong, also called for Oxford to be allowed to charge unlimited tuition fees to help keep its place at the top of the world rankings.
He argued that the elite university would struggle unless it raised much more private funding and charged higher fees.