Two years on and Sir David Bell, Michael Gove's first permanent secretary at the Department for Education, is reflective about the way the country's education system is being run.
He thinks it is "remarkably unintelligent" of Mr Gove to characterise all those who oppose his education reforms as "The Blob".
"I've always thought that 'The Blob' was an extremely lazy term used by otherwise highly intelligent people," he told The Independent in an interview. "It does lump together a very diverse group of people and assumes they are homogenous in their objections."
This, says Sir David is not the case and many of these so castigated may even support some - or the majority of Mr Gove's education reforms.
"I think it is a remarkably unintelligent way to drive progress forward in the education system," he adds.
"Whoever is Secretary of State of the day has to strike a difficult balance. You don't want to be known as in the teachers' pockets or hating the profession - it is not an either-or.
"It is very hard to bring about change in the education system if you're not encouraging a coalition of the willing to get behind you and help you drive on with your reforms."
Sir David has abandoned a daily journey on the 6.19am train from Bedford to the Department for Education's offices in Westminster. Instead, as vice-chancellor of Reading University, he now lives just four miles away from his office and can go home to "freshen up" if he has a hard day at the office followed by an evening meeting.
He admits to sometimes hankering after his old jobs in the corridors of power (he was chief inspector of schools before becoming Permanent Secretary). "Then I get better," he quips.
On the day I visit, Ofsted has just published a damning report on one of the country's biggest academy chains - E-ACT. He shows interest but soon moves on to start discussing higher education issues - the part of the system that most concerns him now.
The biggest potential problem facing the sector is the non-repayment of student loans - estimates suggest 45 per cent of all loans will remain unpaid potentially plunging universities into a financial crisis in the years to come.
"I'm not one of those who wants to bandy about figures of what I think fees should be - we've hard £10,000 or even £16,000 in the past few months," he said.
He puts his civil service hat on and analyses where he thinks the politicians will go on this issue.
"It's not a short-term problem so I doubt if they'll do anything about it until after the election," he said. "I think we're in for another Browne-style review in the aftermath of the general election." (Lord Browne was appointed by Labour to head the inquiry which led to the raising of fees to up to £9,000 a year.)
For good measure, he predicts this new review will have a remit to report after the following general election. No-one wants to have to tackle controversial policy immediately before they go to the country, he argues.
Since becoming vice-chancellor of Reading University, he has become prominent on the higher education scene - taking on the chairmanship of a review group set up by the Higher Education Funding Council to England to look at language provision in universities.
"I wasn't at the Department for Education at the time the policy to remove compulsory languages for 14 to 16-year-olds from the national curriculum was taken," he says. "I think at the time."
He was with Ofsted at the time and remembers anecdotal evidence of widespread non-engagement with education on the part of 14-year-olds forced to study a language.
"Looking back at the decision 10 years on and now I'm not so sure it was the right decision," he added.
The impact of the decision in terms of the slump in take-up of languages at GCSE and A-level was far greater than had been anticipated and - although the introduction of Mr Gove's English Baccalaureate has spurred an increased take-up of languages at GCSE-level - the rise still has not filtered through to A-level. It is also too early to assess the impact of introducing languages to the primary school curriculum - it only comes into force this September.
Sir David is keen on promoting languages as a second string to degree courses (for instance, students could study business with Spanish). This could help with the promotion of minority languages such as Mandarin, Japanese and arabic considered essential for the future of our economy, he argues. His own university will be offering Spanish for the first time from September 2015.
He is at pains to dismiss any suggestion that a rift between him and Mr Gove led to his departure. "I had a great time working with Ed Balls and Michael Gove - two of the biggest players in their respective government who both had the ear of their Prime Ministers - and I worked with the first Coalition government for 60 years," he said. "We achieved huge things and had our fair share of crises. I just wanted to move on.
"I really don't miss it. I was just lucky to have had those 10 years at the centre of things."