According to the Education Secretary schools should change their term dates to help parents pay a little less for their holidays.
Mr Gove spoke as Parliament was preparing to debate a petition asking “to stop holiday companies charging extra in school holidays.”
An ITV News investigation found East Sussex Council had received nearly £20,000 from under 500 fines issued over the last six months from parents taking their children out of school during term time.
Speaking at the debate in Westminster Hall, Jenny Willott, consumer affairs minister, explained vacation prices are hiked due to high demand during half-term. However as schools are given the option to change the dates of their holiday periods, these may come to drop.
Ms Willott made it clear the coalition would not be taking on a regulatory role over the issue, stating the higher prices were not due to “market abuse” as some had alleged. Instead, she said the prices were a reflection of the current market realities.
Propositions to allow schools to change their half-term dates, and thus make the peak period more flexible, could potentially help in bringing prices down, as the demand would be spread out across a longer length of time.
The debate was brought forward to Parliament following a petition which collected 168,196 signatures on the official government website. The petition was created by Donna Thresher and asked for the current situation of "holiday price hikes" to be assessed.
People have spoken out against “unfairly high” prices during the peak times, however the Government brushed off any allegations that the holiday industry was “cashing in” on the situation and manipulating the market’s offers.
The petition notably demanded a cap on the price increase during school holidays to allow for “guilt-free family time”.
Ms Willott told the debate: "I think the staggering of school holiday periods could indeed lengthen the period of peak demand and it could therefore help bring prices down and I completely understand why people are suggesting as a result the Government should arrange for holiday periods to be spread.
"But at the moment the Government doesn't set term and holiday dates. Currently local authorities set the dates for community and voluntary-controlled schools but academies, free schools and voluntary-aided schools - church schools - set their own dates.
"What the Government is doing is the Deregulation Bill, which is currently before the House, will extend the power to set term dates to all schools by 2015.
"So all schools will be able to set their own dates and be able to vary them.
"So what the Government believes is term dates - (which) have been raised by a number of people today actually - that this is an issue which should be dealt with locally in negotiation and cooperation across an area so that it can take into account the educational needs of the students in the schools and all the practicalities in the school year..."
The minister acknowledged dates would have to be arranged regionally, an issue the bill would look into addressing when it would come to be law.
Concluding, Ms Willott said: "The Government is not convinced higher prices in school holidays are as a result of market abuse by the holiday industry, but rather they reflect the market forces in a very competitive sector, which is also made worse by the fact that we're part of an international competition as well in that market."
Ms Willott said she was "very sympathetic to those who struggle to afford a holiday in peak season", adding that she appreciated the significant differences in price.
On school pupils going on holiday during term time, the minister said it was for headteachers to decide whether the application met the "exceptional circumstances" criteria.
She said: "Just to make it clear, the Government hasn't said that no absence is possible. It has given headteachers the discretion to make that call and we also haven't specified what constitutes exceptional circumstances as we think individual cases need to be considered individually.
"A number of people mentioned the need to trust headteachers and that is exactly what the Government is trying to do to ensure headteachers have the power and the discretion to look at the individual circumstances of a particular application and take that into account."
Read more: Government should spread school holiday dates out to bring travel costs down, say tour operators
Additional reporting by PA