British Olympic Association chairman Lord Moynihan says that the problem of empty seats "has to be sorted".
He said the situation "really seriously concerns" him, adding: "I do not feel any more relaxed today than I did two days ago.
"This has to be sorted. It is unfair on Team GB not to have maximum support with as many people sitting in the seats as possible.
"It is also absolutely unreasonable to the public, who are so supportive of Team GB, not to have the opportunity to come to a Games and enjoy an experience which most of them will never have again in the United Kingdom.
"This is a major take-away issue from these Games. The IOC (International Olympic Committee) will need to look at it very carefully.
"Every empty seat disappoints me because we need every seat filled to radiate the support from the British public, who are passionately interested in sport and absolutely 100% behind Team GB."
It is a subject that has got to "tax the minds" of the people who are responsible for the ticketing each and every day until the end of the Games, he stated.
London 2012 has been looking into the issue, including reallocating seats and allowing volunteers and the military to watch the sport to help with the situation.
Some people who have tickets for accredited seats have not turned up and sports federations are helping out by returning some tickets to a pool for reuse.
The packed venues, as seen at the rowing competitions at Eton Dorney, is an example of how the packed crowds have made a huge difference to the atmosphere.
Britain's rowing team, who have already seen medal success, spoke of the power of coming in to "that tunnel of sound" at about 600m along the course made by the noise of the cheering supporters, according to Lord Moynihan.
He said: "Our rowers have really benefited from that atmosphere down there and, of course, you can replicate that over 26 sports."
London 2012 spokesman Jackie Brock-Doyle said: "Colin (Lord Moynihan) should come and talk to us if he has got other ideas. We think we are moving through this quite successfully.
"To have 80 per cent of the accredited seating filled yesterday was a very different situation to what it looked like on Monday.
"As we move in to the later rounds of a number of sports, the situation changes."
She said London 2012 is having "quite detailed conversations" with the IAAF (International Association of Athletics Federation) on their accredited seating in the stadium ahead of the start of the track and field competition on Friday.