Mr Banks made the comment at a meeting of grass-roots fans after repeating the Government's advice that fans without tickets should not head for the tournament in France.
The minister confirmed he had tickets for matches, but said it would be seen as hypocritical if he attended.
During his speech to delegates at the Football Supporters Association meeting, he criticised the allocation of World Cup tickets, adding: "We have to make sure this mess never happens again."
He also used the meeting at Wolverhampton Wanderers Football Club to deny rumours that England is ready to step in to host the tournament in 2002.
Mr Banks was responding to reports in the media that Japan and South Korea may not be able to fulfil their commitment to the world football governing body, Fifa, because of inadequate finances.Reuse content