The deeply damaging report issued by the committee yesterday accuses Mr Smith of spending too much time on "glamorous and trivial matters", such as Britpop, and not enough generating income from tourism.
The report by the cross-party committee of MPs added that he was not tough enough in negotiations with the Treasury and his department had still not "boxed its weight" in Whitehall.
The culture, media and sport committee derided the department's "Beefeaters to Britpop" image and said Mr Smith must make it his "highest priority" to start advancing its interests in the Cabinet.
The report will be intensely embarrassing for Mr Smith, who has just written a rather self-congratulatory book about his portfolio, called Creative Britain. He has already faced criticism from senior arts figures, including the theatre director Sir Peter Hall, over cash cuts.
Yesterday, the committee's criticisms were being taken seriously enough for the Prime Minister's office to issue a statement supporting the minister. A spokesman for Tony Blair said: "The Prime Minister has complete confidence in the Secretary of State. He is an excellent Secretary of State doing a fine job."
But a senior Conservative member of the committee, Michael Fabricant, said Mr Smith should resign immediately. "I have never seen such a damning report before. If Chris Smith has any dignity at all, he will go before he is pushed," he said.
In its report, the committee accepted that the department was committed to enhancing its influence in Whitehall and highlighting the achievements of the areas it sponsors. "However, a commitment is not an achievement and unfortunately the department has not enhanced its influence in the way it says it would like to do," it said.
"The Secretary of State should now make it his highest priority to advance the department both within cabinet and by taking a much tougher attitude in his negotiations with the Treasury, which has certainly not been achieved and may not even have been attempted."
The committee's main concern was that tourism was being ignored at the expense of more high-profile aspects of the department's work, even though it was Britain third largest industry, worth pounds 40bn a year.
"We are, therefore, deeply concerned that, in policy statements by the department and in public statements by ministers, tourism is subordinated in favour of more glamorous and trivial matters," the report said.
"We further recommend that tourism should be the lead responsibility of a minister in the department and that the Secretary of State should concern himself with it far more, in actions, in policy and in public statements."
Although tourism accounted for more jobs than agriculture, coal mining, steel making, car manufacturing, aircraft building, food production and the textile industry combined, it was not even mentioned in the department's name. The MPs urged ministers to consider renaming in order to reflect the central importance of tourism in its activities - even though it is barely a year since its name was changed by the Government from the Department of National Heritage.
The report also urged more contacts with the Department of Education and Employment over sporting matters, and criticised Mr Smith for not making sure the Arts Council and English Heritage were cost effective.
Mr Smith's own department attempted to accuse the committee of inaccuracies, but without naming any. A spokesman at the department said: "At first sight, the report contains a number of useful suggestions and a number of inaccuracies. We note that the views of a number of senior figures from the outside world, especially in the tourism industry, do not appear to coincide with the analysis deployed by the committee."
And Mr Smith's parliamentary private secretary Fiona Mactaggart, said the report "in key respects, is superficial and fails to provide evidence that substantiates its conclusions. It recycles criticism made by its predecessor committee of the last government."
In response to Mr Fabricant's call for the Secretary of State to resign, she accused him of disclosing the contents of the report in advance.
The Conservative culture spokesman, Peter Ainsworth, said: "I do not think we are in resignation territory yet". But he added that he "would not be at all surprised if he (Mr Smith) got the chop" in the impending Cabinet reshuffle.