In a long-awaited response to MPs, Ms Short said the UNHCR should explain why it failed to react adequately to the exodus of ethnic Albanians fleeing the province.
Her call for an "independent, external valuation" of the agency's performance follows criticism from the Commons International Development Committee that it had badly mishandled the crisis.
Ms Short rejected claims that the Nato alliance should have built refugee camps in advance of the bombing, but agreed that UNHCR should have reacted much more quickly and efficiently.
In its own response to the MPs' criticism, the UNCHR admitted that it "should certainly have been better prepared".
Ms Short revealed that the Department for International Development had received no information about the UN's contingency planning process and was not invited to take part in the project.
The department had paid pounds 1.5m to the UNHCR for its Kosovo activity, but would in future link all funding to improvements in the agency's effectiveness.
Among the areas identified by the UK Government for improvement were contingency preparedness, field performance, financial accountability and openness.
The department's response to the MPs' report also makes clear that the huge problems with registration of refugees in the border camps in Macedonia and Albania were the fault of the UNHCR, and Ms Short complained about the issue at the time.
She denied that Britain had been unwilling to take large numbers of refugees and said the Government had made clear that it was up to the UN to find "suitable candidates" to be flown to Britain.
In her own submission, Sadako Ogata, the UN High Commissioner for Refugees, said she accepted that her organisation had not been sufficiently prepared, but would strive to do better in future.
Ms Ogata stressed, however, that the MPs' criticisms were largely unfair and failed to recognise that the Kosovo crisis had been a difficult and complex situation.
The select committee had described as a "disgraceful case of neglect", the UN's failure to check and keep the identity of all the refugees as soon as the came over the border. Ms Ogata said that such a system would have not been in the best interests of the refugees "whose immediate need for security, shelter and assistance was more important".
"Despite the problems and shortcomings, the rapidly mobilised response to the influxes did meet immediate life-sustaining needs, with minimal avoidable deaths among many hundreds of thousands of refugees," Ms Ogata said.