His long-planned UN summit, which should be the crowning achievement of his second mandate, was thrown into chaos when the American delegation brought in 750 amendments to a 39-page final draft declaration only days before the document should have gone to the printers. That text is now 45 pages long and still under negotiation.
To cap it all, he had to cut short his hiking holiday on the Seven Sisters of the South Downs two weeks ago to return to New York, just after recovering from surgery on his shoulder.
So you would expect that, amid all the turmoil, the Ghanaian, 67, would be showing signs of strain. Yet during an hour-long interview in which he tackles a wide range of issues head on, he seems relaxed and is in combative mood, batting away suggestions that he will have little authority during his remaining 15 months in office.
"I don't feel like a lame duck," he says. "I have work to do, and I'm going to press ahead and do it. I know there's lots of criticism, lots of attacks, lots of accusations of lame duck and all that. But you know the old French saying: les chiens aboient, mais la caravane passe. The dogs bark but the caravan still moves on. I want to keep going, I have work to do, and I hope my staff are saying the same."
It is not the first time that Mr Annan has faced calls for his resignation. As under-secretary general for peacekeeping during the 1994 Rwandan genocide and after the 1995 Srebrenica massacre in Bosnia, he had to account for his actions. Then, as now, he survived thanks to the political support of the governments in the Security Council. Some diplomats even suggest that the big powers may feel that a weakened UN chief will be more biddable than one whose credibility is unchallenged.
Although his relations with the new American ambassador to the UN could prove tricky - many suspect the hawkish John Bolton has been sent to New York with the mission of destroying the organisation - Mr Annan points out that he enjoys the support of US President George Bush and his Secretary of State, Condoleezza Rice, as he did with her predecessor, Colin Powell.
"I work well with the President personally and with Condi Rice. We've been on the phone in the past few days on Katrina; we offered assistance and they've accepted. On many issues, from UN reform to Iraq, on the Iran nuclear issue, the withdrawal of Syrian troops from Lebanon, we've worked closely together."
Nevertheless, he recognises that he has had to cope with constant sniping by a certain sector of American society, which led to open calls for his resignation at the end of last year over the oil-for-food scandal after the head of the UN-administered humanitarian programme, and Mr Annan's son, were implicated. It is said that he feared the Bush administration was out to get him when he said a year ago that the war on Iraq was illegal.
"We have right-wing groups who keep attacking us," he says. Chief among his critics has been the outspoken Mr Bolton, who once said that if the 39-floor UN building lost 10 storeys, "it wouldn't make a bit of difference".
The UN chief must be privately hoping that the new ambassador will be kept on a tight rein by his political masters. "My advice, when he said he was going to come and shake the place up, was to remind him that there are 190 ambassadors like him, and that the way you get things done at the UN is to convince and to persuade, and you have to persuade your peers to get results."
Since becoming secretary general in 1997, Mr Annan has walked a tightrope as the servant of the 191 UN member states, and the five permanent members of the Security Council in particular. He has had to gauge the right moment to intervene as the moral compass of the world.
In the current impasse over the summit declaration, which is supposed to be a blueprint for UN reform, Mr Annan says he warned the General Assembly president, Jean Ping of Gabon, that 39 pages were too long for heads of state and government to sign up to. But "they are jealous about their prerogatives. Sometimes they think I push them too hard or I interfere too much. When they are stuck, sometimes they want me to intervene. When they are divided and I intervene, they say I am taking sides."
But what is more damaging, the US not engaging at all, or the US coming in at the last minute with 750 amendments as Mr Bolton has just done? "The Americans are engaged and they should engage. They are an important member of the organisation. I have assurance from Condi Rice and from the President that they take reform seriously. Having said that, the approach you take to achieve your goal can help you move faster or can slow you down."
Mr Annan, the "Teflon" secretary general, is already making sure that he will not be the scapegoat if the summit fails. "It's been clear to all who have followed the process that I took time to really prepare this well. It was three years ago that I gave them the signal that the UN had reached a fork in the road and that we needed to do something. I set up a high-level panel which gave us a solid report" in December last year. This was followed by the millennium development project led by the American economist Jeffrey Sachs who consulted with 240 economists to produce a report last January. Mr Annan synthesised their conclusions in his own report, In Larger Freedom, in March, which became the basis for the summit's draft declaration.
"I made sure they got all these documents well ahead of time." Some will say that the secretary general lost his attempt to implant his legacy, he admits, "but really the failure will be that of the member states".
Among the burning issues that Mr Annan worries about is the "tension and instability" in Iraq, which is unlikely to die down as a result of the draft constitution. He is also trying to coax President Robert Mugabe of Zimbabwe into accepting international help to end the crisis in his country. "Zimbabwe will be there after Mugabe," he says. "We have to find ways of engaging the government and getting assistance to the people and ensuring that the situation does not get beyond repair, because if we were to do that we would inherit a much more difficult situation to deal with."
But with the end of his term in sight, Mr Annan now allows himself to ponder his legacy. As the organisation's first secretary general from sub-Saharan Africa, he would like to be remembered for development achievements that will largely benefit his own continent.
"I have tried to get governments and the people to understand that there is an economic basis for conflict which we also have to tackle. I have really tried to broaden and deepen the UN's involvement in development issues as epitomised by the Millennium Development Goals. which lay down a timetable for the eradication of poverty and disease.
"Whereas in the past we were much more focused on political issues, I have tried to stress good governance, human rights and the dignity of the individual. You cannot have development without security, and you cannot have security without development and you will enjoy neither if you do not respect human rights.
"So I would hope that when it's time for me to move on, one would say that the peoples of the world are now much more engaged in UN activities. After all, the ideals that we exist to protect and defend belong to the people, it's in their interest that we do this. And finally, I would want it said that maybe the UN is working a little better now than it did when I took over."
BORN: Kumasi, Ghana, 8 April 1938
FAMILY: Married to Nane Annan, a Swedish-born lawyer and artist who is the niece of Raoul Wallenberg, the Swedish diplomat who saved thousands of Jews from Nazi concentration camps. They have three children from previous marriages.
EDUCATION: Studied economics in Kumasi and completed his degree at Macalester College in St Paul, Minnesota, in 1961. He was a graduate student in Geneva in 1961-62 and, a decade later, a Sloan Fellow at the Massachusetts Institute of Technology's school of management.
CAREER: Joined the UN in 1962 as a budget officer with the World Health Organisation in Geneva. Rose through the ranks in Geneva and New York to become under-secretary general for peacekeeping from 1993 to 1996. Became secretary general in 1997. Re-elected in 2001.Reuse content