The Liberal Democrat leader disclosed in an interview with The Independent that members of his own shadow cabinet are "going around saying 'well I do think this or that'."
But Mr Kennedy believed that attacks from within his party were not widespread.
He was "prepared to take" criticism he said, adding: "If I am not getting criticised I am not making waves."
He said he agreed with remarks about his laid-back style by his former speech writer who said he should act more like a leader, and less like a party chairman.
Asked on the BBC about an article in The Independent by Richard Grayson, in which he said Mr Kennedy "has work to do in persuading the country that he is a potential prime minister", Mr Kennedy said: "I quite accept what he says. It's a good reflection of what I am feeling myself."
He has approached Mr Grayson to congratulate him on the article.
But Mr Kennedy's admission will be seen as a blow to his authority and will be seized on as an admission that he must up the ante to prove he is up for the job. He told Channel Four last night: "We all must up our game, me included."
He acknowledged there was public talk of colleagues being put in the frame to take over from him. He said this was better than people saying they were "stuck" with him because there were no able alternatives.
"We have got to a position of people like yourself who do interviews and write speculative pieces saying there are quite a number of people who could lead this party one day," he said. "That is a lot better than saying they are stuck with what we have got because there is nobody who is interested in it because it is not worth their ambition or there is nobody we think is really terribly up to it."
Mr Kennedy insisted he had no plans to stand down before the next election.
"I am saying, if you are willing to have me then I am willing to do it. I don't do that thinking 'well I might be here for four or five years of a parliament then again I might not'. That is not responsible," he said.
Mr Kennedy is planning to make a highly personal conference speech tomorrow, instead of the normal policy-filled address, to bolster his personal standing.
He is expected to set out his view of a multi-layered British identity - discussing his Scottish roots - and to stress the contribution of immigrants to Britain.
He told The Independent he was concerned about the absence of ethnic minority MPs in his party. He said supported using positive discrimination to increase the number of black and Asian MPs.
"As far as positive discrimination generally is concerned I have become a convert to the cause because I think it has become so pressing. I just feel that, yes, we have got to do something," he said.
He said he was not sure whether proposals for a form of positive discrimination to be put forward this week by Simon Hughes, president of the party, were the right way to boost the number of black and Asian MPs because there may be "practical problems".
But he said he supported his colleague's drive to use affirmative action to increase representation. "I think the push that Simon is leading to get more people coming forward who can then become Parliamentary candidates in winnable seats for next time is absolutely essential," he said.
Mr Kennedy also intends to take a more robust line on increasing the number of women in the party. He said more work needed to be done to make the parliamentary party reflect the make up of the country. In a departure from the line peddled by his colleagues that having nine women MPs out of 62 is an admirable electoral outcome, Mr Kennedy said: "We have done significantly better. We still have a long way to go."
He signalled he would challenge activists who have blocked moves to allow the use of affirmative action to promote women. "I think now in this Parliament with a longer timescale and a growing base of women, it could be that there is more of a receptiveness" to using positive action.
Yesterday, Mr Kennedy appeared in good spirits despite a string of negative headlines reporting defeats for the leadership and questions about his style.
He said the attacks on the leadership were due to his opening up of a review of party policy, which has placed the party under the spotlight.
He defended the review of the party's election programme, including on tax and said if he had gone into the last election with the tax policy formulated by his predecessor Paddy Ashdown, he would have looked like an "idiot".
"If I had fought this election on a penny on income tax for education I would have looked a bloody idiot and we would have looked a bit out of date," he said.
Mr Kennedy brushed aside an accusation made by a former Liberal Democrat candidate this week that he may be an election liability.
In a question and answer session on Monday, chaired by Simon Kelner, editor-in-chief of The Independent, Nasser Butt, a former candidate in Mole Valley, said he had been diverted during campaigning by the need to defend Mr Kennedy.
Mr Kennedy said the comments did not reflect the general view about his leadership. If they had been he would have "felt the need to say more".
"You have one parliamentary candidate fighting Mole Valley, which I don't remember was a target seat at the election but never mind he obviously felt it was," he said. "If I thought that was indicative to a broad feeling then maybe I would have felt the need to say more."
Leader's ally refuses job as 'minister for Today'
Charles Kennedy has suffered a blow to his authority with the refusal of a close political ally and personal friendto act as public spokesman for the party.
Matthew Taylor, who shaped the party's election manifesto, has turned down an invitation from his party leader to take on an official role as the Liberal Democrats' "minister for the Today programme". Mr Taylor, who was ousted by MPs from his role as chairman of the parliamentary party earlier this year because he was viewed as too close to Mr Kennedy's leadership, turned down the offer because it would have restricted his freedom to speak freely.
The move will be interpreted as a snub to Mr Kennedy and a move by the MP for Truro to distance himself from the party leadership, and even position himself as a potential leadership contender.
Friends of Mr Taylor say he refused the role because he felt he would have been forced publicly to defend Mr Kennedy's leadership repeatedly in the media.
Mr Taylor denied the move was a snub and said he had refused the official job "after a lot of thought" because it would have restricted his freedom to feed into the policy debate. He said he had told Mr Kennedy of his decisionlast week. "I told Charles this is what I decided to do. I asked for his support and he gave it to me," he said.Reuse content