Analysts said the country was split as the prospects of Palestine and Israel making peace were dimmer than ever.
Centrist Tzipi Livni's Kadima party won the most votes but had little chance of building enough support for a coalition. Right-winger Benjamin Netanyahu can get the support, but analysts said the likely coalition would prove dysfunctional.
"I won," read the headline of the country's biggest newspaper, Yedioth Ahronoth, next to photos of both leaders.
But some commentators said the rival claims showed Israel had lost. "One thing is clear to all Israeli voters," said the paper's Eitan Haber. "The political system is shattered."
President Shimon Peres must decide whether to call on Livni or Netanyahu, who then has 42 days to form a government.
Israeli media said he would have no choice but to tap Netanyahu if rightist parties all back him.
But it would be the first time in Israel's 60-year history that the winner of an election ends up sitting in opposition.
"With God's help I will lead the next government," Netanyahu, 59, told supporters of his Likud party.
Nearly final results gave him 27 seats in the 120-seat Knesset, while Livni's Kadima won 28.
She said she would be prime minister and invited Netanyahu to join a "unity government". But Netanyahu said he would lead the "nationalist camp" in parliament, and control 64 seats.
"Tzipi Livni has only the slightest chance, or none at all, of forming a government under her leadership," said Abraham Diskin, a political scientist at Jerusalem's Hebrew University.
Avigdor Lieberman's far-right, anti-Arab Yisrael Beiteinu party, now third largest, emerged as a potential kingmaker.
"We want a nationalist government. We want a rightist government ... and we are not hiding this," he said. But keeping his options open, he added: "The decision will not be simple."
Lieberman and religious parties in a Netanyahu coalition would oppose any peace moves with the Palestinians, a fact not lost on the people of the occupied West Bank.
"Israelis voted for the right and against peace. We will not see progress in the peace process in the coming years," said Ali Zaidan, an office worker in Ramallah.
Netanyahu would also face demands from Jewish religious parties for a big slice of the budget to fund their studies. If no budget passes in 45 days there must be another election.
The outcome was bad news for US President Barack Obama, who plans to revive peace talks aiming for a two-state accord.
"The Obama administration is going to inherit the worst of both worlds," said former US mediator Aaron David Miller.
"They have already inherited a dysfunctional Palestinian house, made worse by Gaza, and now what they are inheriting is a dysfunctional Israeli house," he said.
The Palestinian side is split between hardline Hamas Islamists in Gaza and Western-backed President Mahmoud Abbas in the West Bank, a leader who seems to be losing support.
"I'll talk with whatever government emerges in Israel," Abbas was quoted as telling Italy's La Repubblica. "The ascent of the Israeli right does not worry us."
"Take for example Netanyahu: in the past he took important steps, he signed two accords with us."
Netanyahu and Livni were both meeting leaders of potential coalition parties on Wednesday, and Peres was also due to start hearing the views of the 12 parties which won seats.
His discussions with Knesset factions could take about a week, but coalition talks might drag on for more than a month.
Ehud Olmert of Kadima, who resigned in September in a corruption scandal but stayed on as caretaker prime minister, will remain in the post until a government is in place.
The election was held in the middle of delicate, indirect talks with Hamas to agree on a durable Gaza ceasefire.
Israel's January war in Gaza cost 1,300 Palestinian lives versus 13 Israelis killed, and had strong public support.
Netanyahu was cruising to victory until Olmert's centre-left coalition, including Livni, launched the three-week offensive.
Livni also led peace talks with Abbas's Palestinian Authority and would try to revive them. Netanyahu is cooler on ceding occupied territory and curbing Jewish settlement.
Comments