The loss in Karnataka, the only south Indian state to be ruled by the BJP, comes as a huge setback for Mr Modi ahead of the general elections next year.
At the time of writing on Saturday, the Congress party won 100 seats, whereas the BJP was at 45.
The third party in the fray – Janata Dal (Secular) – won 16 seats. The counting of votes will continue till the end of the day.
Home to India's Silicon Valley city of Bengaluru, Karnataka voted on Wednesday to elect a state government in the 224-seat Assembly. A party needs 113 seats to win a simple majority.
The elections saw a lengthy and aggressive campaign by the three parties over local and federal issues such as regional caste identities, reservations, and hardline groups.
The ruling BJP tried to bank on Mr Modi's popularity, who launched a blitzkrieg across the state, holding miles-long roadshows over the past several weeks.
BJP had initially promised to spur development and wooed voters with social welfare measures, however, in the months leading up to the polls, the party's stance inclined toward Hindu nationalism.
The Congress courted controversy in the run-up to the election by vowing to ban the Bajrang Dal – a hardline Hindu group that the party equated with the outlawed Muslim group Popular Front of India (PFI).
The party built its campaign by targetting the BJP over rising inflation, allegations of corruption and poor infrastructure development, while promising electricity subsidies and rations to poor families.
Communal polarisation between Hindus and Muslims deepened as the then BJP government in 2022 banned Muslim girls from wearing Islamic headscarves to school. At the time of writing, BC Nagesh, the state Karnataka education minister, was trailing by over 17,000 votes.
According to the 2011 census, 84 per cent of Hindus made up Karnataka’s 65 million population, whereas almost 13 per cent were Muslims and less than 2 per cent Christians.
"The shop of hatred has been shut down," Congress party leader Rahul Gandhi told reporters in New Delhi, referring to the BJP. "I am happy we contested the Karnataka polls without using hate, bad language. We fought the polls with love."
This was the first major electoral face-off between BJP and Congress since Mr Gandhi lost his parliament seat after being convicted of defamation in March.
Earlier in December the Congress unseated Mr Modi’s party in the small northern state of Himachal Pradesh.
Experts believe that the rise in the vote share for Congress is because of the local leaders such as DK Shivakumar along with Mr Gandhi’s 136-day-long journey on foot through the length of the country.
Mr Gandhi, the son of former prime minister Rajeev Gandhi and the scion of the dynastic Congress party, set on a 3,500km walking tour of Indian cities, towns and villages to rejuvenate the party and win people’s support.
The election is the first of five crucial state polls this year that are seen as setting the tone for parliamentary elections due in April and May 2024 where Mr Modi will seek to extend his prime ministership for a third consecutive term.
Join our commenting forum
Join thought-provoking conversations, follow other Independent readers and see their replies