The protest was called by the leader of a highly influential conservative Islamist group, but is being backed by the two main opposition political parties.
Mr Khan has dismissed the call to quit, but the former cricketer faces the sternest test to his authority since he came to power with a convincing general election last year.
Friday’s Azadi (Freedom) rally was attended by the president of the main Pakistan Muslim League (N) opposition party, Shehbaz Sharif, whose brother Nawaz is Mr Khan’s predecessor as prime minister and currently in custody on corruption charges.
Shehbaz Sharif told supporters that under Mr Khan “there are no jobs, no business, labourers are unemployed and prices are going up”.
He called on the opposition to unite and “rid ourselves of this government and Imran Khan”.
The march began as a caravan of cars and buses from the southern city of Karachi last Sunday, led by Maulana Fazal-ur-Rehman, chief of the radical Islamist Jamiat Ulema-e-Islam (JUI) party. Rehman was due to address the crowd late on Friday.
The opposition says the government is illegitimate and is being propped up by the military, which has ruled Pakistan for about half of its history and sets security and foreign policy.
“We want new free and fair elections” overseen by the election commission “not the army”, said Mian Iftikhar Husain, leader of the Awami National Party.
Security was tightened in the capital in anticipation of Friday’s rally, with Islamabad’s main government and diplomatic sector – just a few miles from the rally site – entirely sealed off by police and army roadblocks.
Schools were closed, public transport suspended and internet services interrupted in several areas, a service provider said.
The army chief, general Qamar Javed Bajwa, urged Mr Khan to handle the protest peacefully and avoid violence by both sides, a military spokesman said.
But the government has warned that it will not tolerate chaos on the streets.
“There is an undeclared martial law here,” said protester Habib ur Rehman, 35, a supporter of Rehman’s JUI party.
Mr Khan won the 2018 election on promises of breaking Pakistan away from its legacy of corruption and plans to pull 100 million people out of poverty.
But an economy in crisis forced his government, like many of its predecessors, to turn to the International Monetary Fund. Pakistan got a $6bn (£4.6bn) bailout in July.
Habib ur Rehman said people were struggling to make ends meet, and he blamed the prime minister. “My family voted for him thinking he’s a new person and looked sincere,” he said. “But he’s failed us.”
Additional reporting by agencies
Join our new commenting forum
Join thought-provoking conversations, follow other Independent readers and see their replies