The popular Scot, who will continue to serve as MSP for Edinburgh Central, has been credited with reviving Tory fortunes north of the border but repeatedly clashed with Mr Johnson over Brexit and the direction of the party.
In her resignation letter, Ms Davidson said it was the “privilege of my life” to hold the post but the prospect of spending so much time away from her family, including her young son Finn, filled her with “dread”.
She pointedly failed to refer to Mr Johnson’s leadership in her resignation speech in Edinburgh, adding that she had not hidden the “conflict” she had felt over Brexit.
Her decision will come as a major blow to the Scottish Conservatives, who have become the main party of opposition at Holyrood under her eight-year tenure.
Jackson Carlaw has been appointed interim leader of the party in Scotland pending a leadership election. The MSP held the position during Ruth Davidson's recent maternity leave.
Mr Carlaw, who became an MSP in 2007, is the current Scottish Conservative depute leader. But having first stood for election in 1982, it is uncertain if he would want to take on the leadership role on a permanent basis.
In her letter, Ms Davidson said: “Inevitably, much has changed over the years of my leadership – both personally and in the wider political context.
“While I have not hidden the conflict I have felt over Brexit, I have attempted to chart a course for our party which recognises and respects the referendum result, while seeking to maximise opportunities and mitigate risks for key Scottish businesses and sectors.”
Ms Davidson, who was once touted as a possible leader of the UK party, cited the birth of her son Finn with her partner Jen as the biggest change in her life.
She went on: “However, as I look to the future, I see the Scottish election due in 2021 and a credible threat from our opponents to force a general election before then.
“Having led our party through seven national elections and two referenda, I know the efforts, hours and travel required to fight such campaigns successfully.
“I have to be honest that where the idea of getting on the road to fight two elections in 20 months would once have fired me up, the threat of spending hundreds of hours away from my home and family now fills me with dread. That is no way to lead.”
In a parting message to Mr Johnson, the Tory MSP told him to “get us a deal” with the EU and urged MPs to vote for an agreement in the Commons.
She said Mr Johnson had “categorically assured” her that he was fighting for a deal and his efforts at the G7 summit in Biarritz had “opened the door a crack”.
Speculation about her departure emerged on Wednesday night, after she failed to comment during a day of high drama in Westminster, when Mr Johnson asked the Queen to allow him to suspend parliament for five weeks.
The timing of her decision prompted questions about her strained relationship with the prime minister, whom she clashed with during the Brexit referendum. She also backed several other candidates during the Conservative leadership contest.
The prime minister paid tribute to Ms Davidson, thanking her for her “wonderful service over the last eight years”.
“She has dedicated so much time and energy to the role and has been instrumental in the revival of our electoral fortunes in Scotland,” he said.
“She should take immeasurable pride in the pivotal role she played during the Scottish independence referendum campaign, where her passionate support for our union was crucial to the result.”
He added: “I will miss the incredible leadership she has given to our party in Scotland, but I am delighted that she will continue to use her enthusiasm, passion and dedication to make the case for Scotland’s place in the United Kingdom from the back benches and beyond.
Theresa May also tweeted her praise for Ms Davidson, adding: “Enjoy your well-deserved family time with Jen and Finn.”
David Mundell, the former Scotland secretary, said: “I understand and respect Ruth’s decision to stand down for personal reasons. I will miss her greatly.
“As leader she transformed Scottish Tory fortunes and was a passionate voice for the union. Our politics needs more authentic voices like Ruth’s and I hope that can be her legacy.”
Nicola Sturgeon, Scottish first minister, tweeted: “I wish Ruth well for the future. I know well the toll political leadership can take on family life and no one will grudge her more time with her young son.
“There will be opportunity later to discuss the politics of her decision – but for now, I’ll simply send my best wishes.”
Join our commenting forum
Join thought-provoking conversations, follow other Independent readers and see their replies