The Norwegian has overseen a poor start to the season culminating in deeply disappointing defeats to rivals Liverpool and Manchester City in back-to-back home games.
Despite widespread criticism of recent results and performances the club have so far opted against pulling the trigger.
By Ferdinand, who was a teammate of Solskjaer's at Old Trafford and has been a vocal supporter of him as manager, thinks enough is enough.
"We started this season with such hope but also the transfer window that we had, the excitement, we were all sitting there going 'this is our time now'," he said on his Five with Vibe podcast.
"Ole was brought in to get us to that point. He was brought in to get the squad intact, get the players in to make us challenge for the league.
"I sit here now and I don't think we can challenge for this league. I look at our team every week wondering what are we gonna do tactically?
"I don't see any philosophy or identity in the Man United way of playing whatever that should be from the management. I sit here looking confused at the team.
"I was always deep down a bit sceptical, could he take us on to be champions? I wasn't fully convinced – I wasn't sure he could do that.
"But the showing with the squad that he accumulated, to the beginning of this season – and what I've seen this season – I just feel it might be about the time now for the baton to be handed over to somebody else who can take us on now.
"And I think Ole would leave now with his head held high because because from when he came in to where he got us at the beginning of the season I think it's been positive."
United's next game is at Watford before the Champions League game at Villarreal is followed by an away trip to Premier League leaders Chelsea.
Register for free to continue reading
Registration is a free and easy way to support our truly independent journalism
By registering, you will also enjoy limited access to Premium articles, exclusive newsletters, commenting, and virtual events with our leading journalists
Already have an account? sign in
Join our new commenting forum
Join thought-provoking conversations, follow other Independent readers and see their replies