The Scotland manager has been annoyed that attention has shifted to his own position when he is confident that his team can reverse the result from the first leg.
Despite the 2-0 defeat last Saturday, Brown said the Scots matched Kevin Keegan's England in every department and would not hear of any defeatist talk at, as he put it, "half-time".
The wretched luck that seems to have plagued him in the last few weeks continued however, with the news that Everton midfielder Don Hutchison may miss the second leg at Wembley with a calf strain.
Brown feels it is high time Scotland enjoyed a couple of lucky breaks and, after the injury to Celtic playmaker Paul Lambert and Billy Dodds' shot hitting the bar against the English, many agree with him - at least north of the border.
So he is calling on everyone with Scotland's best interests at heart to forget about any side issues and, instead, to unite in support in the common cause of beating England.
Brown said: "We are determined to recover the situation and one thing I can guarantee is whoever plays on Wednesday will give their all. In any sport, be it golf, snooker or football, leads are lost and the least we need is two goals.
"We have got to look to get them, but at the same time we must be patient - look at Manchester United in last year's Champions' League final. We have to be patient even if it takes us until then [the dying minutes]."
After the setback of Paul Scholes' two goals at Hampden, there are those questioning Brown's future as national team manager and they are not just disgruntled Tartan Army foot soldiers.
But Brown has hit back and said: "I would hope for support from Scotland's media, everything should be geared to supporting the team and my future is irrelevant as far as that is concerned."
On the injury front, it is not all bad news for the Scots as Gary McSwegan, who missed the first leg with an ankle injury, is in the frame for a recall on Wednesday night.
Brown feels the Wembley factor is part of the reason he is confident his shot-shy team can prove everyone wrong when the oldest international rivals do battle for the final time this century.
He added: "I don't believe for one minute the Wembley crowd will allow England to sit back and try and hold the lead they already have. They will want to be entertained and they will be desperate to see their team beat Scotland again.
"We have seen Tony Adams and Martin Keown struggle at Wembley with Arsenal in the Champions' League and that's because there is more space to be exploited by the opposition than at Highbury.
"During my six years as Scotland manager, my teams have never lost two competitive games on the trot and I can't think of many other managers who could say that."