Hay, a pounds 330,000 signing from Celtic, continued his extraordinary goalscoring run with a 23rd-minute header from a cross by Mark Walters.
Portsmouth, however, did their best to ease the pressure on their manager, Terry Fenwick, who was booed by spectators calling for his resignation.
Mildenhall, called in as a last-minute replacement for the injured Fraser Digby, was playing for the first time in his preferred role as goalkeeper after a brief appearance last season as an outfield player.
He was in action after three minutes, saving bravely at the feet of Paul Hall and he also denied the Australian international Jon Aloisi.
Mildenhall suffered what appeared to be a nasty injury in a collision with Robbie Pethick which left the Swindon goalkeeper doubled up in pain, holding his groin. But he managed to struggle back to his feet to thwart Pompey time and again in the second half.
In the 46th minute he saved brilliantly, diving full stretch to his left to deny Fitzroy Simpson, and he prevented a certain goal in the 55th minute, beating out a Sammy Igoe shot. Mildenhall also denied Pethick, tipping a long-range shot over his bar to keep Swindon in the game and allow Steve McMahon's side to move above Nottingham Forest.
Hay almost grabbed a second goal late in the game when he turned Andy Thomson but fired his left foot shot just wide with Alan Knight beaten.
Fans rounded on Fenwick in the last 20 minutes to display clear unrest at Fratton Park where the opening of a new pounds 2.2m all-seater stand was marred by another inept display.
Thomson was booked in the 47th minute for a foul and even the introduction of the substitutes John Durnin and Alan McLoughlin could not improve Portsmouth.
Portsmouth: Knight, Pethick, Thomson, Awford, Whitbread, Perrett, Hall, Simpson, Aloisi, Foster, Igoe. Substitutes: Durnin, Hillier, McLoughlin.
Swindon: Mildenhall, Borrows, Culverhouse, Bullock, Taylor, Casper, Watson, Walters, Hay, Allison, Gooden. Substitutes: Cowe, Hulbert, Elkins.
Referee: P S Danson (Leicester).Reuse content