The 17-year-old had pushed world number one Ashleigh Barty close in a second round Adelaide meeting last week and began well against the unseeded Kostyuk, taking the first set 6-3.
The Ukranian hit back to level proceedings, letting several set points slip at 5-3 in the second set but eventually sealing it 7-5 to force Gauff to a decider.
But Gauff, the third seed, broke in the seventh game of the third to take an eventual 6-3, 5-7, 6-3 win.
It tees up a meeting with Ana Konjuh of Croatia for a place in the semi-final.
Should Gauff beat Konjuh, she could face 2017 US Open finalist Keys in the last four.
Keys showed strong form in dispatching Tereza Martincova (6-1, 6-3), while Brengle was also a convincing victor in straight sets against Sweden’s Rebecca Peterson (6-3, 6-2).
Seeds one and two at the second of two events in the South Australian city were Aryna Sabalenka and Elina Svitolina, who both made first-round exits in a blow to their hopes of Australian Open contention.
The towering Isner slammed 23 aces in a three-set match decided by a triplet of tie-breaks, with Kokkinakis coming from behind to knock out the second seed, 6-7(5) 7-6(5) 7-6(4).
Fellow Australian Aleksandar Vukic will be his next opponent, while Karen Khachanov and Marin Cilic are also through.
The Australian Open begins on Monday 17 January.
Register for free to continue reading
Registration is a free and easy way to support our truly independent journalism
By registering, you will also enjoy limited access to Premium articles, exclusive newsletters, commenting, and virtual events with our leading journalists
Already have an account? sign in
Join our new commenting forum
Join thought-provoking conversations, follow other Independent readers and see their replies