Britain's Clare Wood, who had beaten two higher-ranked players in the first two rounds, also went out, losing 7-5, 6-4 to Helena Sukova, of the Czech Republic.
Austin, the 1979 and 1981 United States Open champion and former world No 1, admitted she was outplayed by the 76th-ranked Rottier, who reeled off the last 11 games.
Austin, 30, said she had trouble sleeping after her second-round win over the second-seeded Katerina Maleeva, of Bulgaria, as she was thinking about the numerous congratulatory messages she received from both current players like Michael Chang and past champions like Rosie Casals.
Prior to this week, Austin had played just two matches since 1984, losing both during her 1989 comeback which was cut short by a car accident which required surgery. She had not beaten a top-ranked player for 10 years.