Police said the 20-year-old Iraqi refugee, who has not been named, assaulted his victim in a toilet cubicle at the Theresienbad swimming pool in Vienna on 2 December last year.
The child reported the rape to a lifeguard and his attacker was arrested at the scene, reportedly telling officers in initial interviews that he was experiencing a “sexual emergency” after not having sex in four months.
In June, he was jailed for a minimum of six years for rape and aggravated sexual abuse of a minor, and ordered to pay €4,700 (£3,700) compensation to the boy’s family.
But on Thursday, Austria’s Supreme Court overturned the rape conviction and ordered a re-trial on the charge.
While the sexual abuse verdict was “watertight”, the more serious offence requires evidence that the defendant knew their victim did not consent to sex.
Supreme Court judges ruled that the first court should have established whether the attacker thought his victim agreed to a sexual act and intended to act against the boy’s will.
“This intention was not sufficiently established, so the Supreme Court quashed the rape conviction,” Austria’s national ORF broadcaster reported.
The re-trial is not expected to begin until next year, with the defendant remaining in custody.
The unnamed man worked in Iraq as a taxi driver before leaving in 2015 and journeying to Europe and settling in Vienna.
His trip to the swimming pool was said to be part of integration efforts, sparking outrage amid tensions over the refugee crisis in Austria.
Speaking to local media, the victim’s mother revealed her son had been “screaming and crying every night” since the attack and had talked of suicide.
Prosecutors said the boy, known as Goran, suffered serious physical injuries from the rape as well as “profound depression”.
The attack sparked a backlash against migrants in the country, which has since seen support for anti-immigration groups rise and implemented a controversial cap on refugee numbers.
Reports of sexual harassment and attacks by asylum seekers at swimming pools have generated controversy across Europe, sparking a ban on male migrants at one pool in Germany and “vigilante” patrols in Sweden.
Police statistics in Germany showed that sex offences make up a tiny proportion of crimes committed by refugees and migrants in the country, which are mostly related to transport and documents