Alen Hadzic was suspended in June after three women alleged he had committed sexual misconduct against them between 2013 and 2015.
The suspension was overturned by an arbitrator last month, allowing Hadzic to take part in the Games and join the team.
But USA Fencing has reportedly put restrictions in place, including making him stay in a hotel while the rest of the team lives in the Olympic Village.
He was also forced to fly out to Tokyo two days after the rest of the fencing team.
Hadzic, 29, has strongly denied the allegations made against him.
“Frankly, they’re untruths,’’ Hadzic told USA Today.
“They’re just frankly not true.’”
The United States Olympic and Paralympic Committee and USA Fencing say the restrictions are a safety issue.
Hadzic’s lawyer is now asking USOPC to immediately life the restrictions in time for the opening ceremony on Friday.
Attorney Michael Palma told the newspaper that an arbitration hearing is set to take place on Thursday.
Mr Palma says that USA Fencing has known about the allegations “for years” and has never before insisted on restrictions for Hadzic during international competitions.
“Fundamentally, (USA Fencing) wants to hide Mr Hadzic and keep him from participating in the Olympic experience that he has rightfully earned,” his attorneys wrote in the complaint.
“(USA Fencing) states that they are required to segregate Mr Hadzic from the Olympic experience in order to ‘increase parties’ physical and emotional safety throughout an investigation, support a fair and neutral process, and aid in prevention of retaliatory behaviors from all parties.”
Hadzic says he has received a decent welcome from his teammates in Tokyo.
“I didn’t know what the atmosphere would be like until I came here, and then when I actually got to the training facility all of the coaches shook my hand and congratulated me on making the team,” he said.
“All of the fencers that I thought would be afraid to speak with me, all came up to me and said hi. Even the women.”
Join our new commenting forum
Join thought-provoking conversations, follow other Independent readers and see their replies