"We are clearly going to file a complaint against him," Gilbert Collard, Virenque's lawyer, said on French television.
"Willy Voet is the accused who has become the accuser; the pyromaniac who the firemen want; the thief who cries robbery," he said.
Voet accused Virenque, one of France's top cyclists, of taking the same substances as other members of the Festina team that was expelled from the Tour de France.
"How dare Richard say to me he doesn't take drugs," Voet said in an interview published on Tuesday by the newspaper Le Parisien. "He takes the same products as the other racers - no more, no less."
The Festina team was kicked out of this year's Tour de France after admitting to the systematic use of the banned substance erythropoietin. But Virenque has consistently denied using the drug and has urged authorities not to take disciplinary action against him.
Voet said only three members of the nine-man Festina team, Christophe Bassons, Patrice Halgand and Laurent Lefevre, had not taken the drugs.
He also accused Festina's team doctor, Eric Ryckaert, of administering the drug-taking, claiming that if he was not available the cyclists would inject themselves.
Voet is one of three Festina team officials currently under formal investigation after the Tour de France drug scandal. The others are Ryckaert and the team director, Bruno Roussel.
On Monday the French Cycling Federation prevented Festina from racing in October's world championships.