They were working underneath the Avonmouth bridge, which carries the M5 over the Avon, near Bristol, when the accident happened just before noon.
Part of the gantry ended up in the Avon and on its banks and a 35ft-long section was left dangling after it was blown by the wind hundreds of yards along the bridge. A team from the Health and Safety Executive went to the scene to conduct an inquiry into the accident.
The men who died were named by Avon and Somerset police as Paul Michael Stewart, 23, from Newcastle; Ronald Hill, 38, from Glasgow; Andrew Rodgers, 40, from Middlesbrough and Jeff Williams, 42, from South Wales.
A spokeswoman for Kvaerner Cleveland Bridge, the company repairing the Avonmouth bridge, said all four were specialist steel workers.
She said three worked for the company, based in Darlington, Co Durham, which is a subsidiary of Kvaerner Construction, and that the fourth was employed by the Scottish specialist steel firm Motherwell.