A crew of city workers quietly removed the statue from its place at the centre of a roundabout in Louisville's historic Cherokee Triangle neighbourhood.
According to the Louisville Courier Journal, the city's mayor, Greg Fischer, has called for the statue's removal for years.
"Although John B Castleman made civil contributions to Louisville, he also fought to keep men, women and children bonded in the chains of slavery and touted his role in the Civil War in his autobiography years later," Mr Fischer said last February. "We cannot and should not erase our history, but it is important that art and monuments displayed on public property reflect our values today as a welcoming city."
The statue will be moved to Cave Hill Cemetery, where it will stand at the site of Castleman's grave.
Other Louisville officials supported the statue's removal, including City Councilman Brandon Coan, a Democrat who said he was "thrilled" that the statue was being removed. Mr Coan represents the section of the city where the statue originally stood.
The 15 foot statue was built in 1913.
Castleman's statue is the latest monument to oppression to face criticism or removal since the start of the protests.
Robert W Lee IV, a descendant of Confederate leader Robert E Lee, penned an editorial in the Washington Post calling for the removal of monuments to his distant relative.
A statue of Confederate General Williams Carter Wickham was toppled in Richmond, Virginia on Saturday night, and a statue of segregationist mayor Orville Hubbard was removed from the grounds of the Dearborn Historical Museum in Michigan.
In London, protesters demonstrating against racism toppled a statue of slave trader Edward Colston before tossing it into Bristol Harbour.
Join our new commenting forum
Join thought-provoking conversations, follow other Independent readers and see their replies