Mr Goldsmith, a former director-general of the Institute of Directors, is to leave in spite of calling the prospect of his resignation "an absurd proposition" at the end of last year.
He will be replaced as chairman by Robert Norbury, a former chairman of investment banking at NatWest Markets, the defunct investment arm of the high street bank.
Tom Walker, company secretary and a director for seven years, will retire along with two non-executive directors, Kevin Morley and Ian Steven. The changes will take place at the company's next agm.
Flying Flowers saw its market value plummet last summer as its shares fell from 600p to 150p. The plunge followed two profits warnings in the space of two months. The shares rose yesterday from 215p to 217.5p
Behind the warnings were tumbling response rates to newspaper adverts for Gardening Direct, the key driver of Flying Flowers' growth. Flying Flowers admitted it also failed to manage the business properly during the summer months.
Paul Fraser, one of the biggest shareholders in the group, saw the paper value of his stake fall by pounds 9.5m. After the two profits warnings, he joined the company to share the chief executive post with the company's founder, Tim Dunningham.