Brian Stones was given the pots when he bought Manor Cottage from the owners of Abersoch Manor in north Wales in 1979. They stood on an 18in- wide stone wall, the boundary with the manor.
But in 1989, Emlyn and Valerie Jones bought the manor to run as a retirement home and "from then on the boundary disputes arose", said Lord Justice Aldous.
Mr Stones had been given an "inaccurate" plan of his property by the previous owners and believed his boundary ran through the middle of the wall.
Mr and Mrs Jones proved they owned the whole of the wall and claimed Mr Stones was trespassing by putting anything on it. They took him to court last year and the judge found Mr Stones had trespassed on some counts, ordering him to demolish a fence and block wall. But the judge said the Joneses "acquiesced" over the stone wall by not complaining earlier, and allowed Mr Stones to keep the pots there.
Mr Jones, 60, took the case to the Court of Appeal, which ruled that Mr Stones was trespassing with the pots. Lawyers estimated the cost of the five-day county court and appeal court hearing would be about pounds 40,000. Mr Stones, a sales manager who retired through ill health, has been legally aided since the Joneses appealed.
Mr Jones said later: "I can't see the flower pots from the manor, there is a line of trees in front of them. But it is the principle I was fighting for."
Mr Stones must pay the cost of the county court hearing. The appeal was on legal aid.