It turns out to be nothing of the sort.
Abbots Lodge, set just across the green from Ledbury church of St Michael's and All Angels, is a low, long building, with touches of Victorian additions, built on earlier foundations. From the early 1800s through to 1959, when the Church sold it, it was the vicar's residence.
Next to it is Church House, which is thought to have been the original vicarage. It is the true age of Abbots Lodge that exercises the detectives in this programme.
The owners, David and Ann Tombs, both question the legend of a tithe barn; to their minds the house has too much of a "lived-in" feel to it.
The interior features give little away, but it is the attic, as so often, that yields the house's most surprising secret. Timbers there confirm the house is, in fact, much older.
Mac Dowdy discovers an early king post in the loft, coming down from the apex of the roof, which he dates back to the thirteenth century. The timbers also show the house was originally two separate dwellings, built side by side.
Using a six-inch Ordnance Survey map, David Austin surmises that Abbots Lodge had been the true vicarage. But hard evidence is still lacking.
After a thorough trawl through local archives, the trio are able to piece together the true history of the house. A major bonus is that the house had been owned by the Church; in this case, the records stretch back as far as the thirteenth century.
Hereford Cathedral's Red Book, a survey of Ledbury borough commissioned in 1288, shows that the building had been two dwellings. "Roger the vicar" held half a burgage - the basic unit of tenure in a medieval town - "Richard the Deacon" another quarter.
By 1607, it had become one full burgage, suggesting the two houses were amalgamated shortly after the Reformation.
The archives prove Church House had never been owned by the Church, and had only been a vicarage for a few years at the end of the sixteenth century.
Just as interesting is the light the archives throw on the lives of the previous occupants - from John de Beverley, who ministered to the sick and dying during the Black Death, only to succumb himself, to the Reverend John Jackson, in the nineteenth century, who seems to have fathered the baby of a young servant girl.
* Tuesday, 8pm, BBC2