Is there any doubt whatsoever? Not in Greenwich (or Beverly Hills, for that matter). Nope, sorry, but the land is the thing around here, that is primarily where your value is, particularly if the house was built before, say, 1980. So really, nearly every house in town is considered, er, replaceable.
This is not necessarily a bad thing. If it’s happening in your town, it means people want to live there, badly! Detroit, Newark, most of Vermont? Not so much. Friends of mine bought a vacation home in Vermont 21 years ago for $225,000 and sold it last year for the same amount! I guarantee that house won’t be torn down.
So if you live in Tear-Downville, count your blessings.