A modernist mansion complete with sauna, pool and waterfall at a Herefordshire beauty spot will not now be built after a government inspector threw out the planning bid.