Just put in planning for another house . Solves all the problems faffing , trying to find some middle ground for you and your neighbor. He will then hate you and sell up . You will then have a new neighbor who will love you . Once you start the new build and deviate slightly ( few cm too long or high ) - they will hate you also .
But you have a nice new house .