The site plan is not necessarily the final boundary position.
The plan is an architects layout of site for planning permission and to show correct spacing of properties- not legal boundaries. The final boundary position would be created when the builder transfers ownership to the buyer.
If the neighbours boundary is along his gable wall and retaining wall as you say and he appears to have only realised then his issue should be taken up with the builder and his conveyancer, although I doubt he has any legal redress to have the boundary moved only a claim for possible compensation from them.
Where does your boundary line run on your deeds? If the land between your fence and his current legal boundary is 'no mans land' I would remove your fence and claim the land before he does.
Be aware boundary disputes get very dirty and expensive but he would have to initiate a legal claim against you.