If it’s a detached property with a couple of metre gap between you and the neighbours, 100mm up or down wouldn’t even notice, so I don’t think they care that much.
i would set it up as high as you dare, it will help with drainage runs and run off from the field behind, and your retaining wall will be shorter.