Your solicitors gave you good advice. Party Wall award will happen anyway, as will planning consent. I don't know how the freeholder consent works when you have a share in it, but they are often "not to be unreasonably withheld" or some such.
Try to let it wash over you and do not deal with the neighbours directly or react to their provocations. Avoid words like "lies", "hate" and "nasty" as they are not helpful.