Ideally, I would replace it with a section of wall as I don't think it will look right having a section of fence there.
I haven't got any concerns about neighbours around me making any complaints as it wouldn't actually affect any of them. My concern was with the planning permission side of things.
Is it morally wrong to crack on and alter the wall, and then worry about applying for retrospective planning later? ?