All these smart answers are well and good but he's your neighbour, even if he's a S**T....pardon the pun....
Neighbourhood disputes can get really nasty. Do you want to live with such a cloud over your life all the time, just for the sake of a debatable principle.
I'd move them away and enjoy your bees in peace and quiet.
Anybody with bees in their urban/suburban garden should have a get out apiary.
I agree. Life is too short. Plus having an out apiary can be enjoyable and gets you away from the house during the summer.