We have lived in just the same situation: a Vietnam veteran next door on weed. Lots of kids and anything that was alive in that family, people and animals, was abused. He also came over one day, and yelled at the door and threatend us.
Since it seemed they had planted themselves to stay forever, we moved. It was the best thing we could have done to live in peace.