Wow, that is really annoying. What did you tell him when he asked you not to walk by his house? He really has a lot of nerve to make that request, but on the other hand maybe it is good to know that he doesn't trust his dog. I live in suburbia and I hate invisible fence. The dogs behind it always seem to charge my leashed dogs. My dogs are always confused and scared. So far none of these dogs has broken through the "fence" but a couple of them are unfriendly characters so I am afraid it would be ugly if they did. I tend to avoid houses with invisible fence, but you don't really have that luxury.
Do you know if his dog is friendly? If the dog is known to be friendly I think I would respectfully tell him that you have no other choice but to walk his direction and perhaps he should get a real fence or tie his dog up. If the dog isn't friendly then it is a bit of a problem. You could call animal control but that is likely to end any relationship you have with your neighbor.