Allowing your dog to relieve themselves in other people's yards is rude.
I live in a rural area. I have live stock. If I see a dog on my property (fence or no fence), my first responsibility is to defend my livestock as well as any and all domestic animals in my care. I have the legal right. If that is the chance you are willing to take because you feel you have the right to allow your dog to relieve itself on my property...go for it.
I also live in a rural area and have livestock, and I have actually shot loose dogs on my property (I try to avoid it, but my animals come first--I'm just trying to illustrate that I know where you're coming from). Again, I'm talking about someone walking in a legal public area (on a sidewalk or in the street) and the leashed dog going right at the edge of the property. Male dogs can manage to pee on people's property without ever setting foot on it, you know?
I already stated that if the dog is actually roaming more than an inch or so onto the yard, or just wandering loose, or whatever, then yes, that is a big problem. But I'm honestly surprised that so many people have a problem with a dog relieving itself right at the edge of the property, as long as the feces is cleaned up. And as has been said, where the OP's dog was peeing isn't even technically the homeowner's property in most areas--the homeowner might maintain it, but it belongs to the city.
I mean, if someone asked me to be sure I didn't ever let my dogs relieve themselves at the very edge of the property or in that area between the sidewalk and the street, I would make sure to do so out of courtesy for their wishes. I just think it's an unreasonable expectation to have as a homeowner.
And again, this may be a bit of a culture thing--for the first few years I owned dogs, I was in an apartment about 3 miles from the nearest park. I really had no option but to let my dogs relieve themselves on people's yards, and because there were a lot of apartment dogs in that area, it was pretty much expected. I've even stood and chatted with homeowners as my dog peed on their bushes or whatever and no one ever had a problem with it.