Are the 3 that are challenging your dog un-neutered as well? Just interested to know because in my experience neutered dogs love un-neutered dogs and dogs that are entire will fight.
These owners are probably going to keep doing what they like (have their dogs off leash in an on leash area) so it may not be worth talking to them, you would have to judge the situation first.
I have had dogs rush at me with both mine on lead, my female will always submit/retreat, my male will not. He would fight to the end and like you I do not want to put him in this position.
What I do, is always carry a pocket of treats and when a dog rushes us I throw as many as can fit in my hand at it, I have found the treats broke their focus and they stopped in their tracks to eat the treats, I quickly walk off.
I have also put myself in between the dog and mine, and growled at the dog - it stopped and just stood there.
These methods are good for a dog not intend on doing damage, however if nothing you do will stop a dog and this dog is determine to get to yours, it is essential you drop your dogs lead, if you hold the lead and pull your dog this puts him in a disadvantage and he may panic.
Also, this is easier said than to do, but it is important you stay calm and don't panic.