Housetraining issues with small dogs are extremely common. However, it can be done. IMO if you have two unhousetrained dogs that you still want housetrained, it may not be the best time to get a third dog. No one can guarantee that the third dog won't feel it is ok to go in the house, since he can cleraly smell that dogs go in the house all the time. Dogs just don't have an ability to reason in the way you are hoping for. Many dogs, especially young dogs are prone to having housetraining lapses when they are in a new home. You can't expect that because he is housetrained at his current house that he will be housetrained in your house, especially with other dogs who regularly potty in the house. So you are definitely looking at the possibleity of having three unhousetrained dogs. I once talked to a woman who had 6 dogs, all unhousetrained that started in a similar situation. She had one unhousetrained dog, got a second dog and that one was also never fully housetrained. And neither were dog number 3, 4, 5 and 6.
That said, your Chi's can be housetrained but it will take strict training on your part and possibly accepting having an indoor potty area such as a litterbox, depending on how long they'd have to hold it. Your first step is crate training and never, ever allowing them out of your sight just like they were baby puppies. If you follow this guide, your dogs can be housetrained but it may not be quick since you have tolerated them going where ever they please for so long. Errorless Housetraining | Dog Star Daily
Another thing you need to keep in mind is that while plenty of people do it without a problem, there is a danger to your small dogs of having them live with a dog that is so much larger. No matter how good a dog is with small dogs, the size difference alone presents a great risk of accidental injury. I know of a sad situation where a couple Cresteds (weighing maybe 10lbs each) accidentally killed a chihuahua - they were jumping at the door and the Chi got caught up under them andtrampled to death. A wrong step on the part of a GSD could result in a Chi with serious injury or worse. Beyond that, there is always
the risk of predatory drift (something you definitely should read up on before getting a much larger dog). You must never trust a dog not to act like a dog.
I'm not saying toy dogs and large dogs can't live together peacefully but precautions really should be taken to protect the small dogs. The dogs shouldn't be left out in the yard together, shouldn't be left unattended in the house together and play should be limited so that it doesn't ever get too rough. Plenty of people will say "my toy dogs and big dogs are together 24/7, unsupervised in the yard and while were at work, etc and we have never had a problem". Such owners have never had a problem yet
. And owners who have experienced tragedy involving dogs of greatly different sizes often say "they had always been fine together". Just food for thought.