I guess it all depends on how much work you want to put into it. With his fearful attitude he would require a ton of very very careful socialization and confidence boosters, but if you have the time and energy to pour into it you could probably make it work.
If your other dog hates him however yeah that might not work. Perhaps you could foster and look for a forever home for him and if things actually work out at your place you can keep him?
Thank you for saving him off the street though, I hope someone else (or you
) end up saving him from the shelter