Slowing down is right.
I think it's also helpful for people in your household -- esp. anyone the dog is afraid of -- to keep a treat pouch clipped to them, stuffed full of very high value yummy morsels (soft meaty treats he really likes). When this person walks by the dog, toss a treat near him without talking to him or looking at him. Let him associate these people with "yummy stuff."
This method has helped us with several foster dogs who had a fear of men, likely due to past abuse. My DH patiently dropped lots and lots of treats and over time, these dogs came to associate him with good stuff, and drew near, and eventually warmed up to him. It takes lots of patience, and calm, quiet energy.
Also, let your stable dogs teach him. They can do more to help rehabilitate a dog like this than you can imagine -- he will see them trusting you and interacting with you. Give them lots of praise and affection, and let him watch. They can help teach him to trust you.