Too much too soon! You have had him for 3 days and it started out all wrong to begin with. Can't change that now but you have to understand that he was uprooted from his home and his whole life has changed. Give him time to get used to you guys before you start introducing him to new things, like a random person he doesn't know coming over.
I would definitely have a crate for him in a quiet location that he can go to if he feels the need. I would also just spend time bonding with him! Walks, training, play, treats, etc. Don't force him into situations he isn't ready for, that will only increase his fearfulness.
Did the previous owners tell you he had fear issues or anything like that?
Commander Charly Titan (GSD)-15 Jan 2010