First off, if his personality is such that he's not going to be a mama or daddy's boy then it's just not going to be something he's into. I have a 6 mo girl who is stuck up my butt when we're at home but would much prefer to pay attention to a skittering leaf or a car half a mile away than dad when we're outside.
We have done a few levels of obedience and three months of classes. That's really helped her focus more on me when necessary. It's still a 'trained response', though... I don't think she'll ever naturally put me first on her 'things to pay attention to' list when we're not in the house.
You already identified one of the issues in your original post, too: you guys need alone time! Go for a walk with just your boy. Play games with just your boy. One-on-one will really strengthen the bond. It can hurt to leave other dogs at home, but they won't die without a jaunt to the park.
Side note, having a dog that relaxes when you leave him unaccompanied is actually a fantastic gift. Dogs that freak out, bark, lunge, pant, pace... they're not having a fun time. I'd much prefer a less-affectionate but stable dog than an anxious, borderline-SA dog myself.
And just remember, at the end of the day he does have an enormous amount of affection for you. He's just expressing it in a way that perhaps isn't as easy for you to see.