How do you do it? Sit/crouch down first, then give command and then put the hand down? Timing is everything.
With mine I always had some space between me and him, like 0.5-1 step to begin with, and slowly extending distance over time and changing it depending on what we are training.
In your case since you already have some response to down command, try this, but try to get the timing perfect. Make him sit, praise. Come/step away to be within 1 step (should be ok) within him and while standing, make your right hand gesture 1-2 inches down with the palm parallel to the floor facing down around your waste area (so it's not too high but so you don't kneel just yet). You may also try in the beginning to hold a treat in right hand while doing it.
They are amazing readers of body language because that's one of the main ways they communicate with each other. Show him clearly what you want him to do. Voice is supplementary that becomes primary or equal to the body language the more you use it.
So you give command down together with the hand gesture, just a small/short movement is enough, they will see it), and only then you kneel and touch treat to the ground before him. As he is laying down, praise and release. Rinse and repeat starting with Sit and ending with release.
Also, give the command once and don't talk until he follows it. Help with other means like hand gestures, sounds/noises and etc.
Once he reliably goes down after each command, you could notice that he starts moving down right after the hand gesture/command. If he does that every time, then skip touching the treat to the ground and give it straight to him as soon as he gets down kind of thing. They also can count
) If he went down on his own the first time without touching treat to the ground, give him 2-3 treats, show him that that's exactly what you want him to do.
When I do it with mine, he usually touches either my hand or leg with his nose really quick, I think he lets me know that he really appreciates the the increased reward. Often he remembers what he did to earn that. More complicated or brand new stuff it may take 2-3 increased rewards for him to remember the desired outcome.
Also try sneaking it in throughout the day. Try down with his hopefully favorite toy or something he is really excited about. For mine it was marrow bones in the beginning. He would go nuts for them as soon as I gestured down with voice command, he would drop down like a sack because the reward is really big.
Lots of text
Hopefully some of it helps