I don't think you're making a mistake.
I use both, for my dogs, and many times both for other dogs that I train or foster as well. Sometimes I just use one or the other depending on the dog. I tend to use the clicker to get precision and understanding in a behavior at the beginning, then use my "yes" to mark it as time goes on.
I also use No Reward Markers with my voice. Mine is "nope".
Danielle, owner of...
Fraggle Rock Collars & ...
Fraggle Vom Richcreek CGC CA
Red Dog's Reconnaissance Man CGC CL1-R CL1-H CL1-F CL1-S USJ
General Patton Vom Winter Storm