You won't lose accuracy, what you actually lose is sighting radius, and the ability to aim the gun accurately. (That and a wee bit of velocity.) The result, however, is the same.
The Glock 20 is a fine gun, however, it is a lot of gun. I carry the 21 (same size) and it can really pull on the right side of my pants. Your question, "Is the 10mm THAT much better than the .357?" I'd have to say no. Especially for what you said was the purpose for the gun, for backup. The Glock 20 is a hell of a back up gun! It will be just as heavy as your .44 hunting gun. Why duplicate yourself? (Maybe for more balanced walking? One on each hip?) I did know one guy who would only deer hunt with his Glock 20, and with 15 rounds of 10mm, who the heck needs backup?!!?
It seems that you are pretty well covered in the .357 realm if you have a Colt and two S&Ws. I don't know what models you have, but the SP101 would be a terrific, small .357 to carry for backup or for a concealed carry gun. For .357 loads, I really wouldn't want anything smaller. It's actually more pleasant to shoot with .357 loads than my K-frame snubbies. (I don't even try to shoot .357s in a J-frame.) Since I have both (3" 101 and Glock 21), if I was gonna be toting something through the woods as a backup gun, I'd sure rather carry the SP101 than the Glock.
My favorite woods gun is a N-frame 4" 57, however, I don't carry that and something else for backup, as well.