Are you planning to wear a whole duty rig, or just the holster dropped down off the belt? I think if you wear a duty belt that is where the holster belongs. If it is a backup or you are more concerned with comfort in a low risk area the drop rig is probably fine.
Try this, walk backwards at an easy pace and try to draw your weapon without breaking stride. If you can do this you are more coordinated than me. You could do the same test walking forward, but I see a more likely scenario as me backing away from a person wielding a contact weapon like a knife.