Hi, Jeff,
Keyholing is when a bullet enters the target sideways or partially sideways and the elongated hole looks like a keyhole, hence the name. Many bullet holes are not clean, but they are round. The only ones that are clean are those made by target bullets (called "wadcutters" because the holes look like those left when cutting wads for reloading shot shells).
Keyholing is the result of bullet instability. At very short range, some bullets are not stable and will keyhole while they will stabilize farther out and leave round holes.
The most common causes of keyholing, though, are worn or no rifling, bullets unsuitable for the rifling twist, wrong size bullet (.32 in a .38 gun), and gun muzzle damage. There is no way I know of that the shooter could cause keyholing or stop it except to make sure none of the above apply. With a good barrel and proper ammunition, no gun should keyhole, though some target pistols with wadcutter bullets will keyhole consistently. Match officials sometimes ban a gun from a match because the keyholing (and bigger hole size) gives the user an unfair advantage.
Some military bullets have been made deliberately unstable so they will keyhole in a human target, making a bigger and more lethal wound.
Jim