I've knocked down plenty of people without being knocked down myself. Just because I generate enough force with my hands, my fist or my shoulder to knock another to the ground doesn't mean that I wind up on the ground myself. I do realize that there is an actual Newtons law of conservation of energy (I think that's the one that applies here) but I've never understood why people cite it as evidence against knockdown power when it is clearly possible to knock someone down without getting knocked down yourself.
Inertia.
You are an object in motion. When you hit something stationary, a portion of your inertia is imparted into the person who you hit. If that person weighs less than you and you have more inertia, they will move and you will slow down.
If you are a 105lb girl and you run into a 300 lb linebacker, even if he isn't expecting it, your inertia won't be enough to overcome his inertia and you will bounce off.
This can be demonstrated (to great amusement) by dropping a basketball and a tennis ball together with the tennis ball on top. When they both stop because they hit the floor and bounce off, the basketball will impart inertia to the tennis ball. The tennis ball will shoot back to the ceiling even though it was only dropped to the floor from waist high. Because it picked up some of the basket balls inertia.
You are hitting with more inertia than a bullet is. Bullets themselves don't have much inertia, they kill based on penetration. (That is why "bullet proof" vests work, you can't penetrate the fibers)
The bullet can't have more force going forward than it did pushing backwards against your hand. If it had enough energy to take a person off their feet, it would take the shooter off their feet as well.
"Knockdown" in both hand to hand combat and ballistics is more of a physiological effect, than a physical effect. Someone punching you in the face may cause you to drop because your central nervous system was shocked by a "knockout punch" or because you torqued the person over because the force of the blow cause them to lose their balance.
There is alot more too this, but this is the best I can do without a chalkboard.