Before you shoot a shotgun the gun and the shot shell pellets are both at rest. After you shoot the pellets/wad/powder goes out the barrel and the gun goes in the other direction. The key here is momentum which is mass times velocity. So although the velocity of the pellets is large, their mass is small compared to the mass of the gun. The key equation here is:
mgunvgun=mshotvshot + mwadvwad + mpowdervpowder
Where the left side is the mass and velocity of the gun and the right is the mass and velocity of the shot pellets, wad and powder gases. The velocity of the shot and wad is given to us by the manufacturers. The velocity of the powder gases is a little more complicated. There are a number of values for the velocity of the gases. The most commonly used one is 4,700 ft/s. Another estimate is 1.7 times the muzzle velocity of the shot. For this calculator I am using 4,700 ft/s. The difference here is one of a few percent, not very significant.
Since we know the mass/weight of the shot, the mass of the powder, the muzzle velocity and the mass/weight of the gun we can calculate the recoil velocity of the gun.
Once we have the recoil velocity of the shotgun we can calculate the kinetic energy of the gun. This is the "free" recoil energy. The recoil energy is calculated as:
Energy = 1/2 mgunvgun2