Check out mythbusters...they had a special about this very subject.
As long as you're firing "UP" not at an angle...it's technically perfectly safe.
They tested a lot of calibers.
I saw that episode. They only recovered pistol bullets--they were not able to recover rifle bullets. In addition they only recovered a limited number of pistol bullets in, at most, two calibers.
A man named Hatcher did some testing for the army many years ago and confirmed that some rifle bullets remained spin stabilized and fell back base first with around 30ft/lbs of energy instead of tumbling which would have reduced the terminal velocity and energy levels. At the time he concluded that was not a lethal level of energy using the military's 60ft/lb threshold for lethality, however people have certainly been killed by projectiles with lower energy levels than that. We get a few deaths from pellet guns each year at energy levels down under 10 ft/lbs.
Hatcher's projectiles were falling between 300fps and 400fps. That's not much compared to the muzzle velocity, but a projectile can compromise the skull at lower velocities than that, particularly the skull of a child.
It's not safe to shoot up in the air, not even shooting straight up. Without testing, it's not possible to know if the particular caliber you're shooting will keep the bullet stabilized and allow them to descend base first at potentially lethal velocities or if the bullets will tumble and be much less likely to do anything other than bruise.
I've shot 410's up in wooded areas...cool sound of buckshot falling through the trees
Shotgun pellets, particularly smaller ones are safer than bullets, but when you get up to the larger buckshot there is, again, some potential for injury. Birdshot is not a problem.