You're talkin twice the amount of lead flyin down range compared to a 2-3/4 shell, remember. 5 out of 18 is a terrible percentage.
I'm not sure what difference the percentage makes. 5 hits out of 18 pellets or 5 hits out of 9 pellets is still 5 hits, isn't it? Maybe I'm interpreting the comment incorrectly.
Getting to the original question, I don't know that I understand that completely either, but I'll throw some random details out there and maybe some of it will apply...
A single 00 buckshot pellet weighs around 60 grains, has a diameter of about .330" and a ballistic coefficient around .045.
Based on Federal Premium 2-3/4" and 3" 00 buckshot, pellets start with a muzzle velocity of 1325 fps. On paper, a single pellet will have approximately 234 foot-pounds of energy at the muzzle, but will already be down to around 127 f.p. at 50 yards.
Federal Premium 3-1/2" 00 pellets start at 1100 fps, resulting in 161 f.p. at the muzzle and roughly 105 f.p. at 50 yards.
Trying to make comparisons between a buckshot pellet and any bullet is sketchy since bullets are shaped and constructed differently. That said, in terms of weight, diameter, and energy (at 50 yards), each 00 pellet is similar to a .32acp bullet, which isn't known for great destructive properties.
Because round projectiles shed speed and energy so quickly, basing maximum range simply on how many pellets hit near the vitals is probably not enough. There may be a gun/choke/shell combination that will put a bunch of pellets in a 10" circle at 75 yards, and maybe they'll hit with a combined 800 f.p., but nothings says any one of them will penetrate deep enough to be effective. On the other hand, at 10 yards, just two on-target pellets may tear up heart and lungs.
From the standpoint of individual pellet penetration, 40 yards
maximum is a common rule-of-thumb I hear, and that's assuming good angles where the pellets don't have to punch through a lot of extra muscle or bone. That said, deer are fairly large in the upper midwest where I live, and the 50 yards alabama mentioned may be suitable elsewhere.
If I really wanted to extend my range with buckshot, I might look into Dixie Tri-Ball shells. Each shell has three 315-grain round balls .600" in diameter with a ballistic coefficient around .085. Starting at 1050 fps in the 3" shell or 1150 fps in the 3.5" shell, each pellet has over 600 f.p. of energy beyond 50 yards.