First some basics:
If you're familiar with the Remington 870, the 1100 is very similar. Both guns employ a similar interchangeable barrel with a tang. When at battery and while firing, the bolt is held in place by an internal lock block that engages a latch cut-out in the barrel tang. In both guns, after firing, the rearward motion of the action bar serves to first drop the lock block free of the latch cut and then move the bolt rearward. When the bolt is fully rearward, in the 1100 it's returned forward by the action spring, and in the 870 by manually advancing the forearm.
To unload an unfired shell, with the 870 you release the action bar lock and then retard the forearm, then the bolt opens and ejects the live round. With the 1100 the process is similar, but there is no action bar lock and the forearm isn't linked to the action bars. Instead, the 1100 is provided with an operating handle which protrudes from the bolt. Retarding the operating handle releases the lock block, then the bolt opens, ejects the live round and locks back in the rear position.
Should you have a repeat of your problem in the field, next time instead of firing off the rounds to clear the gun try this: With the safety on and the muzzle continually pointed in a safe direction, remove the magazine cap and for-end assembly, replace the cap. Now try to use the action bar cylinder as you would the 870's slide (using gloves couldn't hurt). This may act to drop the lock block and free the bolt to be retarded to the rear lock position. With the live round cleared, remove the cap and barrel.
Since the gun fires and cycles, we know the action bar, lock block and bolt are functioning. The operating handle isn't linking to the action bar. First, is the operating handle properly installed -- did you feel it click, or is it just shoved into the slot?
If reinstalling the handle doesn't help, disassemble your 1100 and compare its parts with the parts list diagram. I'm pretty sure you'll find a worn, broken or missing part. Pay close attention to the operating handle plunger assembly. This area can be troublesome and was revised/improved in the 11-87.