They're stuck...
Actually, just fitted tight.
Put the screw back in the grips, but only thread it into the 'other' side about a turn or two. Gently tap on the head of the screw with a bit of wood or plastic (so not to bugger up screw head and slot). That should push the 'other' side panel off the frame. Then, take a pencil or something equally non-marring and tap the first side panel off from the inside, gently.
Replace gently, fitting the holes in the grip panels to the pins on the frame and squeeze with both hands to seat, then replace screw. Tighten snug, but carefully, you can pull the little metal bit (escutcheons) right through the grip panel if you're too enthusiastic.
