Bob, you must have some excessive side clearance at the plunger pivot, while this pivot point needs to be able to slide it should not rattle. Can you tighten the pivot screw? Maybe press the plunger slot on the lever to close up some of the gap. Use a wave washer, a thin steel washer that has a wavy shape to stop rattling but still allow some movement. Out of a couple of dozen Colts with the pivoting lever the only loose one I have is a .44 Navy from Pietta and it's not so loose as the rattle.
How tight is the plunger body in the barrel lug? It should be a fairly tight fit, easy to slide in and out but not sloppy.