Lead in it's metallic state is relatively harmless. Lead oxide is very toxic. The lead in paint is lead oxide. The lead we breathed in from burning leaded fuel was lead oxide. The white crust on things like dug up civil war bullets is lead oxide.
Lead like what you get splashed on you at the range, and the black residue you get on your hands from handling lead is metallic lead and not very toxic. The only real danger would be the metallic lead turning into lead oxide very quickly which isn't very likely.
If you work around a great quantity of metallic lead, like a foundry, there is bound to be a bunch of lead oxide lurking around.
I'm not a expert on this matter. If someone knows more about this subject, please chime in.
This is IMPORTANT.
I don't consider myself an expert on lead toxicity, but I did work in environmental and human health toxicology for many years (primarily with nonpolar organic compounds, not lead) and have read a good bit of the scientific literature on lead because of my shooting hobby, much of it in indoor ranges, resulting in my own elevated serum lead concentrations some years back.
You are correct that lead oxide is more acutely toxic than elemental lead, but elemental lead is quite toxic as well, and does readily enter the body, primarily via inhalation and, secondarily, through ingestion. For shooters, the primary route of exposure is inhalation of fine particulate lead that's suspended in the air from lead bullets and also from lead styphnate priming compound - nearly 100% of the lead you inhale enters the bloodstream. This is obviously more of a problem for those of us who shoot in poorly ventilated indoor ranges. Ingestion of lead results primarily from eating, or touching the mouth, with lead-contaminated hands, as might be the case after a shooting or reloading session. There are a lot of numbers floating around in the literature, but something like 40% of the lead you ingest entering the bloodstream is probably about right. The metallic lead on your hands is most certainly potentially toxic via this route of exposure, but elemental lead does not pass through the skin (well, in trace amounts), so, strictly speaking, the lead on your hands does not pose a risk as long as it stays there. Hand-washing after shooting or reloading takes care of the problem.
With regard to the original question about the California Proposition 65 warning on rubber grips, I have no doubt the warning is there for the grips themselves, not because they would become lead-contaminated during use. But, I seriously doubt that touching rubber grips results in a 1 in 100,000 (what we refer to as a 10^-5) lifetime excess cancer risk, and would really like to see the exposure assumptions and risk calculations that resulted in that conclusion.
ETA: I'm guessing that the warning on lead in rubber grips is due to the "reproductive harm" provisions rather than excess cancer risk. There's probably no established "safe" level for reproductive harm, so manufacturers have to list the risk even though it's so low as to be unquantifiable.