I'm 72 and been reloading and casting for close to 50yrs now. Lead and primer residue hasn't killed me yet. How do those people explain that. I suppose they could bother some people. Hell when I was a kid, I chewed the lead paint off my crib and lived to tell about it!
No, it did not kill you, (rapid onset lead poisoning by large caliber handguns will do that), but perhaps it "enabled" some degree of long term cognitive decline, manifesting in your disputing, denying professional research, published facts, and significant changes in public polices, laws and regulations. It is NOT all about YOU. I do not personally care if you pour milk on your used media and eat it for breakfast (enough sugar will fix everything, right). Those at greatest risk, are not YOU (although you are the cause), it is developing children, future occupants of any contaminated building, and the environment we all live in. They stopped putting lead in paints and gasoline for a reason. One of the factors leading to the fall of one of our planets greatest governments to ever exist and govern huge parts of the world (the Roman Empire) was lead used to line their glorious water aqueducts, inadvertently poisoning everyone. But that didn't kill Julius, Augustus, Nero, or any other Caesars. Metallic mouth tastes is a well-documented symptom of lead poisoning that many Roman era writers mentioned, without understanding the cause.
Detection of lead residue in a building can trigger professional cleaning or abatement of the hazard (tearing down and remove all structures by the county). Many buildings with asbestos are abandoned or torn down; my point on asbestos is that remediation can cost more than the property is worth. Farmers Insurance former Home Office on Wilshire has asbestos and they left after 75+ years. The Scottish Rite Masonic Temple down the street met the same fate. I have personally worked in buildings in Los Angeles that get tested annually, to see if they are poisoning workers. Any family looking to buy your building after you are gone would be well served in testing the property to detect contamination, and avoiding purchase. Contamination is Yes or No. If it is yes, how much comes into play.
I do not shoot cast bullets. I cannot at indoor ranges. They do not allow them even with ventilation systems (legal liability). Also, bullets with any exposed lead (hollow points) are also not allowed. You cannot hunt and have bullets with any lead in them in my state. I could at my outdoor range, but why? I will never carry lead bullets and it is simpler to not have another category of bullets across calibers. I once cast heavy 1, 2, + pounds each for fishing sinkers for deep water rock cod, and almost got kicked out of a very nice apartment complex for it. They were NOT happy. I had to scrub my balcony, and then we got heavy rain, so they let it go at that. I believe it is very likely that any home melting of lead will be regulated out of existence in the United States with 20 years. Battery factories are being shut down now in California because they emit lead vapor into residential neighborhoods.
Lead bullets are easy to avoid: do not use them. Lead primer residue is actually very easy to avoid: clean your used brass (I deprime first with gloves on) in hot dishwasher detergent water, and/or tumble wet or dry, and dispose of the waste properly. It is not hard or expensive.
My grandfather smoked non-filter Chesterfields and sipped a whole lot of Bourbon. It didn't kill him either. Not until he was 86, and developed lung cancer that metastasized to his liver and other organs. The cigarettes and bourbon didn't kill him. The cancer did. So, alcohol and cigarettes are safe, perfectly safe.