If it were a real/jacketed bullet, it would not flake or crumble at all.
It would instead act like (well) a chunk of lead*... which when
scraped by a knife tip leaves a
very bright line that looks just
like polished silver. Unmistakable.
Instead, it just looks like something in the brick/clay that was left in
the mix when the brick was formed/cast
*If it were a frangible (fragmenting) bullet, it would just
leave a pock mark in the brick, and disintigate.
* If it were just the copper jacket left in the brick, scraping it with a
knife would again leave a very bright/polished copper line