If you use a vibratory tumbler, you are OK. The ammo has already been through a vibratory tumbler twice - once at the factory immediately prior to packaging, and once enroute to the retail outlet. The second vibratory tumbler is huge, and most people call it a truck.
Modern propellents are essentially a type of plastic. In fact nitrocellulose, one of the first mass produced plastics which also happens to be the main ingredient in modern propellants, was used in billiard balls, collars, and early movie film at one time. You don't expect plastics to be modified by vibrations when they are not under stress.
The concern for people who think tumbling loaded ammo is a bad idea is that they think the powder granules will break down into fine dust, changing the burning characteristics of the propellent. Several interested people have performed experiments with different cartridges, posted their findings on the internet, and there seems to be no cause to worry. I believe Geoff Beneze tumbled a few rounds for 2 weeks straight, and the propellant appeared to be unchanged.
I do believe you could alter the physical structure of modern propellents in a tumbler, but the mechanics of it would require some sort of media that is harder than the powder and you would probably need a larger volumetric chamber than a cartridge case.
So, in the end, tumbling to clean the ammo is OK.