My thinking is along the lines of JohnKSa, that your accuracy or inaccuracy with those particular loads may well be related to bullet-jump. It's possible that some other brand or load of 38 Spec. would work fine in your gun as others have mentioned, but I have a concern/problem with shooting a lot of 38s in my 357s.
It seems that most people consider a 357 to simply be a 38 with more options. Such is not the case IMHO. Not only can accuracy problems arise, firing lots of 38 in a 357 can, over time, score the inside of the chambers to the point that extracting 357 brass can become problematic. You will also notice, especially if you're shooting lots of cheap 38 factory ammo which invariably is loaded with lead bullets, that leading and crud will build up and form a small ring just ahead of the case-mouth in the chambers, and it can be difficult to remove.
If you handload and want to shoot mild loads in your 357, you're all set. Just load 38 loads in 357 brass, add approx .5 grains of powder (give or take, depending on the powder.....less for fast burning, a little more for slower stuff although you'll probably not use slow burning powder in light loads, or shouldn't) to compensate for the higher case volume to 'equalize' the load, and voila. If you want a 38, buy a 38.