The longer barrel won't help. With any barrel length over about 10", the .308 load that is fastest with one powder and bullet combination will also be fastest in any other barrel. As bullets get heavier, that minimum length to give the same load maximum velocity gets shorter. With 180 grains anything much over about 3.5" will show maximum velocity with the same load. It's what has happened by the time the pressure has finished going through its peak range that matters to burn efficiency, so you only need enough for that plus a little to let the higher maintained pressure of a slow powder make up for not peaking sooner.
I infer from your 168 grain bullet experience that the .308 expansion ratio in combination with bullet weights much under 180 grains make for too little confinement for RL17 to burn consistently. A magnum primer might well help, if you didn't use that already.
It's hard to imagine 178 grains wouldn't be close enough to 180 to work similarly.
That said, I personally avoid going with the slowest powder I find listed with a bullet. Too often there are secondary pressures down the tube when the powder is really too slow for best work with the chambering. This can ring a barrel over time.