What's the board you're using with those stepper drivers? Do you have any smoothing diodes or anything downstream from the drivers to the motors?
Turning acceleration down makes sense that you're seeing longer print times, I can see that you're aiming for a smoother print overall, what's your travel and print speeds like?
Have you tried tuning your Vref for the axis that is layer shifting?