As i said before there are two factors at work here. Quite obviously change in cast moves the position of the eye in relation to the pointing of the gun. Moving the eye is just like moving a rear sight, the shot will shift in the direction of the movement.
The other factor is movement of the gun itself while the shot travels down the bore. The fact it does move "Sideways" is brought out by the fact the bbls are not set parallel, but converging. Figuring that the entire gun is pivoting about the point of stock contact with the shoulder, then moving the butt will cause shot to shift in an opposite direction.
I personally think, but don't have proof, the change of sight is going to be larger than change of gun movement, thus impact is more apt to shift in direction of movement.
It wouldn't be the first time Gough got it Wrong.