I'm no expert in ballistics, but I have read a bit from those who do know the subject - however I'm getting well beyond my very limited knowledge here.
The thing that comes to mind is that as the bore decreases, the pressure rises. The shot will therefore be more prone to deformation and damage, fusion etc, Also, the longer the column, the greater the risk of deformation etc.
I entirely agree that there is no hard line beyond which patterns suddenly go wrong, but I do think that the small bores and long shot columns don't make for good patterns