The real answer is that if the finger isn't properly on the trigger (too close to the tip or too close to the knuckle), added pounds of force will add more lateral movement as being too close to the knuckle will cause a pulling motion towards the back of hand and being too close to the fingertip will cause a pushing motion away from the front of the hand (kinda like a slice and a hook in golf).

Even with the finger properly on the trigger, the wrist muscles start working to stabilize the hand and can cause some extra shakiness.

If you're aiming at something many yards away, fractions of degrees translate to many inches.

You can kind of see what I'm talking about if you stretch your arm out and bend your index finger so it's perfect perpendicular to your arm, then poke your finger with your other hand's thumb, relax that thumb, then pull your finger. You'll push your thumb in various directions if it isn't centered. Stiffen your thumb up and do it again while trying to stabilize that trigger hand, and you'll notice your entire hand getting pushed around (and might notice a bit of added shakiness)