Well, I've whittled down the scroll jumping issue. It's only when alt-tabbing, so it's specific to gnome-shell I suppose.
It seems like scroll events on the active window are accumulated while it is active, and like... replayed into the next active window on the first scroll event encountered. This causes a huge scroll jump in scrollable panels. But it doesn't happen if you just click into another window, only when alt-tabbing.