I have noticed the same sort of handling in my VX, but I relate mine to the larger, wider tire. More road coverage means I am grabbing more surface and I tend to carry in many road imperfections. I have been desensitised to this though since I have almost always had automobiles with larger tires and have gotten used to it. Fighting the wheel a bit just sorta seems to go with the territory of trying to tame the beast you are piloting.
Good luck finding your problem.
Gary Noonan
'01 S/C VX / '18 Forester XT