I've had this happen before, but I always attributed it to the varying road conditions and the wide tires on the VX (Nitto Terra Grapplers). If the road crowns to the right then the VX usually pulls right, or if there are some wallows in the road from heavy truck traffic the VX will get a little squirrely as the wide tires bite on different parts of the wallows. I've had other vehicles with wide tires and they tend to squirrel all over the road. I know it sounds goofy, but does the VX pull differently on the same patch of road?
It could be a caliper hanging up, but I would think that there is enough pressure in the brake lines to overcome that unless it's something drastic (like a siezed caliper).
"The major difference between a thing that might go wrong and a thing that cannot possibly go wrong is that when a thing that cannot possibly go wrong goes wrong, it usually turns out to be impossible to get at or repair."
-Douglas Adams, Mostly Harmless