While selecting in visual mode I often accidentally hit H or L instead of h or l due to having used Shift for some previous command (like e.g. V to start selection by line) and not yet released it when trying to move (because hands are not in ideal synchronization).
In normal mode this is not a problem, because a remedy is quick: just hit Ctrl+o and continue. But in visual mode this "go back" command doesn't appear to work. Currently I have to go to normal mode and start selecting all over again, which is an unproductive fix.
So, how can I go back in visual mode? Is there another mapping for this command?