" indicates the position of the cursor when you last exited a buffer. Using that...
Continue to use your cross-file marks but immediately follow with
`" to restore the column as well as the line):
To make it a bit more convenient you can create a mapping that does the same thing:
:nnoremap <leader>'X 'X'"
If you put this in a function so you can do the double-jump with any mark:
exec "norm! '" . a:mark . "'\""
To run it
:call Jump2('X'). A user command and/or mapping can shorten things.
Taking things further, see
This autocommand jumps to the last known position in a file
just after opening it, if the '" mark is set: >
:au BufReadPost * if line("'\"") > 1 && line("'\"") <= line("$") | exe "normal! g`\"" | endif
If you instead used
BufWinEnter then each time you switch to a different buffer you'll start at the last known position. That may be more than you had in mind but you can easily add logic to the
autocommand so it only occurs under certain conditions.
That's just a few things that come to mind. Really, "the possibilities are limited only by your imagination!" ;)
" might not always be set...but from a practical standpoint it usually will be.