Is it possible to change that somehow? Either to make Vim go one symbol further in normal mode, or one symbol less in visual mode?
You can make
$ go one symbol further in normal mode by enabling setting
virtualedit=onemore and remapping
:set virtualedit=onemore :nnoremap $ g$
As @statox mentions in the comments
You can also make $ go one symbol less in visual mode with the following mapping:
:vnoremap $ $h
As far as I know, there's no built-in way to stop visual mode from being inclusive and normal mode from being exclusive.
One could probably remap everything to work like that but that sounds like a bad idea to me.