You state that your current workflow is "inaccurate" and "less usable". You can solve the former (but arguably not the latter) by replacing your current
j mapping with:
nnoremap <expr> j v:count == 0 ? 'gj' : "\<Esc>".v:count.'j'
With this mapping, you can still use a plain j to move down one screen line at a time, but if you use a count, then you will move down to the line indicated by the corresponding
'relativenumber' marker. Accurate! :)
However, because there are no markers on the later screen lines of single wrapped lines, you still cannot use
Nj to jump to those lines. Still less usable :(.
How it works
This is an expression mapping, which maps to different commands depending on the current count.
If there is no count, it simply performs a
gj. If there
is a count it first clears the existing count with Esc, and then performs a regular
j using the value of that count.