I've found myself in a bit of an interesting situation.
I'm writing a plugin in which I need to do a tiny bit of line parsing.
Given a file:
It's easy to verify that
col('.') returns 1, 3, and 5 respectively. This is fine, I can work with this. Other unicode characters (or I guess anything expanding outside ascii) returns similar results: each character has an expanded length,
col('.') is relative to the total string length.
In normal vimscript, with this specific use case in mind, I could:
fun! pndtest#Invoke() let line = getline('.') echoerr line[col('.') - 1] endfun
(note: the autoload prefix is an arbitrary one I picked because I had a testbench ready. They have no practical meaning and are largely random)
call pndtest#Invoke() on one of the unicode characters, and get trash out because it's only half the character in this case. That's actually fine - in fact, this approach is what I used in vim9 and failed to use (this is my actual implementation by slightly butchered for the sake of an MCVE):
var position: number = col('.') - 1 var line = getline('.') var i = position var output = "" while i >= 0 var character = line[i] # in reality, there's other parsing here, but this is a simplification output = character .. output i -= len(character) # This did nothing, by the way. :') endwhile
At the end of which,
output should contain all the characters, unicode and otherwise, prior to the current position in the line.
This works fine in Vimscript, but not in vim9. Here's the exact same case as earlier:
vim9script def pnd2test#Invoke() var line = getline('.') echoerr line[col('.') - 1] enddef
Given the same file as presented at the start of this post, it doesn't echoerr trash - it echoerrs the actual unicode character.
While this is great because it means Vim is starting to do some unicode handling, this means
col() is essentially useless because the strings are suddenly not indexable in the same way as they used to be. It works fine on the first column, but
getline('.') in the previously listed file is
ø, not trash triggered by a partial character. You might see where this is going.
This means that
col('.') would have to return 2 on
ø for the code to work, not 3. The
å would have to be 3, not 5. Unfortunately for me, this stacks. The more unicode characters in the line, the more the error propagates, and the worse the offset is, rendering my approach essentially useless.
And again, this is purely a vim9 thing - vimscript itself treats the characters appropriately relative to
col('.'), presenting different but arguably more managable unicode-related problems (they can largely be solved by subtracting or adding the length of the individual character)
Just for the sake of visualizing the problem: (note: the spaces used in the vimscript example represent the second byte of the character, NOT literal spaces)
VimScript: String index: 01234 file : æ ø å Col index : 1 3 5 # This is fine: subtract 1, the data is still continuous between the indices Vim9: String index: 012 File : æøå Col index : 135 # And this is the problem: complete mismatch with the string indices
Of course, while writing this question, I had a sudden realization: what about
match()? Again assuming the file first in the question and:
vim9script def pnd2test#Invoke() echoerr match(getline('.'), 'å') enddef
the result is 4. This means match() also results in legacy indices. Here's an expanded example:
æøå this is a unicode line
vim9script def pnd2test#Invoke() echoerr getline('.')[match(getline('.'), 'å')] enddef
å. Just for the record, this isn't a problem caused by
getline(). As observed earlier,
getline('.') does give the right results. This also results in
t instead of
def pnd2test#Invoke() var x: string = 'æøå t' echoerr x[match(x, 'å')] enddef
This in turn kinda expands the problem: several of the vimscript functions return indices that are incompatible with vimscript strings. This of course is my question: when there are several functions returning incompatible indices, how do I in vim9 do any operations on strings containing unicode when using these types of functions?