I've found myself in a bit of an interesting situation.
I'm writing a plugin in which I need to do a tiny bit of line parsing.
Given a file:
æøå
It's easy to verify that col('.')
returns 1, 3, and 5 respectively. This is fine, I can work with this. Other unicode characters (or I guess anything expanding outside ascii) returns similar results: each character has an expanded length, col('.')
is relative to the total string length.
In normal vimscript, with this specific use case in mind, I could:
fun! pndtest#Invoke()
let line = getline('.')
echoerr line[col('.') - 1]
endfun
(note: the autoload prefix is an arbitrary one I picked because I had a testbench ready. They have no practical meaning and are largely random)
call pndtest#Invoke()
on one of the unicode characters, and get trash out because it's only half the character in this case. That's actually fine - in fact, this approach is what I used in vim9 and failed to use (this is my actual implementation by slightly butchered for the sake of an MCVE):
var position: number = col('.') - 1
var line = getline('.')
var i = position
var output = ""
while i >= 0
var character = line[i]
# in reality, there's other parsing here, but this is a simplification
output = character .. output
i -= len(character) # This did nothing, by the way. :')
endwhile
At the end of which, output
should contain all the characters, unicode and otherwise, prior to the current position in the line.
This works fine in Vimscript, but not in vim9. Here's the exact same case as earlier:
vim9script
def pnd2test#Invoke()
var line = getline('.')
echoerr line[col('.') - 1]
enddef
Given the same file as presented at the start of this post, it doesn't echoerr trash - it echoerrs the actual unicode character.
While this is great because it means Vim is starting to do some unicode handling, this means col()
is essentially useless because the strings are suddenly not indexable in the same way as they used to be. It works fine on the first column, but getline('.')[1]
in the previously listed file is ø
, not trash triggered by a partial character. You might see where this is going.
This means that col('.')
would have to return 2 on ø
for the code to work, not 3. The å
would have to be 3, not 5. Unfortunately for me, this stacks. The more unicode characters in the line, the more the error propagates, and the worse the offset is, rendering my approach essentially useless.
And again, this is purely a vim9 thing - vimscript itself treats the characters appropriately relative to col('.')
, presenting different but arguably more managable unicode-related problems (they can largely be solved by subtracting or adding the length of the individual character)
Just for the sake of visualizing the problem: (note: the spaces used in the vimscript example represent the second byte of the character, NOT literal spaces)
VimScript:
String index: 01234
file : æ ø å
Col index : 1 3 5 # This is fine: subtract 1, the data is still continuous between the indices
Vim9:
String index: 012
File : æøå
Col index : 135 # And this is the problem: complete mismatch with the string indices
Of course, while writing this question, I had a sudden realization: what about match()
? Again assuming the file first in the question and:
vim9script
def pnd2test#Invoke()
echoerr match(getline('.'), 'å')
enddef
the result is 4. This means match() also results in legacy indices. Here's an expanded example:
File:
æøå this is a unicode line
Code:
vim9script
def pnd2test#Invoke()
echoerr getline('.')[match(getline('.'), 'å')]
enddef
Returns t
, not å
. Just for the record, this isn't a problem caused by getline()
. As observed earlier, getline('.')[2]
does give the right results. This also results in t
instead of å
:
def pnd2test#Invoke()
var x: string = 'æøå t'
echoerr x[match(x, 'å')]
enddef
This in turn kinda expands the problem: several of the vimscript functions return indices that are incompatible with vimscript strings. This of course is my question: when there are several functions returning incompatible indices, how do I in vim9 do any operations on strings containing unicode when using these types of functions?
charcol('.')
what you're looking for? It works similarly tocol('.')
but returns the index of the Unicode character, not the byte position.