5

Question

In MacVim I type :tabe jäderberg.txt to open a buffer for that file. The file does not yet exist. My status line displays the file name jäderberg.txt. I do i<C-r>% to put the file name into the buffer and it puts jäderberg.txt. I write and quit the buffer.

Now I run MacVim from Terminal by typing mvim j<Tab><CR>. The file opens, but the statusline displays ja¨derberg.txt. I type i<C-r>% and i get ja¨derberg.txt. If in Terminal I type out the whole file name, mvim jäderberg.txt, everything is fine. If I type mvim jä<Tab> Terminal refuses to complete the file name.

Why is my status line not displaying the file name correctly when I open MacVim from Terminal and tab-complete the filename?

As far as I've gotten

"Accented characters" can be stored as a single code point or as a series of code points. I don't know much about this, but I can tell there is a difference between how Terminal treats the character ä that I type and the character ä that is stored in a file name and completed with <Tab>.

Typing :

$ printf jä | xxd
0000000: 6ac3 a4                                  j..

Completing :

$ touch jä
$ printf jä | xxd
0000000: 6a61 cc88                                ja..

This seems to correspond to a difference between composed and decomposed characters. I don't understand why Vim should fail to correctly display the decomposed ä that is actually stored in the file name, and only when I give the file name with a decomposed ä.

Observations

Here are some further observations. I'm not sure what other info is relevant, so ask and I'll add.

  1. If I move the cursor over the characters ja¨derberg.txt in the buffer, they change. With the cursor on j and pressing l until end of line, here is how the line changes.

    ja¨derberg.txt
    ja¨derberg.txt
    jadderberg.txt
    jadeerberg.txt
    jaderrberg.txt
    ...
    jaderberg.txtt
    
  2. I wrote a function to loop through all characters on the line and convert each to a number and back to a char, then write the results to the buffer.

    function! JJDebugEncoding(lineno)
        let line = getline(a:lineno)
        let newline = ""
        for char in split(line, '\zs')
            let nr = char2nr(char)
            let newline = newline . nr2char(nr)
        endfor
        call append(a:lineno, newline)
    endfunc
    

    When I run it for a line with ja¨derberg.txt I get jaderberg.txt.

    There is no byte value 228 (ä) or 168 (¨) anywhere in the buffer.

  3. Terminal Vim does not seem to have this problem.

  4. encoding is set to utf-8. I'm not sure where this option is set since latin1 is the default, I'm not setting it in my vimrc and :verbose set encoding doesn't tell me anything. But I have tested that it is already set by the time my vimrc is sourced (that's why I don't set it myself) so I figure that should be ok. fileencoding defaults to encoding if it is not set, but I have also tried setting it explicitly before writing the buffer to file with no difference. fileencodings is set to ucs-bom,utf-8,default,latin1 and fileformats to unix,dos,mac.

Edit:

a.) NFS+ requires file names to only contain characters in Unicode Normal Form D, that is, accented accented characters are decomposed. Other filesystems seem to favor but not enforce UNF-C, composed. Therefore this is likely a MacVim-specific problem.

b.) I was wrong to say this does not affect terminal Vim. It does, and ga with cursor on ä gives

" typed
<a>  97,  Hex 61,  Octal 141 < ̈> 776, Hex 0308, Octal 1410
" from % register or tab-completed
<ä> 228, Hex 00e4, Octal 344

The reason I didn't notice this difference is that terminal Vim renders the characters the same way.

c.) Some relevant and helpful reading

3
  • 1
    For what it's worth, I can't reproduce this in Linux Vim or Linux gVim... Aug 24, 2015 at 12:10
  • @Carpetsmoker Thanks for checking, I've done some more reading and it does seem to be a MacVim specific problem. I'll update my question with some links for now and continue looking when I get a chance.
    – jjaderberg
    Aug 24, 2015 at 14:49
  • 1
    looks like vim can't handle decomposed characters well. Hm, that might be a bug. You might want to create an issue for that. If you do convmv --nfc -t utf-8 -f utf-8 j*derberg.txt does it work? Oct 10, 2015 at 12:33

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.