4

Question

In MacVim I type :tabe jäderberg.txt to open a buffer for that file. The file does not yet exist. My status line displays the file name jäderberg.txt. I do i<C-r>% to put the file name into the buffer and it puts jäderberg.txt. I write and quit the buffer.

Now I run MacVim from Terminal by typing mvim j<Tab><CR>. The file opens, but the statusline displays ja¨derberg.txt. I type i<C-r>% and i get ja¨derberg.txt. If in Terminal I type out the whole file name, mvim jäderberg.txt, everything is fine. If I type mvim jä<Tab> Terminal refuses to complete the file name.

Why is my status line not displaying the file name correctly when I open MacVim from Terminal and tab-complete the filename?

As far as I've gotten

"Accented characters" can be stored as a single code point or as a series of code points. I don't know much about this, but I can tell there is a difference between how Terminal treats the character ä that I type and the character ä that is stored in a file name and completed with <Tab>.

Typing :

$ printf jä | xxd
0000000: 6ac3 a4                                  j..

Completing :

$ touch jä
$ printf jä | xxd
0000000: 6a61 cc88                                ja..

This seems to correspond to a difference between composed and decomposed characters. I don't understand why Vim should fail to correctly display the decomposed ä that is actually stored in the file name, and only when I give the file name with a decomposed ä.

Observations

Here are some further observations. I'm not sure what other info is relevant, so ask and I'll add.

  1. If I move the cursor over the characters ja¨derberg.txt in the buffer, they change. With the cursor on j and pressing l until end of line, here is how the line changes.

    ja¨derberg.txt
    ja¨derberg.txt
    jadderberg.txt
    jadeerberg.txt
    jaderrberg.txt
    ...
    jaderberg.txtt
    
  2. I wrote a function to loop through all characters on the line and convert each to a number and back to a char, then write the results to the buffer.

    function! JJDebugEncoding(lineno)
        let line = getline(a:lineno)
        let newline = ""
        for char in split(line, '\zs')
            let nr = char2nr(char)
            let newline = newline . nr2char(nr)
        endfor
        call append(a:lineno, newline)
    endfunc
    

    When I run it for a line with ja¨derberg.txt I get jaderberg.txt.

    There is no byte value 228 (ä) or 168 (¨) anywhere in the buffer.

  3. Terminal Vim does not seem to have this problem.

  4. encoding is set to utf-8. I'm not sure where this option is set since latin1 is the default, I'm not setting it in my vimrc and :verbose set encoding doesn't tell me anything. But I have tested that it is already set by the time my vimrc is sourced (that's why I don't set it myself) so I figure that should be ok. fileencoding defaults to encoding if it is not set, but I have also tried setting it explicitly before writing the buffer to file with no difference. fileencodings is set to ucs-bom,utf-8,default,latin1 and fileformats to unix,dos,mac.

Edit:

a.) NFS+ requires file names to only contain characters in Unicode Normal Form D, that is, accented accented characters are decomposed. Other filesystems seem to favor but not enforce UNF-C, composed. Therefore this is likely a MacVim-specific problem.

b.) I was wrong to say this does not affect terminal Vim. It does, and ga with cursor on ä gives

" typed
<a>  97,  Hex 61,  Octal 141 < ̈> 776, Hex 0308, Octal 1410
" from % register or tab-completed
<ä> 228, Hex 00e4, Octal 344

The reason I didn't notice this difference is that terminal Vim renders the characters the same way.

c.) Some relevant and helpful reading

  • 1
    For what it's worth, I can't reproduce this in Linux Vim or Linux gVim... – Martin Tournoij Aug 24 '15 at 12:10
  • @Carpetsmoker Thanks for checking, I've done some more reading and it does seem to be a MacVim specific problem. I'll update my question with some links for now and continue looking when I get a chance. – jjaderberg Aug 24 '15 at 14:49
  • 1
    looks like vim can't handle decomposed characters well. Hm, that might be a bug. You might want to create an issue for that. If you do convmv --nfc -t utf-8 -f utf-8 j*derberg.txt does it work? – Christian Brabandt Oct 10 '15 at 12:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.