Encoding issue with filename in MacVim

Question

Question

In MacVim I type :tabe jäderberg.txt to open a buffer for that file. The file does not yet exist. My status line displays the file name jäderberg.txt. I do i<C-r>% to put the file name into the buffer and it puts jäderberg.txt. I write and quit the buffer.

Now I run MacVim from Terminal by typing mvim j<Tab><CR>. The file opens, but the statusline displays ja¨derberg.txt. I type i<C-r>% and i get ja¨derberg.txt. If in Terminal I type out the whole file name, mvim jäderberg.txt, everything is fine. If I type mvim jä<Tab> Terminal refuses to complete the file name.

Why is my status line not displaying the file name correctly when I open MacVim from Terminal and tab-complete the filename?

As far as I've gotten

"Accented characters" can be stored as a single code point or as a series of code points. I don't know much about this, but I can tell there is a difference between how Terminal treats the character ä that I type and the character ä that is stored in a file name and completed with <Tab>.

Typing jä:

$ printf jä | xxd
0000000: 6ac3 a4                                  j..

Completing jä:

$ touch jä
$ printf jä | xxd
0000000: 6a61 cc88                                ja..

This seems to correspond to a difference between composed and decomposed characters. I don't understand why Vim should fail to correctly display the decomposed ä that is actually stored in the file name, and only when I give the file name with a decomposed ä.

Observations

Here are some further observations. I'm not sure what other info is relevant, so ask and I'll add.

1. If I move the cursor over the characters ja¨derberg.txt in the buffer, they change. With the cursor on j and pressing l until end of line, here is how the line changes.

   ja¨derberg.txt
   ja¨derberg.txt
   jadderberg.txt
   jadeerberg.txt
   jaderrberg.txt
   ...
   jaderberg.txtt
   

2. I wrote a function to loop through all characters on the line and convert each to a number and back to a char, then write the results to the buffer.

   function! JJDebugEncoding(lineno)
       let line = getline(a:lineno)
       let newline = ""
       for char in split(line, '\zs')
           let nr = char2nr(char)
           let newline = newline . nr2char(nr)
       endfor
       call append(a:lineno, newline)
   endfunc
   

   When I run it for a line with ja¨derberg.txt I get jaderberg.txt.

   There is no byte value 228 (ä) or 168 (¨) anywhere in the buffer.

3. Terminal Vim does not seem to have this problem.

4. encoding is set to utf-8. I'm not sure where this option is set since latin1 is the default, I'm not setting it in my vimrc and :verbose set encoding doesn't tell me anything. But I have tested that it is already set by the time my vimrc is sourced (that's why I don't set it myself) so I figure that should be ok. fileencoding defaults to encoding if it is not set, but I have also tried setting it explicitly before writing the buffer to file with no difference. fileencodings is set to ucs-bom,utf-8,default,latin1 and fileformats to unix,dos,mac.

Edit:

a.) NFS+ requires file names to only contain characters in Unicode Normal Form D, that is, accented accented characters are decomposed. Other filesystems seem to favor but not enforce UNF-C, composed. Therefore this is likely a MacVim-specific problem.

b.) I was wrong to say this does not affect terminal Vim. It does, and ga with cursor on ä gives

" typed
<a>  97,  Hex 61,  Octal 141 < ̈> 776, Hex 0308, Octal 1410
" from % register or tab-completed
<ä> 228, Hex 00e4, Octal 344

The reason I didn't notice this difference is that terminal Vim renders the characters the same way.

c.) Some relevant and helpful reading

• vim_mac 2007 related discussion
• Comment #2 contains a good explanation of part of the problem
• tab completing precomposed/decomposed characters
• vim_mac distinguish between the encoding and rendering parts of the problem