2

Sometimes I download files in cp1251, cp866, and koi8r encodings, all of which are used to represent Cyrillic characters.

How should I specify them in set fileencodings=... so that when I open such a file in Vim, it will detect the encoding correctly?

For example, I tried:

setglobal fileencoding=utf-8
set encoding=utf-8
set fileencodings=utf8,cp1251,cp866,ucs-2le,koi8r

It doesn't really work. Here is the text used for text files and the results of each test:

Съешь ещё этих мягких французских булок, да выпей же чаю.

file in cp1251 - encoding is detected correctly
file in cp866  - encoding is detected as cp1251
file in koi8r  - encoding is detected as cp1251
1
5

Vim uses the first encoding that's considered "valid"; for multi-byte encodings such as UTF-8 this is more or less reliable since many documents are not valid UTF-8 documents (although it can sometimes fail for shorter texts), but for fixed-with encodings such as cp1251, cp866, koi8r you almost always end up with a valid document, which is why Vim selects cp1251. "Valid" in the sense "this is a valid codepoint"; Vim doesn't "know" anything about the text and whether or not you intended to write "д" instead of a "{".

You do have a few options:

  • Set the encoding in a modeline; see :help modeline.

  • Store the encoding in the filename (hello.cp1251.txt) and set it with an autocmd. If files with a specific encoding are always in the same directory, then you can use that too. For example:

    augroup set-encoding
        au BufReadPost *cp1251*      set encoding=cp1251
        au BufReadPost /path/to/dir  set encoding=cp1251
    augroup end
    
  • A function to change the encoding easily might help (e.g. something like this). You could even extend this by showing the results of multiple encodings in splits and allowing you to choose one of them.

  • And finally, you could write a more robust "detect encoding" function which does a more thorough analysis of the decoded content; for example if you're dealing with prose you could check that at least 80% of the words appear in a Russian dictionary. I can't really find a plugin that does this already right now, and writing it is a bit beyond the scope of this answer.

    This most likely also won't be 100% foolproof, especially not if you use these encodings for non-prose documents (such as code).

In general, the more encodings you have the harder all of this gets.

3
  • Something else I just thought of: another option might be to use the file and/or iconv tools, which may have better detection for these files. I'm not sure if they do though Sep 12 at 22:02
  • Thanks a lot, this answer is really helpful.
    – jsv
    Sep 12 at 22:14
  • I think the modeline does not work per se. Once Vim reads the modeline, it already detected the encoding. Changing it at that time means you have to re-load the file or it wouldn't have an effect I think Sep 13 at 10:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.