How can I detect the encoding of a file correctly?

Question

Sometimes I download files in cp1251, cp866, and koi8r encodings, all of which are used to represent Cyrillic characters.

How should I specify them in set fileencodings=... so that when I open such a file in Vim, it will detect the encoding correctly?

For example, I tried:

setglobal fileencoding=utf-8
set encoding=utf-8
set fileencodings=utf8,cp1251,cp866,ucs-2le,koi8r

It doesn't really work. Here is the text used for text files and the results of each test:

Съешь ещё этих мягких французских булок, да выпей же чаю.

file in cp1251 - encoding is detected correctly
file in cp866  - encoding is detected as cp1251
file in koi8r  - encoding is detected as cp1251

Answer 1

Vim uses the first encoding that's considered "valid"; for multi-byte encodings such as UTF-8 this is more or less reliable since many documents are not valid UTF-8 documents (although it can sometimes fail for shorter texts), but for fixed-with encodings such as cp1251, cp866, koi8r you almost always end up with a valid document, which is why Vim selects cp1251. "Valid" in the sense "this is a valid codepoint"; Vim doesn't "know" anything about the text and whether or not you intended to write "д" instead of a "{".

You do have a few options:

• Set the encoding in a modeline; see :help modeline.

• Store the encoding in the filename (hello.cp1251.txt) and set it with an autocmd. If files with a specific encoding are always in the same directory, then you can use that too. For example:

  augroup set-encoding
      au BufReadPost *cp1251*      set encoding=cp1251
      au BufReadPost /path/to/dir  set encoding=cp1251
  augroup end
  

• A function to change the encoding easily might help (e.g. something like this). You could even extend this by showing the results of multiple encodings in splits and allowing you to choose one of them.

• And finally, you could write a more robust "detect encoding" function which does a more thorough analysis of the decoded content; for example if you're dealing with prose you could check that at least 80% of the words appear in a Russian dictionary. I can't really find a plugin that does this already right now, and writing it is a bit beyond the scope of this answer.

  This most likely also won't be 100% foolproof, especially not if you use these encodings for non-prose documents (such as code).

In general, the more encodings you have the harder all of this gets.