Why does not Vim feed the correct set of newline characters?

Question

I'm trying to wrap my head around how Vim deals with line endings in different file formats. Say you have a file with DOS line endings, file.txt:

echo -en 'line1\nline2' > file.txt
unix2dos file.txt

If I open this one in Vim,

vim -u NORC -b file.txt

I'm given the following:

line1^M
line2

So far so good. I'm expecting ^M which is how Vim displays the DOS carriage return. But how do I edit this file - without changing the file format - and having the correct set of new line characters?

If I do vim -u NORC -c 'set ff=dos' -b file.txt and add some newlines, they do not show up with a ^M carriage return when I save and reopen the file. Is this to be expected? Am I missing some Vim configuration?

Answer 1

There are two problems with your approach.

First is using -b which turns on the 'binary' setting and the binary setting will ignore fileformat. See this snippet of :help 'binary':

Also, 'fileformat' and 'fileformats' options will not be used, the file is read and written like 'fileformat' was "unix" (a single <NL> separates lines).

The second part is that using -c to set fileformat is too late, since -c only executes after the file given on the command-line has been already opened.

You can work around that by using a --cmd argument in the command-line, though it's not ideal because --cmd is executed too early (before any vimrc is read) and ideally you'd want to be more specific about opening that file with DOS format only.

In any case, this will work:

$ vim -u NORC --cmd 'set ff=dos' file.txt

(BTW, I really recommend you also pass -N here to set nocompatible mode.)

Note that when you properly open the file with DOS fileformat, you won't see any ^Ms displayed inside Vim. That's the whole point, Vim has parsed the file from its file format, so now it's no longer a "DOS newline" but simply a "newline", so Vim will just show it as such. Vim still knows this file was a DOS fileformat file, so when it writes it back to disk, it will preserve that format and use "DOS newlines". That is, unless you change fileformat while editing or right before saving it, which is useful to have Vim convert file formats.

Your file also doesn't have a newline at the end of the file. Vim should correctly point that out when you open the file by setting noeol. (If you're not in nocompatible mode, then Vim reports this loudly as [Incomplete last line].) Which is fine, Vim can properly deal with such files. But the default behavior is for Vim to "fix" that when writing the file, by adding a newline to the last line. That is controlled by the 'fixendofline' option, which defaults to being enabled. If you want to preserve your last line without a newline, you can just set nofixeol in addition to your other options.

---

I previously mentioned that using --cmd 'set ff=dos' was too strong of a workaround. A better approach to use a specific file format while opening a file is using a ++opt to the :e command. So after you start Vim, passing it no file in the command-line, you can:

:e ++ff=dos file.txt

Which will open that file only forcing a DOS fileformat.

You can also use ++bin to open a file in binary mode.

---

Finally, it's possible that binary mode is what you'd really like, for full control of each and every byte in the file.

But beware that fileformat doesn't really work with binary mode (it makes sense, since it's meant to be a raw mode, so you wouldn't want any preprocessing.)

If you want to use binary mode and you still want to have "DOS newlines", you'll need to add the <CR> characters yourself. Perhaps manually with <C-V><C-M> or <C-V><Enter>, or perhaps using a global substitution to add a ^M character to lines that don't have it, once you're done editing the file.