What is the rationale for \r and \n meaning different things in s command?

Question

We all know that, when searching, \n is newline and \r is carriage return (^M), but when replacing \r is newline while \n is a null byte (^@).

What is the origin of this asymmetry? Given that this behavior is... peculiar to say the least (and quite counterproductive when you get it wrong the first time) I expect there's some bizarre historical reason.

(incidentally, is there some way to "fix" this behavior and get something more intuitive?)

Answer 1

At the most basic level, there's already an asymmetry between the search and replace portions of :substitute because the former is a regular expression and the latter is text, with specific additional escape sequences. This is just highlighted by the intuition you have about what \n means.

For example, consider that \n in the search doesn't match a literal \n. It matches the end of line (EOL) byte sequence, which may be \r, \r\n, or just \n depending on the 'fileformat' of the buffer.

As far as why \r is used to mean "insert an EOL", there's some history behind that. Vi had no way to handle a NUL byte in a file. Vim improved on that by replacing NUL bytes with an NL byte internally (since C strings are NUL delimited).

This implementation detail leaked into the behavior of :substitute since \n in the replacement is simply inserted into the internal representation of that line, which is used to indicate a NUL byte. \r inserts an EOL, breaking the internal line in two. Vim doesn't actually store the EOL bytes in memory, instead (de)serializing them when reading/writing the buffer.

It can't be changed now without breaking the many scripts and the muscle memory of many users. Thankfully, it's documented in :help sub-replace-special.

Answer 2

A NUL byte is a string terminator in C, and for this reason Vim uses this convention, described in the manual at :h NL-used-for-Nul:

<Nul> characters in the file are stored as <NL> in memory. In the display they are shown as "^@". The translation is done when reading and writing files. To match a <Nul> with a search pattern you can just enter CTRL-@ or "CTRL-V 000". This is probably just what you expect. Internally the character is replaced with a <NL> in the search pattern. What is unusual is that typing CTRL-V CTRL-J also inserts a <NL>, thus also searches for a <Nul> in the file. {Vi cannot handle <Nul> characters in the file at all}

This convention has spilled over to the :s/.../.../ command, but not to the substitute() function. \r and \n in replacement strings in substitute() calls keep their original meaning.

I don't think there are deeper reasons for either behaviour. Vim has simply evolved organically from the original vi. There never was any big blueprint for it, features were just piled on top of each other, with relatively little effort to keep them organized.

Answer 3

Other Vi clones do not support \r or \n (as a real backslash and letter) in substitution, but the behaviour of a real ^M (CTRL-V Enter) meaning to split the line into two lines is standard behavior:

Entering a <carriage-return> in repl (which requires an escaping <backslash> in ex mode and an escaping <control>-V in open or vi mode) shall split the line at that point, creating a new line in the edit buffer. The <carriage-return> shall be discarded.

In the Unix History archive, the first version of BSD ex/vi it appears in is 4.1cBSD (@(#)ex_re.c 7.2 10/16/81, and is not present in 4BSD (@(#)ex_re.c 6.2 10/23/80) [4.1a and 4.1b are not present in the archive].

The relevant code is:

/* ^V <return> from vi to split lines */
if (c == '\r')
    c = '\n';

This is also mentioned in the news file:

It is now possible to split lines with substitute commands from vi, by using ^V<return> in the rhs. This takes care of the last good reason for using ex command mode.

The previously supported behavior in ex command mode was for backslash-enter (i.e. backslash followed by a real newline) to insert a newline.

Answer 4

The origin of the asymmetry goes back a ways into computing history.

Short version:

<CR> & <LF>  (Carriage-Return and Linefeed) 
== 
\r & \n

Long version:
The first screens were basically digital versions of teletypes (TTY) and used control codes to generate similar behavior to printers. Carriage-return took the cursor (or print-head) to the start column. Linefeed advanced to the next row (on a screen) and fed the paper forward one line.

For printers, you had to do a paired <CR><LF> or your output would not look right. On early screens, the issue still held true.

DOS (and sorta-Windows after) followed the old standard and saves text with <CRLF>.

*NIX text (as most vi users are familiar) only uses <LF> for efficiency.

To test in Windows, use Word/Wordpad and save a few lines of text "as type: Text - MS-DOS format". Then open that same file in Notepad. It should look normal. Then save the same file in Word/Wordpad "as type: Text". Notepad will ignore all newlines and run the lines together. [Notepad's text format defaults to the \r\n combination while Word/Wordpad default to \n.]

\r is the code equivalent of <CR>

\n is the code equivalent of <LF>

And in my (very limited) experience with vi, it would try to "fix" the <CRLF> combination from my DOS text editor. vi ended up removing one character, replacing with <NUL>. A big part of the reason I stopped using vi.