Why is a Line Feed converted into a Null character inside the search register and into a Carriage Return on the command line?

Question

If I have the following text:

foo
bar

I visually select it and copy it.
The text is now stored in the unnamed register " and here is its contents (output of :reg "):

""   foo^Jbar^J

According to this chart, it seems ^J is the caret notation for a Line Feed.

If I want to duplicate the unnamed register in the a register by typing: :let @a = @"
Here is its contents (output of :reg a):

"a   foo^Jbar^J

It didn't change.

If I now duplicate it in the search register by typing :let @/ = @", here is its contents (output of :reg /):

"/   foo^@bar^@

According to the previous chart, it seems ^@ is the caret notation for a Null character.
Why is a Line Feed automatically converted into a Null character inside the search register (but not the a register)?

If I insert the unnamed register on the command line (or inside a search after /), by typing :<C-R>", here is what is inserted:

:foo^Mbar^M

Again, according to the last chart, ^M seems to be the caret notation for a Carriage Return.
Why is a Line Feed automatically converted into a Carriage Return on the command line?

Edit:

Usually you can insert a literal control character by typing:
<C-V><C-{character in caret notation}>

For example, you can insert a literal <C-R> by typing <C-V><C-R>.
You can do it for seemingly any control character.
However I've noticed that I'm unable to insert a literal LF inside a buffer or on the command line, because if I type: <C-V><C-J> it inserts ^@, a null character, instead of ^J.
Is it for the same reason a LF is converted into NUL inside the search register?

Edit 2:

In :h key-notation, we can read this:

<Nul>       zero            CTRL-@    0 (stored as 10) <Nul>
<NL>        linefeed        CTRL-J   10 (used for <Nul>)

The stored as 10 part on the first line and used for <Nul> on the second line could indicate that there's some sort of overlap between a LF and a NUL, and that they could be interpreted as the same thing. But they can't be the same thing, because after executing the previous command :let @/ = @", if I type n in normal mode to get to the next occurrence of the 2 lines foo and bar, instead of getting a positive match, I have the following error message:

E486: Pattern not found: foo^@bar^@

Besides this link seems to explain that a NUL denotes the end of a string, whereas a LF denotes the end of a line in a text file.

And if a NUL is stored as 10 as the help says, which is the same code as for a LF, how is Vim able to make the difference between the 2?

Edit 3:

Maybe a LF and a NUL are coded with the same decimal code, 10, as the help says. And Vim makes the difference between the 2 thanks to the context. If it meets a character whose decimal code is 10 in a buffer or any register, except the search and command registers, it interprets it as a LF.
But in the search register (:reg /) it interprets it as a NUL because in the context of a search, Vim only searches for a string where the concept of end of line in a file doesn't make sense because a string is not a file (which is weird since you can still use the atom \n in a searched pattern, but maybe that's only a feature of the regex engine?). So it automatically interprets 10 as a NUL because it's the nearest concept (end of string ≈ end of line).

And in the same way, on the command line / command register (:reg :) it interprets the code 10 as a CR, because the concept of end of line in a file doesn't make sense here. The nearest concept is end of command so Vim interprets 10 as a CR, because hitting Enter is the way to end/execute a command and a CR is the same as hitting Enter, since when you insert a literal one with <C-V><Enter>, ^M is displayed.

Maybe the interpretation of the character whose code is 10 changes according to the context:

• end of line in a buffer (^J)
• end of string in a search (^@)
• end of command on the command line (^M)

Answer 1

First, thank you for this very comprehensive and thoughtful post.

After some testing, I have come to this conclusion:

1. Control characters are displayed using the caret notation: ^M for <CR> (carriage return) and ^J for <LF> (line feed). In buffers, <EOL> (end-of-line) are displayed as new screen lines and are input with the enter key. <EOL> depend on the file format of the buffer: <EOL> = <CR>|<LF>|<CR><LF> for mac|unix|dos respectively.

2. When editing a buffer, the file format is always set. To change the file format of an opened buffer, you can use the following command that converts <EOL>:

   :set f[ile]f[ormat]=mac|unix|dos
   

   In addition to converting <EOL>, this command converts <LF> to <CR> when changing the file format from mac to unix|dos, and conversely, <CR> to <LF> when changing the file format from unix|dos to mac. To see the real bytes of the buffer, you can use the following command that transforms the textual representation of the buffer into its hexadecimal representation using the convenient hexadecimal editor xxd:

   :%!xxd
   

3. In registers (showed with the command :reg[isters] or :di[splay]), <EOL> are always displayed as ^J (but not all ^J are <EOL>), regardless of the file format of the buffer. However <EOL> are stored as they should. To be able to distinguish visually real ^J (that is <LF>) from the others ^J (that is <EOL>) in registers, you can use the following command that displays the hexadecimal values instead of the caret notation of control characters different from <EOL>:

   :set d[ispla]y=uhex
   

4. In search patterns and substitution strings:

   \r = newline different from <EOL> (<CR> if <EOL> = <CR><LF>|<LF>, <LF> if <EOL> = <CR>)
   \n = <EOL>
   

5. Everywhere:

   <C-V><C-M>|<C-V><EOL> = newline different from <EOL>
   <C-V><C-J> = <NUL>
   

   This shows that when the file format is dos, it is impossible to input <LF>, since <EOL> = <CR><LF> and <C-V><C-M>|<C-V><EOL> = <CR>.

6. In substitution strings:

   • newline different from <EOL> are interpreted as <EOL>;

   • <EOL> are interpreted as <NUL>.

   So, according to 4., :%s[ubstitute]/\r/\r/g replaces every newline different from <EOL> in the buffer with <EOL>, while :%s[ubstitute]/\n/\n/g replaces every <EOL> in the buffer with <NUL>.

7. In the search register / and command register :, <EOL> are converted to

   • newline different from <EOL> when inserted from a register with /<C-R>{register} or :<C-R>{register} respectively;

   • <NUL> when inserted from a register with :let @/=@{register} or :let @:=@{register} respectively.

8. In buffers, newline different from <EOL> are converted to <EOL> when inserted from a register using i<C-R>{register}.

Why is a Line Feed converted into a Null character inside the search register and into a Carriage Return on the command line?

Before copying <LF> from the unnamed register " to other registers, you need to input <LF> and put it in the register ". If the file format is unix, you can do that by using yy on an empty line; if the file format is mac, you can do that by using i<C-V><C-M><Esc>yl; if the file format is dos, you cannot input <LF> (cf. 5.).

Now your statement is partially wrong, since

• you do not use the same method for copying <LF> from the register " into the search register / and command register :. You use :let @/=@" for copying into the register / and :<C-R>" for copying into the register :. Using /<C-R>" and :<C-R>" respectively will give you the same result (<CR>) in both cases;

• the conversions of <LF> that take place with your two different copy methods happen only when the file format is unix. If it is mac, <LF> is not converted when copied to the register / or the register :, and if it is dos you cannot even input <LF>.

The right statement is given by 7. But I really do not the know the reasons behind it.