Why do the \| characters in an :execute "normal ..." mapping require extra escaping backslashes?

Question

I'm struggling to understand a character escaping situation involving \|.

First I'm running the following search which matches either the A character or B character. I'm using the default magic option so the parentheses () and alternatives separator | must be preceded by backslashes:

/\(A\|B\)

Next I'd like to execute this as a normal command, so I run the following command and it works as expected. Since the search expression is now in a double quoted string, I have to escape each backslash character:

:execute "normal! /\\(A\\|B\\)\r"

Finally I'd like to create a normal mode mapping which executes this command. I run the following command:

:nnoremap <Leader>h :<C-u>execute "normal! /\\(A\\|B\\)\r"<CR>

However, when attempting to use this mapping, I get the following error:

E486: Pattern not found: \(A|B\)

In order to get this mapping to work, I have to add an additional backslash before the | character, but I don't have to add additional backslashes before the ( or ) characters! The working command is as follows:

:nnoremap <Leader>h :<C-u>execute "normal! /\\(A\\\|B\\)\r"<CR>
                                                 ^
                                                 |
                                   three backslashes instead of two

Why is an extra backslash required here to make this work? And why do the parentheses each not also require an extra backslash?

I'm running Vim 8.0, patches 1-1453, compiled myself on WSL.

Answer 1

| is used to separate a map command from the next one, you need to escape it or use <bar> instead, read :h map-bar for further detail.

Your map is unnecessarily complicated, {rhs} of a map without :h :map-<expr> is applied as typed, no need to use execute and normal in this case:

nnoremap <Leader>h /\(A\<bar>B\)<cr>

Answer 2

This is a bit of a guess, but I suspect it has to with | also being the ex command separator. So execute probably mangles things, and needs to see \|, which in string form would \\\|—in other words, | needs to be escaped.

Alternative ideas:

1. :/ is an ex command that addresses the search pattern entered. If memory serves, it works rather like a normal /
2. You can also set the search register with let @/ = ... and then simply :normal! n