Inserting the content of a register where a macro is recorded results in control characters becoming unicode characters

Question

If I hit Ctrl+V and then Escape while in insert or ex mode, I get ^[ as a single character.

If I hit qqEscapeq and then go see :registers, I see that q contains ^[<80><fd>a, which are actually 4 atomic entites: ^[ (the same one as in the previous paragraph), <80>, <fd>, and a. (I thought the a was a whole with <fd>, but it is not; to check it, I pasted q register into a line and then run :s/.//gn on that line, which resulted in 4 matches on 1 line.)

If now I enter ex mode and hit Ctrl+R and then q, I get ^[<80>ýa, where ^[, <80>, and a are as before, whereas ý has substituted what was <fd>.

This conversion from <fd> to ý is something that bugged me while I was editing some long macro, for quite a few minutes before I realised some characters where not as I expect them.

What is this behavior?

Answer 1

The ^[<80><fd>a sequence is a <Nop>, which means Vim recognizes this key sequence as valid, but it produces no action at all.

The reason why Vim will insert a <Nop> right after an <Esc> while recording a macro is that <Esc> is typically a key that starts a special sequence (special keys such as <Up>, <Down>, <F1>, etc. all produce sequences that start with <Esc>), so if you were to press a sequence of keys that produced a valid escape sequence, right after pressing <Esc>, then when replaying that sequence, Vim would recognize that key, instead of <Esc> and the other characters that were actually recorded.

While you're using Vim interactively, this is typically not a problem, because Vim uses a typically very short timeout after an <Esc> waiting for more characters to recognize a special key sequence, but in a recording all characters are interpreted right away, so a timeout wouldn't be preserved... For that reason, Vim has this special case, when a timeout happens after an <Esc> and recording is on, it inserts a <Nop> there, since following it with a <Nop> will cause the <Esc> to never be misinterpreted.

This behavior was actually implemented in Vim 8.1.1003 to fix the issue reported in #4068.

---

To understand the ^[<80><fd>a sequence, file src/keymap.h in the Vim source code has all the keycode definitions:

• ^[ is the <Esc> that starts all key sequences;
• <80> is K_SPECIAL;
• <fd> (which is 253 in decimal) is KS_EXTRA; and finally
• a (which has ASCII code 97) is KE_NOP.

These are then assembled into a K_NOP sequence.

---

Finally, about pasting those contents into a buffer. The ý character corresponds to Unicode codepoint U+00FD, so there's really no reason why Vim wouldn't show it as ý when the character is inserted into a buffer.

You can see the codepoint information by using the ga command with the cursor on top of the ý, which will show:

<ý> 253, Hex 00fd, Oct 375, Digr y'

Indeed, it's a bit odd that when showing register contents it will show up as <fd>, but I guess that's because it's OK to encode contents of registers when displaying them, it's not that important to display all Unicode characters literally in that display.

There's also the odd difference that <80> shows as <80>, but well that's because Unicode U+0080 is actually just a padding character. Since it's a non-printable character, Vim decides to just show its sequence, which it then represents by the <80> also used in other contexts, such as when showing the contents of registers.