How to use arrow keys mappings in vim-submode plugin on linux terminal?

Question

Following this answer I've been using vim-submode for some months on a windows gVim. Now that I'm trying to apply the same configuration on a terminal vim on linux I have some problems:

With the folowing lines:

call submode#enter_with('test', 'n', '', '<leader><UP>', '<C-w>-')
call submode#map('test', 'n', '', '<UP>', '<C-w>-')

LeaderUp let me enter in the submode test without any issue but in this submode when I hit Up I leave the submode and insert a A character in my buffer.

If I replace the second line with:

call submode#map('test', 'n', '', 'a', '<C-w>-')

Everything works properly i.e. hitting a execute the submode command.

The issue comes from the terminal which doesn't interpret correctly the arrow key because the commands works perfectly fine on a gVim on the same system.

My problem is that I don't understand why this happens because:

• The arrow key is correctly interpreted in the enter_with mapping. Even if I replace the first line with call submode#enter_with('test', 'n', '', '<UP>', '<C-w>-') hitting Up will trigger the submode

• I can create a mapping like nnoremap <Up> :echo "FOO"<CR> which will work properly.

I've tried to replace '<UP>' with the code of the arrow using Ctrl+r Ctrl+r Up to insert it which gave me OA but that still doesn't work.

My setup is:

• Debian Wheezy
• rxvt Unicode
• vim 7.4

What can I do to solve my problem?

Answer 1

TLDR: Make sure that g:submode_keyseqs_to_leave does not include the <esc> key.

Either reset it to an empty list:

let g:submode_keyseqs_to_leave = []
                                 ^^

Or double the <esc> key:

let g:submode_keyseqs_to_leave = ['<esc><esc>']
                                        ^^^^^

In the latter case, you'll need to press Esc twice to leave the submode.

Make sure to set the variable before invoking submode#enter_with() and submode#map().

---

My problem is that I don't understand why this happens

Suppose you want <c-w>- to be repeatable by just pressing -, without any plugin. A first attempt could look like this:

nmap <c-w>- <c-w>-<c-w>
                  ^^^^^

The mapping preserves the original meaning of <c-w>-, and simply appends <c-w> in the rhs so that you don't have to press it again the next time you want to resize the window. Note that the first <c-w> in the rhs is not remapped; which is why <c-w>- is not replaced with itself indefinitely:

If the {rhs} starts with {lhs}, the first character is not mapped
again (this is Vi compatible).
For example: >
   :map ab abcd
will execute the "a" command and insert "bcd" in the text.  The "ab" in the
{rhs} will not be mapped again.

See :h recursive_mapping.

However, something unexpected may happen if you add another mapping like nno <c-w>i dd. If you press <c-w>- followed by a few -, the window will be correctly resized; but if you then press i to enter insert mode, Vim will instead delete the current line.

One solution is to introduce an ad-hoc prefix which – contrary to the <c-w> prefix – can't be typed interactively, and thus should not trigger another user mapping. The special key name <plug> is a good fit for this (<sid> would work too; see :h map-<sid>).

nmap <c-w>- <c-w>-<plug>(prefix)
                  ^^^^^^^^^^^^^^

But this creates another issue; when you press - after <c-w>-, the window is not resized again. You need a second mapping for your ad-hoc prefix to be able to play the role of <c-w>:

nmap <c-w>- <c-w>-<plug>(prefix)
nno <plug>(prefix)- <c-w>-
    ^^^^^^^^^^^^^^

This will allow you to repeat <c-w>- by pressing -, but only once. If you want to be able to repeat <c-w>- as many times as desired, the second mapping needs to be recursive:

nmap <c-w>- <c-w>-<plug>(prefix)
nmap <plug>(prefix)- <c-w>-
^^^^

After pressing <c-w>- once, as long as you press -, <plug>(prefix) will always be written in the typeahead buffer; you may see it on your command-line if 'showcmd' is set; actually, you may see its internal representation in Vim, which isýS(prefix) on my machine (vim-submode hides this by appending 5 no-break spaces to the prefix).

Reciprocally, as long as this ad-hoc prefix is written in the typeahead buffer, you don't have to press <c-w> anymore to execute <c-w>-, which gives you the impression of being in a submode.

In any case, that's basically what vim-submode does; it emulates a submode by making sure that:

• an ad-hoc and non-interactively typable prefix is always fed to the typeahead buffer
• this ad-hoc prefix can play the role of the original prefix (the one you press interactively, like <c-w>)

---

Getting back to your original code:

call submode#enter_with('test', 'n', '', '<leader><UP>', '<C-w>-')
call submode#map('test', 'n', '', '<UP>', '<C-w>-')

Here is a gross simplification of what submode#enter_with() and submode#map() do:

nmap <leader><up>       <c-w>-<plug>(prefix)
nmap <plug>(prefix)<up> <c-w>-<plug>(prefix)
nmap <plug>(prefix)     :echom 'restoring some settings, and leaving submode'<cr>

But #enter_with() also maps escape inside the submode, with something similar to this:

nmap <plug>(prefix)<esc> <nop>

If you write these 4 lines of code in a minimal vimrc and start Vim with it, you should be able to reproduce your issue:

vim -Nu NONE -S <(cat <<'EOF'
    nmap <leader><up>        <c-w>-<plug>(prefix)
    nmap <plug>(prefix)<up>  <c-w>-<plug>(prefix)
    nmap <plug>(prefix)      :echom 'restoring some settings, and leaving submode'<cr>
    nmap <plug>(prefix)<esc> <nop>
EOF
)

That is, pressing <leader><up> works, but <up> does not repeat <c-w>-; instead, it inserts A in the buffer.

Here's what happens:

1. you press <leader><up> which writes <c-w>-<plug>(prefix) in the typeahead buffer
2. Vim processes <c-w>- by decreasing the height of the current window; <plug>(prefix) remains unprocessed because there's an ambiguity (3 mappings start with this plug key), and Vim must wait until the timeout to decide what to do
3. you press <up> and Vim receives Esc O A; now the typeahead buffer contains <plug>(prefix) Esc O A
4. Vim expands <plug>(prefix) Esc into <nop> – which does nothing – because of the last mapping; now the typeahead buffer contains O A
5. Vim processes what remains from the typeahead buffer (i.e. O A), and opens a new line above the current line, as well as inserts A

It could be a Vim bug, because on linux Vim should wait 1s (provided 'timeout', 'ttimeout', 'timeoutlen' and 'ttimeoutlen' have their default values) to decide whether Esc O A is a sequence of key codes matching <up>, or just 3 separate keys.

It looks like this other issue where a Terminal-Job mode mapping using <esc> in its lhs breaks the arrow keys.

The solutions to both these issues are identical:

• either remove the mapping with <esc> in its lhs
• or make sure <esc> can't be confused with the start of a sequence of key codes produced by a special key (like <f1>, <up>, <A-b>...)

Note that Nvim does not seem to be affected by either of these.

---

Btw, here's an even simpler code example which suffers from the same issue:

nno <up>  :echom 'up'<cr>
nno <esc> :echom 'esc'<cr>

The first mapping works as expected by itself, but not anymore once you add the second mapping.

---

Bottom line: don't use <esc> in the lhs of a mapping, or, if you have to, build your lhs so that <esc> can never be parsed as the start of a sequence of key codes (on my machine, doubling <esc> seems to achieve this goal).

Knowing that, here's how you could have debug the issue. Start by running this command:

:new|pu=execute('nno')

It should dump all your normal-mode mappings in a new buffer. Now look for the pattern <esc> in the lhs of a mapping, and you should find these lines (:v/\c<esc>/d_ may help reduce the noise):

n  <Plug>(submode-rhs:test:for:<Esc>) * <Nop>
n  <Plug>(submode-prefix:test)     <Esc>   <Plug>(submode-rhs:test:for:<Esc>)<Plug>(submode-leave:test)

Then ask Vim from where these mappings were installed, with the :verbose modifier:

:verb nno <Plug>(submode-rhs:test:for:<Esc>)

The answer should contain something like:

Last set from /path/to/vim-submode/autoload/submode.vim line 247

If you then look for the line 247 inside the script, you should find:

execute s:map_command(a:mode, s:filter_flags(a:options, 'r'))

From there, you just have to walk back the function calls. The line 247 is written inside the function s:define_submode_mapping(), which is only called from submode#map(), which itself is only called from submode#leave_with(), which in turn is called from s:define_entering_mapping():

for keyseq in g:submode_keyseqs_to_leave
  call submode#leave_with(a:submode, a:mode, a:options, keyseq)
endfor

And here, you can see a global variable used to configure the plugin. If you then look at its documentation:

*g:submode_keyseqs_to_leave*
g:submode_keyseqs_to_leave    list of strings (default: ['<Esc>'])
The default key sequences to leave a submode.

You should see the problematic <Esc> in the default value.