3

TLDR

I have this text in a buffer:

a 'string' and 'another one' !
            ^
            cursor is here

And I search for this regex in a / command:

\%(.*\%#\)\@=\%('[^']*'\)\@>\%(\%#.\+\)\@<=

There is no match.

Now I remove \@> from the regex:

\%(.*\%#\)\@='[^']*'\%(\%#.\+\)\@<=

And search for it again. This time ' and ' is matched.

What did \@> do exactly in the first regex, which explains why ' and ' was not matched?


Additional information

Suppose I'm looking for a regex to match the single-quoted string under the cursor.
And I use this text to test the regex:

a 'string' and 'another one' !

I don't want any match while the cursor is positioned on any of these characters:

a 'string' and 'another one' !
^^        ^^^^^             ^^

I want 'string' to be the match when the cursor is on any of these characters:

a 'string' and 'another one' !
  ^^^^^^^^

And I want 'another one' to be the match when the cursor is on any of these characters:

a 'string' and 'another one' !
               ^^^^^^^^^^^^^

Here's one possible regex:

\%(.*\%#\)\@=\%('[^']*'\)\@>\%(\%#.\+\)\@<=
├───────────┘   ├─────┘     ├─────────────┘
│               │           └ the cursor must be before the closing quote
│               └ a quoted string
└ the cursor must be after the opening quote

I understand why it works when the cursor is inside a string, but I don't understand why it works when the cursor is outside. For example, assuming the cursor is on the n of and:

a 'string' and 'another one' !
            ^

the regex fails to match anything, which is what I want. This is due to the atom \@> – which is inspired from the atomic group (?>pattern) in perl – because if I remove it, the regex becomes:

\%(.*\%#\)\@='[^']*'\%(\%#.\+\)\@<=

and when the cursor is on the n of and, this new regex matches ' and ':

a 'string' and 'another one' !
         ^^^^^^^

If you think this last regex matches 'string' instead of ' and ', because that's the text which is highlighted by the Search highlight group, run this command:

:s///c

When the substitution command is asking for your confirmation before replacing the pattern, you should see that the text highlighted by the IncSearch highlight group is ' and ', not 'string'.

real text matched by previous regex


I know what the help says about \@>. It prevents the regex engine from backtracking.

For example, let's assume you want to match all sequences of several uppercase characters not followed by a comma, nor by another uppercase character; and you test your regex against this text:

ABC,
DEF
GHI,
JKL

If you try this regex:

\u\{2,},\@!

It correctly matches DEF and JKL, but it also wrongly matches AB and GH.

This is because when the regex engine processed the token \u\{2,}, it initially matched ABC and GHI (\{2,} is greedy so it matches as many uppercase characters as it can); but then it realized that there was a comma, and that the negative lookahead assertion ,\@! could not be satisfied. So it backtracked into \u\{2,} again, and this time was a little less greedy; it matched only AB and GH. Since there is no comma after AB and GH, the negative lookahead assertion can be satisfied and there is a match.

Now if you apply \@> to \u\{2,}, then you prevent the regex engine from backtracking into \u\{2,}. That is, once the regex engine finds some text matching \u\{2,}, it never comes back into it again; even if some later part of the regex fails. This is why the following regex correctly finds DEF, JKL, and nothing else:

\%(\u\{2,}\)\@>,\@!
^^^       ^^^^^

Going back to the original text:

a 'string' and 'another one' !
            ^
            cursor is here

The regex:

\%(.*\%#\)\@=\%('[^']*'\)\@>\%(\%#.\+\)\@<=

finds nothing, because of \@>. But how does preventing the regex engine from backtracking into '[^']*' also prevents the ' and ' match?

The core of the regex is '[^']*' – two consecutive quotes with a minimum amount of text in-between – for which there are 3 possible matches:

'string'
' and '
'another one'

I understand why the first match fails; it's because of the negative lookbehind, which asserts that the cursor is positioned before the last quote, while it's after when it's on the n of and.

But I don't understand why the second match fails. When the engine reaches the second quote on the line, it should find that ' and ' matches, even if it can't backtrack into '[^']*'. It can't give up any character inside and anyway (if it did, it couldn't match the second quote anymore), so there is no submatch which \@> can prevent.


The only explanation I could find was the following one:

because of \@>, the regex engine is not allowed to start a match from any character in any previous text which was successfully matched by \@> (even if the global regex failed to match). \%('[^']*'\)\@> successfully matched 'string' – even though the overall regex failed – so now it can't start a match from any character inside 'string'. And to find ' and ', the engine would need to start its match from the second quote in 'string'. It can't; thus it doesn't find ' and '.

But this feature of \@> doesn't seem to be documented at :h /\@>. Unless this is implied by:

Matches the preceding atom like matching a whole pattern.

If '[^']*' matches a string like a whole pattern, it means that it doesn't matter whether the rest of the regex matches; it's processed as if it was alone (i.e. no backtracking).

But maybe it means something more; maybe it also implies that every character inside the text matched by '[^']*' is ignored when searching for further matches. Somewhat in the same way that A.*C\|B.*D can't match BCD in ABCD because ABC has already been matched, and the regex engine can't start a match inside a previous one (unless it's processing a multi-line regex).

Unfortunately, I don't think that can be the explanation. Consider this text:

ABCD

And this regex:

\%(A.*C\)\@>E\@=\|B.*D

A.*C matches ABC, but the regex fails because of the positive lookahead E\@= which asserts that an E follows immediately afterward. Nevertheless, ABC is matched like a whole pattern thanks to \@>. If that means that no further match can start inside, then BCD should not match. In practice, BCD does match; so the previous explanation is either wrong or incomplete.

2

TLDR

I think the issue is due to the new NFA regex engine which doesn't support some particular combination of atoms.

So, even though the following regex can be used to match the string under the cursor:

\%(.*\%#\)\@=\%('[^']*'\)\@>\%(\%#.\+\)\@<=

It only works because of an implementation detail; which probably means that there is no guarantee it will still work in the future.

A more reliable regex is:

\%(^\%('[^']*'\|[^']\)*\)\@<=\%('[^']*\%#[^']*'\|\%#'[^']*'\)
       ├─────┘  ├──┘  │         ├─────────────┘  ├────────┘
       │        │     │         │                └ a string with the cursor on the opening quote
       │        │     │         └ a string with the cursor in the middle
       │        │     └ there can be several of them
       │        └ there can be a character outside a string before
       └ there can be a text surrounded by a string before

This one works no matter the engine.


Additional information

The problematic regex:

\%(.*\%#\)\@=\%('[^']*'\)\@>\%(\%#.\+\)\@<=

is similar to the following one, which is a little simpler (because it gets rid of \%# and its results don't depend on the cursor position):

\%(.*B\)\@=\%(A[^A]*A\)\@>\%(B.*\)\@<=

I would expect the latter to match A B A when compared to this text:

AxA B AxA
  ^^^^^

But in practice it does not. However, if Vim is forced to use the old engine by prefixing the regex with \%#=1:

\%#=1\%(.*B\)\@=\%(A[^A]*A\)\@>\%(B.*\)\@<=
^^^^^

Then it does match A B A.

This can be confirmed with a shell command:

$ vim -es -Nu NONE +"put ='AxA B AxA'" +"put =search('\%#=2\m\%(.*B\)\@=\%(A[^A]*A\)\@>\%(B..\)\@<=')" +'2p' +'qa!'
0

$ vim -es -Nu NONE +"put ='AxA B AxA'" +"put =search('\%#=1\m\%(.*B\)\@=\%(A[^A]*A\)\@>\%(B..\)\@<=')" +'2p' +'qa!'
1

The first command outputs 0 because Vim does not match anything when using the new engine, while the second command outputs 1 because Vim does match something when using the old engine.

It turns out that the same is true for the original regex and text. That is, if the original regex is prefixed with \%#=1:

\%#=1\%(.*\%#\)\@=\%('[^']*'\)\@>\%(\%#.\+\)\@<=
^^^^^

It does match ' and ' while the cursor is on the n of and in the text:

a 'string' and 'another one' !
            ^
            cursor is here

It seems that the new engine does not support the combination of:

  • an atomic group
  • a variable-width lookahead

So, this regex does not match anything in the text AxA B AxA:

         the lookahead is variable-width
         |      atomic group
         v      vvv       vvvvv
\%#=2\%(.*B\)\@=\%(A[^A]*A\)\@>\%(B.*\)\@<=
├───┘├─────────┘               ├──────────┘
│    │                         └ positive lookbehind
│    └ positive lookahead
└ new NFA engine

But if any of the 2 previous conditions is not satisfied, the regex matches A B A.
For example, if the atomic group is removed, the regex matches:

\%#=2\%(.*B\)\@=A[^A]*A\%(B.*\)\@<=

Same thing if the lookahead is fixed-width, instead of variable-width:

\%#=2\%(..B\)\@=\%(A[^A]*A\)\@>\%(B.*\)\@<=
        ^^^
        fixed-width: three characters

Remaining question

If the old engine does not support the original regex, then why does it still match a string when the cursor is inside? Shouldn't it fail to match anything, just like when the cursor is outside a string?

Maybe the engine parses the regex in a special way, which by accident matches the inside of strings but not the outside...

Or maybe it's just a bug, because I have found out another regex with an atomic group, which gives another type of unexpected result when the new engine is used.

Let's assume you want bar in [foo](bar) when it's under the cursor. Here's a one possible regex:

\%(.*\%#\)\@=\%(\[.\{-}\](\zs.\{-}\ze)\)\@>\%(\%#.*\)\@<=

Which can be tested against this text:

one [two](three) (four) [five](six) seven
                                ^
                                cursor here

With the old engine, the match is six, which is expected.

But with the new engine, the match is [five](six), while I would expect just six (thanks to \zs and \ze). It's as if the new engine ignored \zs and \ze inside the atomic group.


Recommendations

Here are a few recommendations for the next time you find a regex which does not give the expected match(es).

Use the regex in a substitution command with the c flag: :%s///gc (an empty pattern means that the contents of the search register – @/ – is used; i.e. the last search). Answer no to each question, just to see which text is highlighted by IncSearch for each match. This command seems more reliable to test what a regex matches, rather than looking at the text highlighted by Search after running a / or ? command, or by looking at the text highlighted by IncSearch while writing a search pattern in a / or ? command.

As an example, consider this regex:

a\_.*b

And this text file:

a
a
b

If you search for the regex in a / command, all the text is highlighted by Search (and by IncSearch right before pressing Enter), which may make you think that there is only 1 match. But in reality, there are 2 matches: from the first a up to b, and from the second a up to b.

Even if you can get the exact number of matches by pressing n and counting how many locations the cursor jumps to before wrapping around the end of the buffer, or by removing the S flag from the 'shortmess' option which enables the display of a search count message on the command-line, it can be difficult to see exactly which text is matched, especially with overlapping matches, or with some regexes using lookarounds.

As another example, consider this regex:

^###\n.*\n###

And this file:

###
foo
###
bar
###
baz
###

If you only look at the Search highlighting, it seems there are only 2 matches; this one:

###
foo
###

and this one:

###
baz
###

But in reality, there is a third match in the middle, which is clearly shown by %s///gc:

###
bar
###

As a final example, consider this regex:

\%(.*\%#\)\@=`[^`]*`\%(\%#.*\)\@<=

And this text, while your cursor is anywhere on four:

one `two` three `four` five
        ^^^^^^^^
        highlighted by the Search highlight group

The real text matched by the regex is not the one highlighted by Search, but this one:

one `two` three `four` five
                ^^^^^^

The Search highlighting is probably unreliable because right after the search command, the cursor is moved, and thus the position expressed by \%# is altered.


If that doesn't help, then prefix your regex with \%#=1. This will force Vim to use its old regex engine, which "supports everything". From :h two-engines:

  1. An old, backtracking engine that supports everything.

If your regex now works as expected, it means that Vim was using the new NFA engine, and you were using some feature not supported by the latter.

Otherwise, try to prefix it with \%#=2 to force Vim to use the new NFA engine. If it now works as expected, it means that Vim was using the old engine, and you were using some feature not supported by the latter.

Note that \%#=1 and \%#=2 must be written at the very beginning of the regex. Even before something like \m or \v.


If that still doesn't help, then simplify your regex, and the testing text, as much as possible. Then try to translate this simplification into PCRE, and test the result on regex101.com.1

If it produces the expected match(es), it probably means that there is a bug in Vim. Have a look at :h todo and search for Regexp problems:, or just regex. If you don't find anything relevant, visit Vim's issue tracker, and type a query in the Filters field (remove is:open from the pre-populated text; your issue could have been fixed/closed). Maybe it is a known bug.

Otherwise, it probably means that you are missing some knowledge. In that case, regex101.com can help you by providing an explanation of the PCRE regex.

1 As an example, this Vim regex which exhibits the unexpected behavior described in the OP:

\%(.*B\)\@=\%(A[^A]*A\)\@>\%(B..\)\@<=

can be translated into PCRE like this:

(?=.*B)(?>A[^A]*A)(?<=B..)

Which regex101.com describes like this.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.