1

I'm working on a plugin that needs to parse a YAML-like list and I can't quite nail the pattern needed to split the string. The pattern needs to match a comma delimited list that can have both singly quoted fields and embedded (doubled) single quotes.

The pattern in question is:

let s:pattern = '\m\%(^\|,\)\%(\s*\%([^''][^,]*\)\|\%(''\%(''''\|[^'']\)*''\)\s*\%(,\|$\)\)\@='

The crux of the issue seems to come down to choosing one of the following two inner branches to match either a field that is not quoted, or one that is:

[^''][^,]*

and:

''\%(''''\|[^'']\)*''

If I use only one of these sub-patterns I'm able to match successfully, however combining the two seems to fail for some reason. I've tried every variety of lookaround I can think of to no avail. At this point, I'm ready to give up using an expression and switch to a pushdown automaton.

TIA!

Edit: Example input/output (the last case is probably the gnarliest):

'a, b, c, d' -> 'a, b, c, d'
'a', 'b', 'c', 'd' -> 'a', 'b', 'c', 'd'
'a,''b'',c', d -> 'a,'b',c', d
  • How about some examples of data that should be matched? – B Layer Feb 29 at 5:47
  • 1
    I would bet money YAML is context-free, since it’s fairly recursive. You’re far better off using a real parser than regular expressions—your RE will fail at some point. – D. Ben Knoble Feb 29 at 16:08
  • Sorry about that! I've added some test cases. – Steven Stallion Feb 29 at 16:11
1

You have a few problems with your regex that I can see:

  • the zero-width forward lookup with @= covers essentially all of the match, so highlighting your matches only gets the commas or beginning of lines.
  • your \| is splitting the two alternatives in a way that the first ([^''][^,]*) isn't really anchored to the end (\%(,\|$\)), so it doesn't match all the way to a comma or end of line.
  • the [^''] match is too weak, in fact it can match a whitespace or a comma, it should be more restrictive.

I tweaked your regex and got to this:

let pattern = '\m\%(^\|,\)\s*\zs\([^'' \t,][^,]*\|''\%(''''\|[^'']\)*''\)\ze\s*\%(,\|$\)\@='

I'm using \zs and \ze to match the strings themselves, which makes the pattern useful for use in a search. It's also friendly to matchstr() or matchstrpos().

You can change that if you're planning to use it differently (for example, in split() or similar.)

Same as you, I'm starting at the beginning of the line, or a comma. Then skip whitespace and then branch into either a literal or a single quoted string. The match for the first character of the literal is stronger, we reject ' but also whitespace (space or tab) and a comma. (From the YAML specification, we could decide to reject even more characters here.)

After matching one of those, we skip whitespace again (if needed) and then doing a zero-width forward lookup for a comma or the end of the string. (The forward lookup can be important if you're searching repeatedly with n, since you want the comma to match again on the next use of the pattern.)

One last suggestion I'd have is to use "very magic" mode for the regular expression, which lets us get rid of 13 backslashes in total!

let pattern = '\v%(^|,)\s*\zs([^'' \t,][^,]*|''%(''''|[^''])*'')\ze\s*%(,|$)@='

I still see two issues with this pattern:

  1. If a literal has trailing whitespace, it will be returned as part of the expression. Perhaps we could tighten the first regex even more to ensure the last matched character will always be a non-blank character.
  2. When the string doesn't follow the correct syntax, the match may be misleading. For example, if there are literal characters after a quoted string that has a comma inside it, it will match the comma and think that the second part (including a single quote on its own) will form a literal. Depending on how you plan to use this regex (for example, calling matchstrpos() repeatedly?), you can use stronger anchors that will help you catch invalid syntax.

I tested the pattern with a search, by enabling highlighting, setting the search register and pressing n:

:set hlsearch
:let @/ = pattern
n

I recommend this as a way to debug regular expressions such as this one.


UPDATE: Since the problem is splitting a YAML list rather than using the pattern in a search, we need to take a slightly different approach.

Using split() is problematic, since the regex matches are not anchored, so it's really hard to match a comma that's guaranteed to be outside of a quoted string.

A better approach is to call matchlist() repeatedly, always starting the match at the position we last stopped.

In that case, we can always anchor the pattern on ^ (since matchlist() with a position will match that to that specific position, not the very beginning of the string.) We should also consume the ending comma and whitespace as we match. Since we now need the full match (to advance the position counter), we'll use a capture group to get the part that interests us, which is the YAML literal or quoted string.

Here's an implementation of this method:

function! SplitYamlList(expr)
    let pattern = '\v^\s*([^'' \t,]%([^,]*[^ \t,])?|''%(''''|[^''])*'')\s*%(,|$)'
    let result = []
    let pos = 0
    let end = len(a:expr)
    while pos < end
        let match = matchlist(a:expr, pattern, pos)
        if empty(match)
            throw "Parse error, can't match at position ".pos
        endif
        let [mstr, mgroup; _] = match
        call add(result, mgroup)
        let pos += len(mstr)
    endwhile
    return result
endfunction

This function will also be able to detect malformed items (such as a quoted string with trailing characters) and it will throw an error in that case, indicating the position in which the syntax error was found.

| improve this answer | |
  • 1
    Thanks! I think the catch I'm running into is this is being passed into a split, so the intent is to split on a comma, but only if it is not within a quoted field. I didn't even think of enabling very magic - that definitely makes everything more readable. I've updated the expression you provided to account for differences in a split, but I'm still seeing the same behavior I was before (I'm probably missing something). – Steven Stallion Feb 29 at 16:09
  • 1
    It's probably worth mentioning the only thing I really need to care about is splitting on the correct commas. I'm using other functions to ensure that substrings are properly trimmed and unquoted. – Steven Stallion Feb 29 at 16:21
1

It looks like there's an issue with how splits operate versus run of the mill pattern matching. I wasn't able to come up with a good explanation for why this problem exists, so I ended up taking a more iterative approach similar to @filbranden's suggestion:

const s:list_pattern =
        \ '^\s*(%(''%(''''|[^''])*'')|%([^'',][^,]*))\s*%(,|$)')

function! s:list_split(val) abort
  let list = []
  let idx = 0
  while idx < strlen(a:val)
    let [match, str ; _] = matchlist(a:val[idx:], s:list_pattern)
      let idx += strlen(match)
      call add(list, str)
  endwhile
  return list
endfunction
| improve this answer | |
  • 1
    I got to something quite similar! I'll update my answer shortly. – filbranden Mar 1 at 15:24
  • 1
    Wow what I had was almost identical! I think main difference is I'm passing the idx to matchlist() as an extra "pos" argument. I also added some explicit error handling when it doesn't match. Great job!!! – filbranden Mar 1 at 15:36
  • 1
    Ah! That's much cleaner than what I had (I think I ended up with the substring because I had originally started with matchstr rather than matchlist). Thanks so much for the help! – Steven Stallion Mar 1 at 15:46
  • 1
    and thank you for the very interesting question! – filbranden Mar 1 at 15:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.