How can I search/replace and retain a part that varies between the results?

Question

What I'm trying to do is reformat some ugly JSON, replacing a pile of lines that are formatted:

[whitespace][number],\n

for example:

         5, 

I have thousands of those lines, but don't know how to search and replace while retaining that number which differs with each line. Would someone be able to help me out?

The end result I want is to group all sets of those lines on to a single line, like " 8, 6, 0, 6" etc.

Answer 1

This was harder than I thought it was going to be. Naïvely, I expected something like

global/^\s*\d\+,/join

to work, but it turns out this global applies one :join at a time to each matching line, so it doesn't work. (P.S. :join is the Ex command that joins two lines; its normal-mode equivalent is J.)

Here's a test case:

a
1,
b
c
    5,
        6,
    7,
    d
    e
    f
8,
    9,
  10,

If you run the command I tried, you get

a
1, b
c
    5, 6,
    7, d
    e
    f
8, 9,
  10,

---

Instead, we need to be smarter: we need to find the "blocks" of numbers and then :join them all at once. I had hoped to be able to do this with :global (pseudocode):

global /pattern for digit-lines/,/pattern for non-digit lines/-1 join

but I couldn't quite make a negative patterns work (and I'm now disappointed that vim doesn't have a "negation" operator that applies to a pattern to produce its exact negative—that, or I misunderstood \@! &co., and negative matching is just hard).

So I came up with the following solution (which should be quite fast, but will take time proportional to the length of the file):

function! GroupNums() abort
  const line_pat = '^\s*\d\+,$'
  1
  while search(line_pat, 'W')
    let first_line = line('.')
    +
    while getline('.') =~# line_pat && line('.') != line('$')
      +
    endwhile
    " now current line matches and is the last line, or doesn't match (and the
    " end of the block is one above)
    let last_line = getline('.') =~# line_pat ? line('$') : line('.') - 1
    execute first_line ',' last_line 'join'
  endwhile
endfunction

Create this function (you can type it interactively if you want, but you can also paste it into /tmp/code.vim and do :source /tmp/code.vim if you want), then switch to the buffer to fix and run :call GroupNums().

With the test-case, I get

a
1,
b
c
    5, 6, 7,
    d
    e
    f
8, 9, 10,

---

For the curious, the algorithm follows at a high level. Starting from the top of the file (1), we find the start of a block (search(line_pat, 'W')). We never wrap around, so we only process each block once. We record the start of the block (line('$')). Then, we move down one line at a time (+) scanning until we reach the end of the file or the end of the block (the inner while's condition: getline('.') =~# line_pat && line('.') != line('$')). The loop invariant guarantees that one or both of the following are true:

1. We are at the end of the file;
2. The current line does not match the pattern.

If the current line matches the pattern, then we must be at the end of the file, so we can use that for the end of the block (line('$')). Otherwise, we can use whatever the line before the current line is, because it matched and now the current line does not (line('.')-1). (Notice that it does not matter if we are the end of the file and the current line does not match; the second case covers this adequately.)

Having reached the end of the block, we do a :<start>,<end> join to group the lines together, and search for the next block. Of course, if we are already at the end, this search will fail because we do not wrap and because the pattern does not match already-joined lines, which the last line may be).

Answer 2

You can do this with a relatively simple regular expression with :substitute:

:%s/\(\d\+,\)\n\s*\ze\d\+,/\1

• :%s start a substitution on the whole file
• \(\d\+\),\n\s* is the pattern to be replaced
  • \(\d\+,\) create a group (()) (to use later) of any number (\+) of numbers (\d) followed by ,
  • ,\n\s* match a , followed by a new line (\n), space (\s), anything until... (*)
• \ze stop matching here: do not replace the following, but check if it's here
  • \d\+, ...numbers followed by a coma (to make sure we join only lines with numbers)
• \1 replace the matching pattern with the group created earlier (that is, the x numbers followed by , from \(\d\+,\))

Courtesy to filbranden