1

What I'm trying to do is reformat some ugly JSON, replacing a pile of lines that are formatted:

[whitespace][number],\n

for example:

         5, 

I have thousands of those lines, but don't know how to search and replace while retaining that number which differs with each line. Would someone be able to help me out?

The end result I want is to group all sets of those lines on to a single line, like " 8, 6, 0, 6" etc.

6
  • 1
    Welcome to Vi and Vim!
    – filbranden
    May 26, 2020 at 18:24
  • 4
    Welcome indeed! Can you clarify one thing for me? Are you saying you have blocks of consecutive lines following this pattern, and you want to condense each block of lines into a single line?
    – D. Ben Knoble
    May 26, 2020 at 18:36
  • @D.BenKnoble: Yes, that's correct; thousands of lines of [spaces][number][comma]. I actually managed to resolve this with a bunch of search/replaces as the lines I want were unique in being indented 12 characters. I just replaced occurrences of \n[12 spaces] with a single space. It would nonetheless be very nice to know if there's a "proper" way to do that. Thanks! May 26, 2020 at 19:11
  • Oh - I think I should have clarified that. Yes, these are groups of lines separated by other distinct lines (additional data). The sequences of numbers are in individual groups of as much as 100-ish lines. May 26, 2020 at 19:16
  • 2
    @JacobEwing given that additional constraint, your way is probably easier to come up with interactively. My answer took a bit of thinking, but should work more generically in case this comes up again (plus you can tweak the pattern).
    – D. Ben Knoble
    May 26, 2020 at 19:20

2 Answers 2

3

This was harder than I thought it was going to be. Naïvely, I expected something like

global/^\s*\d\+,/join

to work, but it turns out this global applies one :join at a time to each matching line, so it doesn't work. (P.S. :join is the Ex command that joins two lines; its normal-mode equivalent is J.)

Here's a test case:

a
1,
b
c
    5,
        6,
    7,
    d
    e
    f
8,
    9,
  10,

If you run the command I tried, you get

a
1, b
c
    5, 6,
    7, d
    e
    f
8, 9,
  10,

Instead, we need to be smarter: we need to find the "blocks" of numbers and then :join them all at once. I had hoped to be able to do this with :global (pseudocode):

global /pattern for digit-lines/,/pattern for non-digit lines/-1 join

but I couldn't quite make a negative patterns work (and I'm now disappointed that vim doesn't have a "negation" operator that applies to a pattern to produce its exact negative—that, or I misunderstood \@! &co., and negative matching is just hard).

So I came up with the following solution (which should be quite fast, but will take time proportional to the length of the file):

function! GroupNums() abort
  const line_pat = '^\s*\d\+,$'
  1
  while search(line_pat, 'W')
    let first_line = line('.')
    +
    while getline('.') =~# line_pat && line('.') != line('$')
      +
    endwhile
    " now current line matches and is the last line, or doesn't match (and the
    " end of the block is one above)
    let last_line = getline('.') =~# line_pat ? line('$') : line('.') - 1
    execute first_line ',' last_line 'join'
  endwhile
endfunction

Create this function (you can type it interactively if you want, but you can also paste it into /tmp/code.vim and do :source /tmp/code.vim if you want), then switch to the buffer to fix and run :call GroupNums().

With the test-case, I get

a
1,
b
c
    5, 6, 7,
    d
    e
    f
8, 9, 10,

For the curious, the algorithm follows at a high level. Starting from the top of the file (1), we find the start of a block (search(line_pat, 'W')). We never wrap around, so we only process each block once. We record the start of the block (line('$')). Then, we move down one line at a time (+) scanning until we reach the end of the file or the end of the block (the inner while's condition: getline('.') =~# line_pat && line('.') != line('$')). The loop invariant guarantees that one or both of the following are true:

  1. We are at the end of the file;
  2. The current line does not match the pattern.

If the current line matches the pattern, then we must be at the end of the file, so we can use that for the end of the block (line('$')). Otherwise, we can use whatever the line before the current line is, because it matched and now the current line does not (line('.')-1). (Notice that it does not matter if we are the end of the file and the current line does not match; the second case covers this adequately.)

Having reached the end of the block, we do a :<start>,<end> join to group the lines together, and search for the next block. Of course, if we are already at the end, this search will fail because we do not wrap and because the pattern does not match already-joined lines, which the last line may be).

7
  • 1
    This one works quite well for me, and it's fairly simple...: :%s/\(\d\+\),\n\s*/\1,
    – filbranden
    May 26, 2020 at 21:01
  • 1
    @filbranden That's will join the last line with a number, even if there's no number in the next one, won't it? Maybe :%s/\(\d\+\),\n\s*\ze\d,/\1, (not tested). (Tested, looks like I'm more right than Fil -- champagne).
    – Biggybi
    May 26, 2020 at 21:24
  • 1
    Good job @Biggybi! 🎉 Yeah I guess using \n on a regex will be easier than using :join in a :global...
    – filbranden
    May 26, 2020 at 21:32
  • 1
    @filbranden Indeed! I don't mean to command you, but should maybe make it an answer as well.
    – Biggybi
    May 26, 2020 at 21:39
  • 1
    @filbranden it'll be an honor!
    – Biggybi
    May 26, 2020 at 21:50
3

You can do this with a relatively simple regular expression with :substitute:

:%s/\(\d\+,\)\n\s*\ze\d\+,/\1
  • :%s start a substitution on the whole file
  • \(\d\+\),\n\s* is the pattern to be replaced
    • \(\d\+,\) create a group (()) (to use later) of any number (\+) of numbers (\d) followed by ,
    • ,\n\s* match a , followed by a new line (\n), space (\s), anything until... (*)
  • \ze stop matching here: do not replace the following, but check if it's here
    • \d\+, ...numbers followed by a coma (to make sure we join only lines with numbers)
  • \1 replace the matching pattern with the group created earlier (that is, the x numbers followed by , from \(\d\+,\))

Courtesy to filbranden

4
  • 2
    Well done, you two; glad to see there was a simple solution. I struggled hard with global before writing the loops bc I didn't thing substitute could do it! I think this works over global because of the way :%s is processed, but I want to do some digging. Nit: you could include the comma in the capture and remove it in the replace.
    – D. Ben Knoble
    May 26, 2020 at 23:16
  • @D.BenKnoble indeed, capturing the comma is cleaner. Although I do not understand what you mean by /I think.*processed, I wish you luck in your research :P
    – Biggybi
    May 27, 2020 at 0:03
  • @D.BenKnoble One more is that this could potentially use \zs, so you actually only remove newline and space... %s/\d,\zs\n\s*\ze\d\+,//. I also don't know whether matching whitespace before the initial \d\+ will work. It would be nice to do so to ensure we're only matching cases that consist entirely of numbers, but it might break the solution since the \s would be matched on both lines... Really quite tricky and interesting problem!
    – filbranden
    May 27, 2020 at 0:43
  • @filbranden Both our substitute command and this one with \zs match numbers only, or I'm not sure what you mean. Btw, even nicer implementation with that \zs! I'm almost certain you can't improve it further.
    – Biggybi
    May 27, 2020 at 1:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.