3

I want to ask how to delete repeated lines in vim. For now, I have a file which contained 12740 lines. There are only 98 unique lines, and each line is duplicated 129 times.

I have tried:

g/^\(.*\)$\n\1$/d

But my problem is, some unique lines are the same to each other, I don't know which lines are those.

When I input that command, my file becomes 95 lines, not 98 which I want. (3 lines are the same to each other, I guess, so it's also deleted).
Any other solution?
I'll edit if my question is still not clear enough.

My file:
-0.00697568 - line 1
-0.00697568 - line 2
...
0.00697568 - line 131
0.00697568 - line 132
0.00697568 - line 133
...
0.020909881 - line 261
0.020909881 - line 262
...
And so on.
I need to keep the 130n+1 line, from n=0 until n=97.

Edit I need the lines to keep their order so a solution sorting the lines would not be an acceptable one.

  • 3
    If some "unique" lines are the same, how are they unique? O.o What makes them unique? – muru May 20 '16 at 7:39
  • 3
    It's quite complex (as in O(N²)) to delete duplicates when the file is not sorted. In the case your file is sorted, or can be sorted, your question becomes a duplicate of this one: vi.stackexchange.com/questions/5362/… – Luc Hermitte May 20 '16 at 7:48
  • @muru I think I can say it is like a 2-dimensional array: 98x130. I need to keep 98 lines and delete the other 129 lines, some of the line is the same to each other (but I still need to keep it 98). – mfakhrusy May 20 '16 at 7:57
  • 5
    @v8areu: You should rephrase your question to indicate that clearly: it is much easier to do that :-) Something like: qaq then qa129dd@aq then @a should do the trick. – statox May 20 '16 at 8:10
  • 2
    @LucHermitte on second thought, it might not be: it seems they want to reduce "1,1,1,2,2,2,3,3,3,3,3,3,5,5,5" to "1,2,3,3,5" (not "1,2,3,5" as uniq would have done). – muru May 20 '16 at 8:37
8

To delete n lines starting from every line, you can do:

:%norm ndd

For example, if I started with the numbers 1 to 10 repeated 5 times:

1
1
1
1
1
2
2
2
2
2
3
3
3
3
3
4
4
4
4
4
5
5
5
5
5
6
6
6
6
6
7
7
7
7
7
8
8
8
8
8
9
9
9
9
9
10
10
10
10
10

And I do:

:%norm 4dd

I get:

1
2
3
4
5
6
7
8
9
10

So, try :%norm 129dd.

This is one of those instances where you want :% instead of :g. :g tags the lines so that modifications while running the g command aren't counted for running the remaining iterations. %, on the other hand, will run from every line from 1 to the end, and it will only look at the current line numbers at each iteration.

  • 4
    If the file is really organized this way, it'd be safer to use one of the uniq emulation: vi.stackexchange.com/questions/5362/…, no need to detect/count the number of replications. – Luc Hermitte May 20 '16 at 8:24
  • 4
    @LucHermitte yes, it's just re-implementing plain ol' uniq in Vim. I suppose it's a matter of how the question should have been phrased: remove duplicates, or keep every nth line? – muru May 20 '16 at 8:26
  • 3
    I've missed the fact that the OP observes that too many lines are removed when he applied uniq. I guess he has duplicated series of lines he wish to keep. In other word, he seems to be looking for "keep one line every n" and not uniq. My mistake. You're answering his needs. – Luc Hermitte May 20 '16 at 8:39
  • 2
    I just know the %norm command can do the trick just fine, the other answer is also right, but this answer is neater. Thanks guys. – mfakhrusy May 20 '16 at 10:00
  • 2
    @Wildcard no. (Try it, you'll get an empty buffer.) g works by tagging every line that matches, and then running the command on only those lines, irrespective of whether those lines were modified, displaced or deleted by the command. % just runs from the first line to the last. So, in this case, g/^/ norm 4dd, for example, would delete lines 1, 2, 3, 4 while at the first line, then move to the now-deleted second line, and delete lines 2, 3, 4, 5 (instead of moving to 5, and deleting 5, 6, 7, 8). See vi.stackexchange.com/a/3967/205. Compare: g/\%<8l/ norm 2dd and 1,7 norm 2dd. – muru May 21 '16 at 17:35
4

I usually solve problems like this with a macro. Position yourself at the top of your file, then start recording with qq. The first "q" starts the recording. The second "q" doesn't have to be "q". It can be any register letter, I just use "q" because it's convenient.

Press j to go to the second line, then 129dd to delete all the duplicates. Then press q to stop recording the macro.

Now you've taught vim how to do the operation once, and stored it in the q macro. You just have to replay the macro 97 more times, by typing 97@q.

I really like macros for this sort of problem, because it's relatively easy to do the action once manually, even if it's difficult to precisely describe how you do it in an easily repeatable manner. I can just record myself and let vim do the repetitious work.

3

Let's get the exact number of unique lines first.

 :sort u | echo line('$')

This command returns the total unique lines. In your case, this might be 95.

Let's do that with a macro. Empty register n first. Press qnq.

  1. Start recording by pressing qn. (n is the register name I have chosen)

    qn
    
  2. Go to first line.

    :1
    
  3. Press Y (same as y$) to copy till end of line

    Y
    
  4. Type :g/ and then press Ctrl+R and ". The line is pasted as search pattern. Then, type d.

    :g/Ctrl+R"/d
    
  5. All similar lines will be deleted including that one. Clear K register by pressing qkq. Now, let's copy the line to K register.

    :let @K="@/\n"
    
  6. Press q to stop recording.

    q
    
  7. Now, you can press count and @n. (We have obtained the count in the beginning through sort u and echo commands.)

    98@n
    
  8. File will be emptied and all the contents will be stored in @K. Go to insert mode and type Ctrl+R and k. The contents of register is inserted there. i Ctrl+R k Now, the unique lines are pasted in file itself.

Edit: Recursive mapping is difficult for me to work with. This answer also gives what the user wanted.

  • 1
    the 5th phase, I think my similar lines is not deleted according to your answer. Or perhaps I do things wrongly? This is my first time using macro though, I would be very glad if you can edit the answer to become more "newbie-friendly" – mfakhrusy May 20 '16 at 8:02
  • 2
    That's an interesting approach, but it could be improved: as you don't provide an end condition the macro will loop indefinitely (It is of course possible to stop it with ctrl-c. Also it changes the order of the lines (the last one becomes the first one). – statox May 20 '16 at 8:09
  • 1
    Don't know how to stop looping macro with a condition (without using a function). So, editing answer by getting the unique lines from :sort u command and then proceeding macro. – SibiCoder May 20 '16 at 10:37
  • 2
    @SibiCoder I think I prefer: ggqqj129dd@qq@q will do just as same as you do. I just managed to learn about this kind of stuff :) – mfakhrusy Jun 2 '16 at 3:31
1

An elegant one which shows the power of g command:

:g/\v^(\-?\d\.\d+)(\r?\n\1)+$/d

Just replace the pattern in the first parentheses to meet your other needs. It works even if the duplicated lines are not of the same number. And the order will be preserved also.

  • wow, I never think of this, great addition! – mfakhrusy Sep 28 '16 at 22:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.