delete repeated line in vim, keep the n-th line in Vim

Question

I want to ask how to delete repeated lines in vim. For now, I have a file which contained 12740 lines. There are only 98 unique lines, and each line is duplicated 129 times.

I have tried:

g/^\(.*\)$\n\1$/d

But my problem is, some unique lines are the same to each other, I don't know which lines are those.

When I input that command, my file becomes 95 lines, not 98 which I want. (3 lines are the same to each other, I guess, so it's also deleted).
Any other solution?
I'll edit if my question is still not clear enough.

My file:
-0.00697568 - line 1
-0.00697568 - line 2
...
0.00697568 - line 131
0.00697568 - line 132
0.00697568 - line 133
...
0.020909881 - line 261
0.020909881 - line 262
...
And so on.
I need to keep the 130n+1 line, from n=0 until n=97.

Edit I need the lines to keep their order so a solution sorting the lines would not be an acceptable one.

Answer 1

To delete n lines starting from every line, you can do:

:%norm ndd

For example, if I started with the numbers 1 to 10 repeated 5 times:

1
1
1
1
1
2
2
2
2
2
3
3
3
3
3
4
4
4
4
4
5
5
5
5
5
6
6
6
6
6
7
7
7
7
7
8
8
8
8
8
9
9
9
9
9
10
10
10
10
10

And I do:

:%norm 4dd

I get:

1
2
3
4
5
6
7
8
9
10

So, try :%norm 129dd.

This is one of those instances where you want :% instead of :g. :g tags the lines so that modifications while running the g command aren't counted for running the remaining iterations. %, on the other hand, will run from every line from 1 to the end, and it will only look at the current line numbers at each iteration.

Answer 2

I usually solve problems like this with a macro. Position yourself at the top of your file, then start recording with qq. The first "q" starts the recording. The second "q" doesn't have to be "q". It can be any register letter, I just use "q" because it's convenient.

Press j to go to the second line, then 129dd to delete all the duplicates. Then press q to stop recording the macro.

Now you've taught vim how to do the operation once, and stored it in the q macro. You just have to replay the macro 97 more times, by typing 97@q.

I really like macros for this sort of problem, because it's relatively easy to do the action once manually, even if it's difficult to precisely describe how you do it in an easily repeatable manner. I can just record myself and let vim do the repetitious work.

Answer 3

Let's get the exact number of unique lines first.

 :sort u | echo line('$')

This command returns the total unique lines. In your case, this might be 95.

Let's do that with a macro. Empty register n first. Press qnq.

1. Start recording by pressing qn. (n is the register name I have chosen)

   qn
   

2. Go to first line.

   :1
   

3. Press Y (same as y$) to copy till end of line

   Y
   

4. Type :g/ and then press Ctrl+R and ". The line is pasted as search pattern. Then, type d.

   :g/Ctrl+R"/d
   

5. All similar lines will be deleted including that one. Clear K register by pressing qkq. Now, let's copy the line to K register.

   :let @K="@/\n"
   

6. Press q to stop recording.

   q
   

7. Now, you can press count and @n. (We have obtained the count in the beginning through sort u and echo commands.)

   98@n
   

8. File will be emptied and all the contents will be stored in @K. Go to insert mode and type Ctrl+R and k. The contents of register is inserted there. i Ctrl+R k Now, the unique lines are pasted in file itself.

Edit: Recursive mapping is difficult for me to work with. This answer also gives what the user wanted.

Answer 4

An elegant one which shows the power of g command:

:g/\v^(\-?\d\.\d+)(\r?\n\1)+$/d

Just replace the pattern in the first parentheses to meet your other needs. It works even if the duplicated lines are not of the same number. And the order will be preserved also.