I have a large data file of comma separated variables. I need to remove all lines where the first entry (a number up to four digits) is non-unique. A typical example of what I am working with is below


so in the brief example above, I wish to retain the first and forth lines and discard the rest.

I have been trying to use Vim Regex to achieve this


but of course that deletes all lines apart from the first

  • 2
    If you don't mind having the result sorted and have access to a shell I would use the bash command sort like this: sort -u -t, -k1,1 myfile I know it is not in vim but it is a pretty straighforward way to do that. See man sort -u is uniq result, -t, sets , as the delimitor and -k1,1 gives the fields to use for the sort.
    – statox
    Nov 2, 2017 at 16:54
  • Is the input guaranteed to be sorted, like in your example?
    – DJMcMayhem
    Nov 2, 2017 at 16:57
  • @DJMcMayhem the input is not guaranteed to be sorted, though it is easy to do so if necessary.. I require any one of the unique lines, so for example retaining the second and fifth lines would be equally useful
    – AlexD
    Nov 2, 2017 at 17:09
  • @statox thanks, I will need to have a play.. I am using powershell to do as you suggest and am getting the errors Missing argument in parameter list. I will figure it out, I'm sure
    – AlexD
    Nov 2, 2017 at 17:12
  • 1
    @statox Thanks, I tried that on my Linux machine and worked a treat
    – AlexD
    Nov 2, 2017 at 21:36

1 Answer 1


The g/ only solution would probably be


This leaves the last occurrence. We look for lines which have another line starting with the same digits an arbitrary number of characters ahead. The key is the \_.* in the middle which subsumes as much as possible.

The equivalent very magic version is


Another solution is to use :s and the sub-replace-expression \=,

let seen = {} | %s/^\(\d\{4}\).\+\n/\= has_key(seen, submatch(1)) ? '' : execute('let seen[submatch(1)] = 1').submatch(0) /

This leaves the first occurrence. It would be better to write a function if this is to be a commonly used operation.

  • 1
    thank you for your answer, unfortunately my file has so much data, vim just hangs when I attempt this, but I think that is a limitation of trying to do this within vim
    – AlexD
    Nov 2, 2017 at 22:31
  • Does the second approach with %s work? It should be much more efficient, about O(n). I guess I should have mentioned that the first approach is very inefficient, O(n^2) at least, potentially exponential depending on vim's internals
    – Mass
    Nov 2, 2017 at 23:00
  • is that entered in ex mode? If so, vim complains it is not an editor command
    – AlexD
    Nov 2, 2017 at 23:37
  • Yes, ex mode or on the command line (:). Works for me, maybe you have an old version of vim? execute( was added a bit over a year ago.
    – Mass
    Nov 2, 2017 at 23:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.