0

I have a large data file of comma separated variables. I need to remove all lines where the first entry (a number up to four digits) is non-unique. A typical example of what I am working with is below

1062,"Mark","Michaelson","1"
1062,"Mark","Michaelson","2"
1062,"Mark","Michaelson","4120"
1075,"Dan","Danson",15"
1075,"Dan","Danson",185"

so in the brief example above, I wish to retain the first and forth lines and discard the rest.

I have been trying to use Vim Regex to achieve this

:0/^\w*/+,$g//d

but of course that deletes all lines apart from the first

  • 2
    If you don't mind having the result sorted and have access to a shell I would use the bash command sort like this: sort -u -t, -k1,1 myfile I know it is not in vim but it is a pretty straighforward way to do that. See man sort -u is uniq result, -t, sets , as the delimitor and -k1,1 gives the fields to use for the sort. – statox Nov 2 '17 at 16:54
  • Is the input guaranteed to be sorted, like in your example? – DJMcMayhem Nov 2 '17 at 16:57
  • @DJMcMayhem the input is not guaranteed to be sorted, though it is easy to do so if necessary.. I require any one of the unique lines, so for example retaining the second and fifth lines would be equally useful – AlexD Nov 2 '17 at 17:09
  • @statox thanks, I will need to have a play.. I am using powershell to do as you suggest and am getting the errors Missing argument in parameter list. I will figure it out, I'm sure – AlexD Nov 2 '17 at 17:12
  • 1
    @statox Thanks, I tried that on my Linux machine and worked a treat – AlexD Nov 2 '17 at 21:36
1

The g/ only solution would probably be

:g/^\(\d\{4}\).\+\n\_.*\%(^\1\D\)/d

This leaves the last occurrence. We look for lines which have another line starting with the same digits an arbitrary number of characters ahead. The key is the \_.* in the middle which subsumes as much as possible.

The equivalent very magic version is

:g/\v^(\d{4}).+\n\_.*%(^\1\D)/d

Another solution is to use :s and the sub-replace-expression \=,

let seen = {} | %s/^\(\d\{4}\).\+\n/\= has_key(seen, submatch(1)) ? '' : execute('let seen[submatch(1)] = 1').submatch(0) /

This leaves the first occurrence. It would be better to write a function if this is to be a commonly used operation.

  • 1
    thank you for your answer, unfortunately my file has so much data, vim just hangs when I attempt this, but I think that is a limitation of trying to do this within vim – AlexD Nov 2 '17 at 22:31
  • Does the second approach with %s work? It should be much more efficient, about O(n). I guess I should have mentioned that the first approach is very inefficient, O(n^2) at least, potentially exponential depending on vim's internals – Mass Nov 2 '17 at 23:00
  • is that entered in ex mode? If so, vim complains it is not an editor command – AlexD Nov 2 '17 at 23:37
  • Yes, ex mode or on the command line (:). Works for me, maybe you have an old version of vim? execute( was added a bit over a year ago. – Mass Nov 2 '17 at 23:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.