I'm trying to remove (non-consecutive) duplicate lines in a file I have loaded in vim (without sorting them). In several places on the web I've found the nugget awk '!x[$0]++' as a pipe command that will do this, and indeed if I run this from my shell directly, i.e. cat myfile | awk '!x[$0]++', it works as expected.

However, in vim, I've tried selecting some lines in line-wise visual mode, then typing :'<,'>!awk '!x[$0]++', and I get an error of cannot read file /tmp/nvimQFXXXX/31.

My guess would that in some way my syntax is not quite right, and maybe I need to somehow escape some of the characters? I've tried \ to escape various characters though, and I can't figure it out. I've looked in the help, but I don't see anything about escaping characters.

Bonus points if you can wrap this up in a mapping and/or function; that's my end goal!


From :h :range!

            Filter {range} lines through the external program
            {filter}.  Vim replaces the optional bangs with the
            latest given command and appends the optional [arg].

The second ! is replaced with last shell cmd, I guess it expand to some command that read the temp file, escape it should work:

'<,'>! awk '\!a[$0]++'

Now wrapper it in a command and map:

vnoremap <leader>uu :UnsortUniq<cr>
command -range UnsortUniq <line1>,<line2>! awk '\!a[$0]++'

This can also be done without awk, although it's ugly compared to awk:

let g:d={} | g/^/ if has_key(g:d, getline('.')) | d | else | let g:d[getline('.')]=1 | endif

It gets uglier if you wrapper it in a command that support range:

command -range UnsortUniq2 let g:d={} | <line1>,<line2>g/^/
      \ if has_key(g:d, getline('.')) | d | else | let g:d[getline('.')]=1 | endif
| improve this answer | |
  • Excellent! This is exactly what I was looking for. Thanks. – Andrew Ferrier Jan 12 at 21:43

When I run :%!awk '!x[$0]++' in my vim, I get E34: No previous command.

I fixed it by escaping the ! (see :help :!).


" ~/.vim/autoload/uniq.vim
function uniq#operator(type, ...) abort
  let l:visual = a:0
  if l:visual
    '<,'>!awk '\!x[$0]++'
    '[,']!awk '\!x[$0]++'

" ~/.vim/plugin/uniq.vim
nnoremap U :set operatorfunc=uniq#operator<CR>g@
vnoremap U :<C-u>call uniq#operator(visualmode(), 1)<CR>

(The key U is somewhat arbitrary. Replace at your leisure.)

This follows my operator plugins format, and defines a new operator. It works with any motion, though of course will always be linewise. It also works with visual mode.

| improve this answer | |
  • This is great, excellent that you wrapped it up in an operator too. I've accepted the other answer as it provided a command, but yours is obviously very useful for anyone who prefers an operator approach. – Andrew Ferrier Jan 12 at 21:42

It's possible to get Vim to handle all the necessary escaping for you by calling shellescape(). If you pass in a non-zero value for its second argument, this will escape characters in a manner suitable for passing as a command line argument to a ! command or filter:

:exe "'<,'>!awk" shellescape('!x[$0]++', 1)

You can wrap this into a command that takes a range (or defaults to the entire file) like this:

command -range=% RemoveDups execute '<line1>,<line2>!awk' shellescape('!x[$0]++', 1)

And you can invoke this command from a visual mode mapping like this:

xnoremap <leader>rd :RemoveDups<CR>
| improve this answer | |
  • Cool. That's actually the original question I had, thanks. – Andrew Ferrier Jan 14 at 17:39
  • Adding silent! in front of it also helps – SergioAraujo Jan 17 at 23:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.