How to edit files non-interactively (e.g. in pipeline)?

Question

I would like to edit files passed in pipeline input using vim in non-interactive way or edit files in-place (similar to sed).

Few examples using sed:

$ sed -i'.bak' s/foo/test/g file   # Edit file in-place.
$ cat file | sed s/foo/test/g      # Parse file in pipeline.

However I could read in here, that:

sed is a Stream EDitor, not a file editor. Nevertheless, people everywhere tend to abuse it for trying to edit files. It doesn't edit files.

Secondly some options such as in-place editing (-i) is portable.

How the same functionality can be achieved in vim?

Answer 1

To edit file non-interactively using ex (vi is the visual mode for ex), you can use +{command} or -c {command} parameters which allows you to execute the vi commands after the first file has been read.

The ex is a standard command-line editor (similar to ed).

There is also vipe (a Vim command pipe editor) should be used which is part of moreutils package and it will allows you to run your editor in the middle of a unix pipeline and edit the data that is being piped between programs.

Examples

Simple standard input and output using pipes can be achieved by this shell syntax:

$ ex -sc'%p|q!' <(echo Example)
$ echo Example | ex -sc'%p|q!' /dev/stdin

Here is simple example how to print the file after substitution:

$ ex /etc/hosts +%s/127/128/ge -sc'%p|q!'

More examples for editing files in-place:

$ ex +'%s/127/128/g' -cswq file
$ ex -sc '%s/olddomain\.com/newdomain.com/g|x' file
$ printf '%s\n' 'g/olddomain\.com/s//newdomain.com/g' w q | ex -s file
$ ex -s "$file" <<< $'g/old/s//new/g\nw\nq'
$ ex -sc 'argdo %s/old/new/ge|x' ./**
$ find . -type f -exec ex -sc '%s/old/new/g|x' {} \;

You can also use -s {scriptin} so the commands are loaded from the file, in example:

$ printf "%s\n" '%s/foo/test/ge' 'wq' > cmds.vim
$ vim -s cmds.vim -es file

or using I/O redirection:

$ vim file < cmds.vim

To edit one file and save the changes to another, check the following examples:

$ ex +%s/127/128/g -sc'wq! new_file' /etc/hosts
$ cat /etc/hosts /etc/fstab | vim - -es '+:%s/foo/test/g' '+:wq! file3'

---

More practical examples.

Real live example from the RPM specification:

vim -E -s Makefile <<-EOF
   :%substitute/CFLAGS = -g$/CFLAGS =-fPIC -DPIC -g/
   :%substitute/CFLAGS =$/CFLAGS =-fPIC -DPIC/
   :%substitute/ADAFLAGS =$/ADAFLAGS =-fPIC -DPIC/
   :update
   :quit
EOF

Extracting html tags:

ex -s +'bufdo!/<div.*id=.the_div_id/norm nvatdggdG"2p' +'bufdo!%p' -cqa! *.html

Removing XML tags:

ex -s +'%s/<[^>].\{-}>//ge' +%p +q! file.txt

Removing style tag from the header and print the parsed output:

curl -s http://example.com/ | ex -s +'/<style.*/norm nvatd' +%p -cq! /dev/stdin

Parse html with multiple complex rules:

ex -V1 $PAGE <<-EOF
  " Correcting missing protocol, see: https://github.com/wkhtmltopdf/wkhtmltopdf/issues/2359 "
  %s,'//,'http://,ge
  %s,"//,"http://,ge
  " Correcting relative paths, see: https://github.com/wkhtmltopdf/wkhtmltopdf/issues/2359 "
  %s,[^,]\zs'/\ze[^>],'http://www.example.com/,ge
  %s,[^,]\zs"/\ze[^>],"http://www.example.com/,ge
  " Remove the margin on the left of the main block. "
  %s/id="doc_container"/id="doc_container" style="min-width:0px;margin-left : 0px;"/g
  %s/<div class="outer_page/<div style="margin: 0px;" class="outer_page/g
  " Remove useless html elements. "
  /<div.*id="global_header"/norm nvatd
  wq " Update changes and quit.
EOF

Even more examples:

• How can I replace a string with another string in a variable, a stream, a file, or in all the files in a directory?
• What is the EX-mode for batch processing for? at superuser SE

---

See also:

• Convert tabs to spaces in every file of a directory? at SO
• Remove trailing whitespaces from all files recursively? at SO
• How to write whole buffer to standard output from the command line? at Vim SE
• Is it possible to execute a VIM script in a non-interactive mode? at unix SE
• Pipe Vim buffer to stdout at SO
• write buffer content to stdout at SO
• Learning the vi Editor/Vim/Modes
• How to remove inner content of html tag conditionally?

---