What are the methods of speeding up parsing of a big file using Ex?

Question

I did recently some performance test where I've created artificially 1GB of random text file by:

$ hexdump -C /dev/urandom | rev | head -c1G | pv > file

which has around 13mln of lines starting from:

|.....!m;}._dTO4x|  bd 9c aa 08 70 12 d6 b3  d7 3b f5 46 45 f4 43 87  00000000
|...DH...e..DN'..|  5e 8d 58 44 84 7f 5a 0d  56 1f dc 44 e4 72 6a de  01000000
|.B.~.^WE.g=...Y.|  1a 24 29 e7 ed e5 75 54  6d 76 d3 fe ec 00 95 ca  02000000

and basically I would like to print the last column, in other words remove everything else exempt the last column and print the result/buffer.

So my goal is match parsing times towards awk, grep, curl or cut within reasonable limits. I understand that other tools could be more suitable for this task (e.g. regex in grep is 4x faster) and it's difficult to compare these times, however as part of the learning process I'd like to see what it can be done in order to boost file processing by Ex.

To focus just on parsing times, I'm ignoring saving the file as well as printing the buffer to the standard output (%print).

Here is my progress how the times are dropping down (on MBR OS X):

• initial version (with regex operating without silent mode):

  $ time ex +'%s/^.*\s//g' -cq! file 
  real    4m36.202s
  user    3m53.264s
  sys     0m41.960s
  

  which seems it was using too much of system time

• with silent mode (which ignores invoking the screen):

  $ time ex +'%s/^.*\s//g' -scq! file
  real    3m39.654s
  user    3m37.176s
  sys     0m2.126s
  

• converting RegEx into normal keystrokes:

  $ time ex +'%norm $Bd0' -scq! file
  real    3m12.051s
  user    3m9.797s
  sys     0m1.957s
  

• now, I've tried to use a black hole register, but it didn't speed up anything, just opposite:

  $ time ex +'%norm $B"_d0' -scq! file
  real    3m21.275s
  user    3m18.800s
  sys     0m2.107s
  

• adding -u NONE makes - no difference

• added -n (for no swap file) - no difference

• adding --startuptime timing.out gives the following highest times (in msec):

  000.008  000.008: --- VIM STARTING ---
  004.124  003.007: shell init
  3372.778  3365.165: opening buffers
  

  which covers only <3.5 seconds of a starting time in total

• what else can I try?

Are there any other methods of speeding up Ex processing for larger files or some ways which are quicker than the others? Or basically this is how it is?

Answer 1

Switch to the old regexp engine, it's much faster than the new one for the particular regexp you're using:

set regexpengine=1

You can also turn off stuff you definitely won't be using (cf. wiki), but that will have a much smaller effect that the regexp engine. To sum it up, run ex with the following vimrc:

set regexpengine=1
set eventignore+=FileType
set bufhidden=unload
set buftype=nowrite
set undolevels=-1