5

I did recently some performance test where I've created artificially 1GB of random text file by:

$ hexdump -C /dev/urandom | rev | head -c1G | pv > file

which has around 13mln of lines starting from:

|.....!m;}._dTO4x|  bd 9c aa 08 70 12 d6 b3  d7 3b f5 46 45 f4 43 87  00000000
|...DH...e..DN'..|  5e 8d 58 44 84 7f 5a 0d  56 1f dc 44 e4 72 6a de  01000000
|.B.~.^WE.g=...Y.|  1a 24 29 e7 ed e5 75 54  6d 76 d3 fe ec 00 95 ca  02000000

and basically I would like to print the last column, in other words remove everything else exempt the last column and print the result/buffer.

So my goal is match parsing times towards awk, grep, curl or cut within reasonable limits. I understand that other tools could be more suitable for this task (e.g. regex in grep is 4x faster) and it's difficult to compare these times, however as part of the learning process I'd like to see what it can be done in order to boost file processing by Ex.

To focus just on parsing times, I'm ignoring saving the file as well as printing the buffer to the standard output (%print).

Here is my progress how the times are dropping down (on MBR OS X):

  • initial version (with regex operating without silent mode):

    $ time ex +'%s/^.*\s//g' -cq! file 
    real    4m36.202s
    user    3m53.264s
    sys     0m41.960s
    

    which seems it was using too much of system time

  • with silent mode (which ignores invoking the screen):

    $ time ex +'%s/^.*\s//g' -scq! file
    real    3m39.654s
    user    3m37.176s
    sys     0m2.126s
    
  • converting RegEx into normal keystrokes:

    $ time ex +'%norm $Bd0' -scq! file
    real    3m12.051s
    user    3m9.797s
    sys     0m1.957s
    
  • now, I've tried to use a black hole register, but it didn't speed up anything, just opposite:

    $ time ex +'%norm $B"_d0' -scq! file
    real    3m21.275s
    user    3m18.800s
    sys     0m2.107s
    
  • adding -u NONE makes - no difference

  • added -n (for no swap file) - no difference
  • adding --startuptime timing.out gives the following highest times (in msec):

    000.008  000.008: --- VIM STARTING ---
    004.124  003.007: shell init
    3372.778  3365.165: opening buffers
    

    which covers only <3.5 seconds of a starting time in total

  • what else can I try?

Are there any other methods of speeding up Ex processing for larger files or some ways which are quicker than the others? Or basically this is how it is?

3

Switch to the old regexp engine, it's much faster than the new one for the particular regexp you're using:

set regexpengine=1

You can also turn off stuff you definitely won't be using (cf. wiki), but that will have a much smaller effect that the regexp engine. To sum it up, run ex with the following vimrc:

set regexpengine=1
set eventignore+=FileType
set bufhidden=unload
set buftype=nowrite
set undolevels=-1
  • The LargeFile plugin tries to automatically optimize Vim settings when opening large files. – Vitor Oct 23 '15 at 17:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.