I did recently some performance test where I've created artificially 1GB of random text file by:
$ hexdump -C /dev/urandom | rev | head -c1G | pv > file
which has around 13mln of lines starting from:
|.....!m;}._dTO4x| bd 9c aa 08 70 12 d6 b3 d7 3b f5 46 45 f4 43 87 00000000
|...DH...e..DN'..| 5e 8d 58 44 84 7f 5a 0d 56 1f dc 44 e4 72 6a de 01000000
|.B.~.^WE.g=...Y.| 1a 24 29 e7 ed e5 75 54 6d 76 d3 fe ec 00 95 ca 02000000
and basically I would like to print the last column, in other words remove everything else exempt the last column and print the result/buffer.
So my goal is match parsing times towards awk
, grep
, curl
or cut
within reasonable limits. I understand that other tools could be more suitable for this task (e.g. regex in grep
is 4x faster) and it's difficult to compare these times, however as part of the learning process I'd like to see what it can be done in order to boost file processing by Ex.
To focus just on parsing times, I'm ignoring saving the file as well as printing the buffer to the standard output (%print
).
Here is my progress how the times are dropping down (on MBR OS X):
initial version (with regex operating without silent mode):
$ time ex +'%s/^.*\s//g' -cq! file real 4m36.202s user 3m53.264s sys 0m41.960s
which seems it was using too much of system time
with silent mode (which ignores invoking the screen):
$ time ex +'%s/^.*\s//g' -scq! file real 3m39.654s user 3m37.176s sys 0m2.126s
converting RegEx into normal keystrokes:
$ time ex +'%norm $Bd0' -scq! file real 3m12.051s user 3m9.797s sys 0m1.957s
now, I've tried to use a black hole register, but it didn't speed up anything, just opposite:
$ time ex +'%norm $B"_d0' -scq! file real 3m21.275s user 3m18.800s sys 0m2.107s
adding
-u NONE
makes - no difference- added
-n
(for no swap file) - no difference adding
--startuptime timing.out
gives the following highest times (in msec):000.008 000.008: --- VIM STARTING --- 004.124 003.007: shell init 3372.778 3365.165: opening buffers
which covers only <3.5 seconds of a starting time in total
what else can I try?
Are there any other methods of speeding up Ex processing for larger files or some ways which are quicker than the others? Or basically this is how it is?