I would like to use vim to do the following operation in vim: I would like to determine the number of entries with avg_deg greater then "'avg_deg': 5.50" from a simple .log file as well as the average of all avg_degs. Afterwards I would like to repeat the same for all entries smaller than "5.00" (the smallest entry in the log is "4.63" and the biggest is at "6.04"). Simply searching and counting the values manually is not possible in realistic time since the file is about 31 MB big.

The file contains many entries with some string in apostrophes with colon following a number but it is necessary to observe only numbers directly following the string "'avg_deg': ".

I would have copied parts of the file as example but even single variables are too big to put them here - at least my browser dies and the website doesn't react anymore when I do so.

  • 1
    Maybe a sample of your document would help
    – Zorzi
    Apr 17 '20 at 17:26
  • Tried to add just one array of the .log and intended to add further parts of the .log - but already that single array killed my browser after the website decided not to respond anymore... so basically you find lots of dictionaries with arrays with dictionaries with entries. The "'avg_deg': 5.50" appears for each test case twice in the file but that shouldn't matter since for the intended operations the first result has to be divided by 2 and the second should be even exactly the same. Actually I intended to save the result as .json but python just failed doing it on the variable size.
    – baxbear
    Apr 17 '20 at 17:39
  • You could create simplified example of the data, though
    – Maxim Kim
    Apr 17 '20 at 18:13
  • 1
    This is not fundamentally a vim exercise... use grep (or your favorite tools)—it's likely to be faster, not consume as much memory (31M to load the file!), and allow you trim out more correctly your search pattern. Example: <big-file grep -c "avg_deg': [5]\.[56789]". If you need average, awk is good but lacks fixed/floating point support, so use something like sed 's/\([[:digit:]]\)\.\([[:digit:]]\)/\1\2/g' to convert everything to 3-digit ints
    – D. Ben Knoble
    Apr 17 '20 at 19:33
  • Thank you very much! In my case grep needs to use -o since some of the results are in the same line. Afterwards I count the output with | wc -l => <.log grep -o "'avg_deg': [4]" | wc -l and <.log grep -o "'avg_deg': [5]\.[56789][0123456789][0123456789]*" | wc -l and finally <.log grep -o "'avg_deg': [6]" | wc -l. I still have to check and understand the sed command of yours to use it.
    – baxbear
    Apr 18 '20 at 21:26

If it is one off task you can try this

  1. avgs greater than 5.50 till 5.99 (remember number of matches)
:%s/avg_deg': [5]\.[56789]\d//gn
  1. avgs greater than 6.00 (remember number of matches)
:%s/avg_deg': [6789]\.\d\d//gn
  1. avgs smaller than 5.00 (remember number of matches)
:%s/avg_deg': [01234]\.\d\d//gn

Now you can sum those number of matches.

command explanation, sort of

  • %s - is to substitute a regexp
  • //g - substitute all matches with nothing... but we have n
  • n - report the number of matches, do not actually substitute

enter image description here

  • This worked great, thank you very much! Could you also tell me how I can sum the actually matching values? (By subtracting the number of values times 5.50 (for values bigger than 5.50) and 5.00 (for values smaller than 5.00)) I am then able to get some information on the average deviation from my expected range. (mathematical I am still not 100% sure whether that makes sense but for the moment it sounds like a good idea.
    – baxbear
    Apr 17 '20 at 19:56
  • You can probably use :g command to some degree -- but I am not sure. Why wouldn't you extract values needed and put it into some spreadsheet?
    – Maxim Kim
    Apr 17 '20 at 20:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.