0

I would like to use vim to do the following operation in vim: I would like to determine the number of entries with avg_deg greater then "'avg_deg': 5.50" from a simple .log file as well as the average of all avg_degs. Afterwards I would like to repeat the same for all entries smaller than "5.00" (the smallest entry in the log is "4.63" and the biggest is at "6.04"). Simply searching and counting the values manually is not possible in realistic time since the file is about 31 MB big.

The file contains many entries with some string in apostrophes with colon following a number but it is necessary to observe only numbers directly following the string "'avg_deg': ".

I would have copied parts of the file as example but even single variables are too big to put them here - at least my browser dies and the website doesn't react anymore when I do so.

Improve this question
5
  • 1
    Maybe a sample of your document would help
    – Zorzi
    Commented Apr 17, 2020 at 17:26
  • Tried to add just one array of the .log and intended to add further parts of the .log - but already that single array killed my browser after the website decided not to respond anymore... so basically you find lots of dictionaries with arrays with dictionaries with entries. The "'avg_deg': 5.50" appears for each test case twice in the file but that shouldn't matter since for the intended operations the first result has to be divided by 2 and the second should be even exactly the same. Actually I intended to save the result as .json but python just failed doing it on the variable size.
    – baxbear
    Commented Apr 17, 2020 at 17:39
  • You could create simplified example of the data, though
    – Maxim Kim
    Commented Apr 17, 2020 at 18:13
  • 1
    This is not fundamentally a vim exercise... use grep (or your favorite tools)—it's likely to be faster, not consume as much memory (31M to load the file!), and allow you trim out more correctly your search pattern. Example: <big-file grep -c "avg_deg': [5]\.[56789]". If you need average, awk is good but lacks fixed/floating point support, so use something like sed 's/\([[:digit:]]\)\.\([[:digit:]]\)/\1\2/g' to convert everything to 3-digit ints
    – D. Ben Knoble ♦
    Commented Apr 17, 2020 at 19:33
  • Thank you very much! In my case grep needs to use -o since some of the results are in the same line. Afterwards I count the output with | wc -l => <.log grep -o "'avg_deg': [4]" | wc -l and <.log grep -o "'avg_deg': [5]\.[56789][0123456789][0123456789]*" | wc -l and finally <.log grep -o "'avg_deg': [6]" | wc -l. I still have to check and understand the sed command of yours to use it.
    – baxbear
    Commented Apr 18, 2020 at 21:26

1 Answer 1

Reset to default
1

If it is one off task you can try this

  1. avgs greater than 5.50 till 5.99 (remember number of matches)
:%s/avg_deg': [5]\.[56789]\d//gn
  1. avgs greater than 6.00 (remember number of matches)
:%s/avg_deg': [6789]\.\d\d//gn
  1. avgs smaller than 5.00 (remember number of matches)
:%s/avg_deg': [01234]\.\d\d//gn

Now you can sum those number of matches.

command explanation, sort of

  • %s - is to substitute a regexp
  • //g - substitute all matches with nothing... but we have n
  • n - report the number of matches, do not actually substitute

enter image description here

Improve this answer
2
  • This worked great, thank you very much! Could you also tell me how I can sum the actually matching values? (By subtracting the number of values times 5.50 (for values bigger than 5.50) and 5.00 (for values smaller than 5.00)) I am then able to get some information on the average deviation from my expected range. (mathematical I am still not 100% sure whether that makes sense but for the moment it sounds like a good idea.
    – baxbear
    Commented Apr 17, 2020 at 19:56
  • You can probably use :g command to some degree -- but I am not sure. Why wouldn't you extract values needed and put it into some spreadsheet?
    – Maxim Kim
    Commented Apr 17, 2020 at 20:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.