parsing / lexing scriptout log file from gvim -w

Question

I've been running gvim -w /dev/shm/vim_keystrokes for some time, to collect keystrokes for later analysis. But now I'm looking for some guidance on how to make sense of the binary log file. Here's a snippet, asking less to render binary data:

I'm looking for at minimum enough information to tokenize this stream, even if I can't make sense of all the tokens. But ideally I'd like to be pointed to a document that can help me properly parse the file.

There is some prior art but with no comments:

https://www.drbunsen.org/vim-croquet/ https://github.com/dstokes/vim-stream

And this SO question suggests gvim is writing garbage to the log in some way, but doesn't provide any clarity:

https://stackoverflow.com/questions/3981535/using-the-w-option-of-vim

EDIT: It looks like three byte sequences starting with 0x80 can be filtered out and represent window events in some way, though I need to look closer.

Answer 1

Someone else has done this kind of processing, also open source (Ruby): https://github.com/igrigorik/vimgolf/tree/master/lib/vimgolf

Players of Vim Golf have their keystrokes recorded also with -w (well, -W actually, but same idea) and the library decodes it into nice human readable form. I discuss at some length here:

How does vimgolf record all keystrokes?

The triple-byte sequences beginning with 0x80 are, indeed, special. This is touched upon in the linked answer. (They include all non-printing and control inputs that can't be easily represented by a typed character. Esc, Tab, Enter, Ctrl+whatever, etc.)

Regarding the suggestion that garbage is being printed, I may have stumbled upon the same issue (a certain terminal code needed to be nulled in my vimrc...also mentioned in my post).

Bottom line: Unless you have zero coding experience it should be relatively easy to incorporate the vimgolf lookup tables (hash maps) into whatever you want to do. Maybe you just want confirmation that you can ignore certain inputs. Regardless, feel free to ask follow up questions if you're not sure about something.