3

I'm trying to use vim as a hex editor.

I am using the %!xxd / %!xxd -r method, and it worked fine until I tried a some bytes that didn't work as expected :

00000000: 1337 6161 6161 6161 6161 6161 6161 6161  .7aaaaaaaaaaaaaa
00000010: 6161 6161 6161 6161 6161 6161 6161 62ff  aaaaaaaaaaaaaabR
00000020: 3713 0000 0a                             7....

With this in the vim buffer, running %!xxd -r and then %!xxd will produce :

00000000: 1337 6161 6161 6161 6161 6161 6161 6161  .7aaaaaaaaaaaaaa
00000010: 6161 6161 6161 6161 6161 6161 6161 62c3  aaaaaaaaaaaaaab.
00000020: bf37 1300 000a                           .7....

Notice the difference at the end of the second line.

Now you might think that xxd is not used correctly, however running this : cat bytes | xxd -r | xxd will yield :

00000000: 1337 6161 6161 6161 6161 6161 6161 6161  .7aaaaaaaaaaaaaa
00000010: 6161 6161 6161 6161 6161 6161 6161 62ff  aaaaaaaaaaaaaab.
00000020: 3713 0000 0a                             7....

which is what I was expecting to see in vim.

Do you have any idea of what I am doing wrong ? Thank you for your help.

3
  • Combinaison = combination? (Et bienvenue à Vi and Vim!) – D. Ben Knoble Apr 4 '20 at 14:14
  • merci :D I'll change that, good catch. – aguilbau Apr 4 '20 at 14:20
  • Interesting! It's turning \u00ff into \xc3\xbf, which is exactly the UTF-8 encoding of that Unicode character. So encoding is the source of the problem here. The solution might be using vim -b to enable binary mode (see :help 'binary'), but not sure if that will break anything on the xxd view (since that's textual and not really binary...) – filbranden Apr 4 '20 at 15:22
2

This will happen if you start editing the xxd-dumped version of the binary file, in which case Vim will assume a UTF-8 encoding (which you can confirm with :set fenc? returning fileencoding=utf-8).

When you turn the file back into binary form, the character still looks correct (using ga on top of it shows <ÿ> 255, Hex 00ff, Oct 377, Digr y:), but that's actuall not considering the encoding, so if you were to save this file now, the UTF-8 encoding will kick in and will turn \u00ff into the bytes \xc3\xbf, which are the UTF-8 encoded version of this Unicode character.

One possible workaround is to use a different file encoding such as latin1, which is a 8-bit encoding and shouldn't expand characters above 127.

A better solution is to use vim -b to enable the 'binary' option. (You can also use :set binary from inside Vim.) Enabling 'binary' will disable a file encoding completely (:set fenc? returns fileencoding=) and should preserve 8-bit characters as they are.

It turns out this works OK with the xxd-dumped output since xxd only uses 7-bit ASCII characters in its output, so using 'binary' doesn't really hurt it too much.

(From a purist's point of view, an ideal solution here would use two separate buffes in Vim, one for the xxd-dumped output and a separate one with 'binary' set for the binary encoded output of xxd. A plug-in facilitating binary editing would probably help manage the separate buffers and set the right options on all of them.)

1
  • 2
    very good analysis, thank you ! – aguilbau Apr 6 '20 at 10:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.