5

I'm trying to use Vim as a hex editor.

I am using the %!xxd / %!xxd -r method, and it worked fine until I tried a some bytes that didn't work as expected :

00000000: 1337 6161 6161 6161 6161 6161 6161 6161  .7aaaaaaaaaaaaaa
00000010: 6161 6161 6161 6161 6161 6161 6161 62ff  aaaaaaaaaaaaaabR
00000020: 3713 0000 0a                             7....

With this in the Vim buffer, running %!xxd -r and then %!xxd will produce :

00000000: 1337 6161 6161 6161 6161 6161 6161 6161  .7aaaaaaaaaaaaaa
00000010: 6161 6161 6161 6161 6161 6161 6161 62c3  aaaaaaaaaaaaaab.
00000020: bf37 1300 000a                           .7....

Notice the difference at the end of the second line.

Now you might think that xxd is not used correctly, however running this : cat bytes | xxd -r | xxd will yield :

00000000: 1337 6161 6161 6161 6161 6161 6161 6161  .7aaaaaaaaaaaaaa
00000010: 6161 6161 6161 6161 6161 6161 6161 62ff  aaaaaaaaaaaaaab.
00000020: 3713 0000 0a                             7....

which is what I was expecting to see in Vim.

Do you have any idea of what I am doing wrong?

3
  • Combinaison = combination? (Et bienvenue à Vi and Vim!)
    – D. Ben Knoble
    Apr 4, 2020 at 14:14
  • merci :D I'll change that, good catch.
    – aguilbau
    Apr 4, 2020 at 14:20
  • Interesting! It's turning \u00ff into \xc3\xbf, which is exactly the UTF-8 encoding of that Unicode character. So encoding is the source of the problem here. The solution might be using vim -b to enable binary mode (see :help 'binary'), but not sure if that will break anything on the xxd view (since that's textual and not really binary...)
    – filbranden
    Apr 4, 2020 at 15:22

2 Answers 2

3

This will happen if you start editing the xxd-dumped version of the binary file, in which case Vim will assume a UTF-8 encoding (which you can confirm with :set fenc? returning fileencoding=utf-8).

When you turn the file back into binary form, the character still looks correct (using ga on top of it shows <ÿ> 255, Hex 00ff, Oct 377, Digr y:), but that's actuall not considering the encoding, so if you were to save this file now, the UTF-8 encoding will kick in and will turn \u00ff into the bytes \xc3\xbf, which are the UTF-8 encoded version of this Unicode character.

One possible workaround is to use a different file encoding such as latin1, which is a 8-bit encoding and shouldn't expand characters above 127.

A better solution is to use vim -b to enable the 'binary' option. (You can also use :set binary from inside Vim.) Enabling 'binary' will disable a file encoding completely (:set fenc? returns fileencoding=) and should preserve 8-bit characters as they are.

It turns out this works OK with the xxd-dumped output since xxd only uses 7-bit ASCII characters in its output, so using 'binary' doesn't really hurt it too much.

(From a purist's point of view, an ideal solution here would use two separate buffes in Vim, one for the xxd-dumped output and a separate one with 'binary' set for the binary encoded output of xxd. A plug-in facilitating binary editing would probably help manage the separate buffers and set the right options on all of them.)

1
  • 2
    very good analysis, thank you !
    – aguilbau
    Apr 6, 2020 at 10:43
1

Here's what I have in my ~/.vimrc to allow quite comfortable and issue-free editing of binary files, which are narrowed down to *.bin and *.hex files in the vimcode below, but that can be adjusted as needed, of course:

" Use xxd(1) to make editing binary files actually usable, by filtering
" the entire buffer through xxd(1) as an external command, and save/restore
" the view position while saving the buffer, to stay at the same position
"
" NOTE: Thanks to the file's dual nature, restoring the last saved position
"       of the cursor upon opening a file this way is effectively broken
"
augroup vimrc-xxd-binary-edit
  autocmd!
  autocmd BufReadPre  *.bin,*.hex set binary
  autocmd BufReadPost *.bin,*.hex
    \ if &binary
    \ |   execute "silent %!xxd -autoskip -cols 32"
    \ |   set filetype=xxd
    \ |   redraw
    \ | endif
  autocmd BufWritePre *.bin,*.hex
    \ if &binary
    \ |   let s:winview = winsaveview()
    \ |   execute "silent %!xxd -revert -autoskip -cols 32"
    \ | endif
  autocmd BufWritePost *.bin,*.hex
    \ if &binary
    \ |   execute "silent %!xxd -autoskip -cols 32"
    \ |   set nomodified
    \ |   call winrestview(s:winview)
    \ |   redraw
    \ | endif
augroup END

As noted in the code comments above, you may also want to disable restoring of the cursor position upon opening a file when &filetype ==# "xxd". See also :h restore-cursor for the usual way the cursor position is restored.

See also my patches that contain both the vimcode visible above and more related vimcode, which have been accepted upstream by the vim project in commits 6a500661a9 and 81b8bf5b4a.

10
  • Although that link is to another answer on this site, it's still good practice to put enough detail in this answer that it stands alone (and helps avoid link rot).
    – D. Ben Knoble
    Aug 11, 2023 at 15:31
  • Wouldn't it be then seen as copying and pasting the same aswer multiple times? I already had a few answers that were the same in different questions, for the sake of completeness, and a couple of them were deleted by the admins. I even received a warning that I might be banned because of my misbehavior. Knowing that, I hope you understand my concerns and confusion.
    – dsimic
    Aug 11, 2023 at 21:30
  • 1
    Your concern makes sense, though I'm not aware of any of your answers being deleted that way on this site. If you find yourself putting the same answers in many places, I suggest that either the questions are duplicates or that the answers need adjustment for differing context of the question. Regardless: the answer as it stands is essentially a comment or incomplete. Including enough detail in this answer that it stands on its own would solve that problem.
    – D. Ben Knoble
    Aug 21, 2023 at 12:51
  • My answers were deleted in these two questions: stackoverflow.com/questions/25447324/… and stackoverflow.com/questions/7142735/… . Those questions are duplicates of another question, but how can I flag them as duplicates when I have very little points? I tried editing those questions to include a note about them being duplicates, so someone can pick that up and actually flag the questions, but those pending edits seem to have been deleted silently.
    – dsimic
    Aug 21, 2023 at 13:44
  • 1
    Looking better now? :)
    – dsimic
    Aug 21, 2023 at 16:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.