Search and replace unicode character codes with actual characters

Question

I've got a text file with unicode character codes in it. It looks like this in vim (utf-8[unix]):

Samuel i<81>ngel - Euro <80>25
Tomas i<81>lvaro - ©Coca-Cola and dagger <87>
Luis i<81>lvaro
Scooter ð<9d><90>ƒð<9d><90>²ð<9d><90>¥ð<9d><90>šð<9d><90>§

When I use less to view it I get this:

Samuel i<U+0081>ngel - Euro <U+0080>25
Tomas i<U+0081>lvaro - ©Coca-Cola and dagger <U+0087>
Luis i<U+0081>lvaro
Scooter ð<U+009D><U+0090>ƒð<U+009D><U+0090>²ð<U+009D><U+0090>¥ð<U+009D><U+0090>šð<U+009D><U+0090>§

I've been trying to replace all the hex character codes in the file with their unicode characters using nr2char and submatch, but I can't get it to work with all of them in one search/replace. Can anyone help? It works to replace a single one (e.g. <90> with :

:1,$ s/\%\(\%x90\)/\=nr2char('0x'.submatch(1),1)/g

I cannot extend this to all of them though - I've tried things like the following:

:1,$ s/\([\%x00-\%xFF]\)/\=nr2char('0x'.submatch(1),1)/g <== replaces loads of characters
:1,$ s/\%\(\%x[80-90]\)/\=nr2char('0x'.submatch(1),1)/g <== Invalid character after \%[dxouU]
:1,$ s/\%\([\x80-\xFF]\{2\}\)/\=nr2char('0x'.submatch(1),1)/g <== only replaces <9D>
:1,$ s/\%\([\x80-\xFF]\{2\}\)/\=nr2char('0x'.submatch(1))/g <== only removes <9D>
:1,$ s/\%\([\x80-\xFF]\)/\=nr2char('0x'.submatch(0))/g <== removes all, does not replace
:1,$ s/\(\%[\x80-\xFF]\)/\=nr2char('0x'.submatch(1),1)/g <== does weird random stuff with new lines
[...]

(I appreciate this is close to this answer, but it's not exactly the same)

Answer 1

I believe the problem is that Vim not opening the file using the right encoding. Your file is in encoding cp1252 but Vim doesn't detect it and guess utf-8.

I would propose to open the file with the ++enc=cp1252 flag.

:e ++enc=cp1252 filepath

To improve the way Vim guess the encoding you can add the following lines (Vim would first try to open as cp1252 and then as utf-8):

:set fileencodings=cp1252,utf-8

Answer 2

I would propose you:

:s/\([\x80-\xFF]\)/\='0x'.printf('%02x', char2nr(submatch(1)))/g

Answer 3

I actually prefer escorting to a python one-liner, using :py3do (if you're on a older version of vim you might need :pydo)

" transforms 'samuel i<81>ngel' to 'samuel i\x81ngel
" Python variable `line` represents the input line
" Return value indicates what to transform the input line into
:py3do return ''.join([x if ord(x) < 128 else f'\\x{ord(x):02x}' for x in line])

Also it looks like you're opening the text file in the wrong encoding because usually when opened in the right encoding, text files do not contain unprintable raw characters.

(Actually I'm not quite sure what your requirements are, but with python it's quite easy to go the other way round, as you've got str::printable etc and you can even import some modules)