0

I've got a text file with unicode character codes in it. It looks like this in vim (utf-8[unix]):

Samuel i<81>ngel - Euro <80>25
Tomas i<81>lvaro - ©Coca-Cola and dagger <87>
Luis i<81>lvaro
Scooter ð<9d><90>ƒð<9d><90>²ð<9d><90>¥ð<9d><90>šð<9d><90>§

When I use less to view it I get this:

Samuel i<U+0081>ngel - Euro <U+0080>25
Tomas i<U+0081>lvaro - ©Coca-Cola and dagger <U+0087>
Luis i<U+0081>lvaro
Scooter ð<U+009D><U+0090>ƒð<U+009D><U+0090>²ð<U+009D><U+0090>¥ð<U+009D><U+0090>šð<U+009D><U+0090>§

I've been trying to replace all the hex character codes in the file with their unicode characters using nr2char and submatch, but I can't get it to work with all of them in one search/replace. Can anyone help? It works to replace a single one (e.g. <90> with :

:1,$ s/\%\(\%x90\)/\=nr2char('0x'.submatch(1),1)/g

I cannot extend this to all of them though - I've tried things like the following:

:1,$ s/\([\%x00-\%xFF]\)/\=nr2char('0x'.submatch(1),1)/g <== replaces loads of characters
:1,$ s/\%\(\%x[80-90]\)/\=nr2char('0x'.submatch(1),1)/g <== Invalid character after \%[dxouU]
:1,$ s/\%\([\x80-\xFF]\{2\}\)/\=nr2char('0x'.submatch(1),1)/g <== only replaces <9D>
:1,$ s/\%\([\x80-\xFF]\{2\}\)/\=nr2char('0x'.submatch(1))/g <== only removes <9D>
:1,$ s/\%\([\x80-\xFF]\)/\=nr2char('0x'.submatch(0))/g <== removes all, does not replace
:1,$ s/\(\%[\x80-\xFF]\)/\=nr2char('0x'.submatch(1),1)/g <== does weird random stuff with new lines
[...]

(I appreciate this is close to this answer, but it's not exactly the same)

2
  • Can you tell us by what you would like to replace <90>? Is it by "0x90"? Commented Mar 8, 2023 at 16:22
  • @VivianDeSmedt good question, thanks. With <90> I just looked it up and it's officially unused. But certainly I'd like to replace the <80> with € and the <87> with ‡. Commented Mar 9, 2023 at 11:45

3 Answers 3

3

I believe the problem is that Vim not opening the file using the right encoding. Your file is in encoding cp1252 but Vim doesn't detect it and guess utf-8.

I would propose to open the file with the ++enc=cp1252 flag.

:e ++enc=cp1252 filepath

To improve the way Vim guess the encoding you can add the following lines (Vim would first try to open as cp1252 and then as utf-8):

:set fileencodings=cp1252,utf-8
4
  • 1
    Hi @Sam if the solution solves your problem I'll be glad if you could accept it using the v green button next to the arrow voting buttons. It allow the question to rest :-) Commented Mar 9, 2023 at 15:48
  • Thanks Vivian! It's nearly there, but interestingly setting the encoding above gives me a little  in front of everything - e.g. Euro €25 - any idea why? Commented Mar 10, 2023 at 13:24
  • 1
    I should have a look to your file. Could you share an extract? Commented Mar 10, 2023 at 14:46
  • 1
    It would also help to get a screenshot and the list of Vim operation that lead to that screenshot together with the result of the following command :set fileencoding? Commented Mar 10, 2023 at 15:47
1

I would propose you:

:s/\([\x80-\xFF]\)/\='0x'.printf('%02x', char2nr(submatch(1)))/g
1
  • Thanks! That seems to work on search (great!), but the replace gives me this: Samuel i0x81ngel - Euro 0x8025 Tomas i0x81lvaro - 0xa9Coca-Cola and dagger 0x87 Luis i0x81lvaro Scooter 0xf00x9d0x90ƒ0xf00x9d0x900xb20xf00x9d0x900xa50xf00x9d0x90š0xf00x9d0x900xa7 I'd actually like to get the symbols €, ‡ etc in there. Commented Mar 9, 2023 at 11:49
1

I actually prefer escorting to a python one-liner, using :py3do (if you're on a older version of vim you might need :pydo)

" transforms 'samuel i<81>ngel' to 'samuel i\x81ngel
" Python variable `line` represents the input line
" Return value indicates what to transform the input line into
:py3do return ''.join([x if ord(x) < 128 else f'\\x{ord(x):02x}' for x in line])

Also it looks like you're opening the text file in the wrong encoding because usually when opened in the right encoding, text files do not contain unprintable raw characters.

(Actually I'm not quite sure what your requirements are, but with python it's quite easy to go the other way round, as you've got str::printable etc and you can even import some modules)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.