Replace embeded escape code in text

Question

I am using the most recent nvim in arch.

I have this text showing in my text doc

customerâ<80><99>s 

the <80><99> appears in blue.

The text should read

customer's 

I wish to use this replace format :%s///g to get rid of the â<80><99> and replace it with an apostrophe '

This command :%s/â<80><99>/'/g does not work

How do I do the replacement?

Answer 1

As pointed out, this is an encoding problem. The sequence formed by the three bytes 0xe2 (â), 0x80, 0x99 (three bytes denoted by their hexadecimal number) corresponds to the UTF-8 encoding of Unicode character U+2019, which is the Right Single Quotation Mark: ’

You can use the Normal-mode command ga to tell you about the character under the cursor (for example,.it will tell you that the "â" has code 0xe2, or that the Unicode character "’" has code 0x2019.)

It may be hard to fix encoding issues, especially when copying and pasting from other applications. PDFs can be a common source of this problem, so I feel your pain there... One thing you could try is to open a buffer in Vim in binary mode with vim -b or :set binary, then save the buffer with the pasted byte sequences, then open the file in Vim again but this time let Vim recognize it as a proper UTF-8 encoded file, hopefully Vim will then be able to interpret the sequences correctly.

To perform search and replace on those expressions, the tricky part is that the <80> and <99> are in fact a single character. You have two options to match them in a search.

First, you can encode them by their character codes. But to do so in a regex, you need to use the \%x sequence, followed by the hexadecimal number representing the character. In your case:

:%s/â\%x80\%x99/'/g

The second option you have is to enter actual <80> and <99> characters in the search expression. One way to do so is to copy them from the buffer and paste them into the search expression.

You can copy them by creating a Visual selection around the â<80><99> sequence and pressing y to copy it into the unnamed register (referred to by the " symbol), then while typing the substitute command, you can use Ctrl+R, " to paste it there.

So what you type is :%s/, then Ctrl+R, " to paste the sequence of three special characters, then type /'/g and then press Enter. What you'll see is:

:%s/â<80><99>/'/g

But the <80> and <99> will be highlighted (in your case, in blue) to indicate they represent a single character each.