2

I'm cleaning data imported from Excel to Access, and ran across funny quotes that I'm having trouble getting unambiguous ASCII codes for.

They are the hooked single quotes and . Vim's ga command show them to have decimal codes 8216 and 8217, but Microsoft's ASC function shows them to have decimal codes 145 and 146. In the end, I used Microsoft's CHR(145) and CHR(146) to locate and clean them, but I was wondering how to interpret the larger numbers returned by Vim.

The Microsoft numbers do not have bit patterns that are subsets of the larger Microsoft numbers:

octave:6> dec2bin([145 146])

   10010001
   10010010

octave:5> dec2bin([8216 8217])

   10000000011000
   10000000011001

My vim version details are below.

Thanks.

Vim version details:

VIM - Vi IMproved 7.4 (2013 Aug 10, compiled May 20 2015 22:20:54)
Included patches: 1-729
Compiled by <cygwin-xfree@cygwin.com>
Huge version with GTK2 GUI.  Features included (+) or not (-):
+acl             -ebcdic          +localmap        +printer         +title
+arabic          +emacs_tags      +lua/dyn         +profile         +toolbar
+autocmd         +eval            +menu            +python/dyn      +user_commands
+balloon_eval    +ex_extra        +mksession       +python3/dyn     +vertsplit
+browse          +extra_search    +modify_fname    +quickfix        +virtualedit
++builtin_terms  +farsi           +mouse           +reltime         +visual
+byte_offset     +file_in_path    +mouseshape      +rightleft       +visualextra
+cindent         +find_in_path    +mouse_dec       +ruby/dyn        +viminfo
+clientserver    +float           -mouse_gpm       +scrollbind      +vreplace
+clipboard       +folding         -mouse_jsbterm   +signs           +wildignore
+cmdline_compl   -footer          +mouse_netterm   +smartindent     +wildmenu
+cmdline_hist    +fork()          +mouse_sgr       -sniff           +windows
+cmdline_info    +gettext         -mouse_sysmouse  +startuptime     +writebackup
+comments        -hangul_input    +mouse_urxvt     +statusline      +X11
+conceal         +iconv           +mouse_xterm     -sun_workshop    -xfontset
+cryptv          +insert_expand   +multi_byte      +syntax          +xim
+cscope          +jumplist        +multi_lang      +tag_binary      +xsmp_interact
+cursorbind      +keymap          -mzscheme        +tag_old_static  +xterm_clipboard
+cursorshape     +langmap         +netbeans_intg   -tag_any_white   -xterm_save
+dialog_con_gui  +libcall         +path_extra      -tcl             +xpm
+diff            +linebreak       +perl/dyn        +terminfo        
+digraphs        +lispindent      +persistent_undo +termresponse    
+dnd             +listcmds        +postscript      +textobjects     
   system vimrc file: "$VIM/vimrc"
     user vimrc file: "$HOME/.vimrc"
 2nd user vimrc file: "~/.vim/vimrc"
      user exrc file: "$HOME/.exrc"
  system gvimrc file: "$VIM/gvimrc"
    user gvimrc file: "$HOME/.gvimrc"
2nd user gvimrc file: "~/.vim/gvimrc"
    system menu file: "$VIMRUNTIME/menu.vim"
  fall-back for $VIM: "/etc"
 f-b for $VIMRUNTIME: "/usr/share/vim/vim74"
Compilation: gcc -c -I. -Iproto -DHAVE_CONFIG_H -DFEAT_GUI_GTK  -D_REENTRANT -I/usr/include/gtk-2.0 -I/usr/lib/gtk-2.0/include -I/usr/include/pango-1.0 -I/usr/include/gio-unix-2.0/ -I/usr/include/cairo -I/usr/include/atk-1.0 -I/usr/include/cairo -I/usr/include/pixman-1 -I/usr/include/gdk-pixbuf-2.0 -I/usr/include/libpng16 -I/usr/include/pango-1.0 -I/usr/include/harfbuzz -I/usr/include/pango-1.0 -I/usr/include/glib-2.0 -I/usr/lib/glib-2.0/include -I/usr/include/freetype2 -I/usr/include/libpng16 -I/usr/include/freetype2 -I/usr/include/libpng16    -ggdb -O2 -pipe -Wimplicit-function-declaration -fdebug-prefix-map=/usr/src/ports/vim/vim-7.4.729-2.x86_64/build=/usr/src/debug/vim-7.4.729-2 -fdebug-prefix-map=/usr/src/ports/vim/vim-7.4.729-2.x86_64/src/vim74=/usr/src/debug/vim-7.4.729-2 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1      
Linking: gcc   -L. -fstack-protector  -L/usr/local/lib -Wl,--as-needed -o vim.exe   -lgtk-x11-2.0 -lgdk-x11-2.0 -lpangocairo-1.0 -lgio-2.0 -lXinerama -lXi -lXrandr -lXcursor -lXcomposite -latk-1.0 -lcairo -lz -lGL -lpixman-1 -lEGL -lm -lpthread -lGL -lm -lpthread -lXdamage -lXfixes -lX11-xcb -lxcb-glx -lxcb-shm -lxcb-render -lXrender -lXext -lX11 -lxcb -lXau -lXdmcp -lgdk_pixbuf-2.0 -lm -lpng16 -lm -lz -lgio-2.0 -lz -lpangoft2-1.0 -lharfbuzz -lpango-1.0 -lm -lgmodule-2.0 -lgobject-2.0 -lffi -lglib-2.0 -lintl -lpcre -lintl -liconv -lpcre -lfontconfig -lexpat -lfreetype -lbz2 -lz -lpng16 -lm -lz -lexpat -lfreetype -lbz2 -lz -lpng16 -lm -lz  -lSM -lICE -lXpm -lXt -lX11 -lXdmcp -lSM -lICE  -lm -lelf    -lncursesw -liconv -lintl   -Wl,--enable-auto-import -Wl,--export-all-symbols -Wl,--enable-auto-image-base -fstack-protector -L/usr/local/lib  -L/usr/lib/perl5/5.14/x86_64-cygwin-threads/CORE -lperl -ldl -lcrypt
4

Vim is using Unicode (probably UTF-8), but Excel and/or Access are using Windows-1252 (or possibly a related Windows code page such as 1251 or 1253). Decimal 8216 is hexadecimal 2018, thus is Unicode character U+2018 LEFT SINGLE QUOTATION MARK.

Asc and Chr are old, pre-Unicode functions, which is why they are reporting the Windows-1252 character codes. If you use AscW and ChrW instead, you will find that Access and Excel use the same values as Vim.

:set encoding=cp1252 will tell Vim to use the Windows-1252 character encoding. Any non-ASCII characters, such as smart quotes, that already in your buffer will displayed as multiple characters (mojibake), but any new smart single quotes you enter will use 145 and 146. If you’re not sure how to type them in Vim, enter Insert mode and press Ctrl-K ' 6 for and Ctrl-K ' 9 for . ga will now show that is decimal 145 and is 146. :set coding=utf-8 will return Vim to UTF-8 encoding.

In general, I would suggest using AscW and ChrW instead of Asc and Chr, and I would not suggest switching Vim’s encoding to Windows-1252.

  • Thanks, Lithis. Lot to chew on there. I will try to gnaw on it this weekend. – user36800 Sep 1 '17 at 14:44
  • Unfortunately, the file encoding-names include cp1251 and cp1253, but not cp1252. Maybe because my gvim was compiled for Cygwin, years ago. The digraphs entered with ctrl-k replicate the odd characters in vim, but I find that I have to use chr() instead of string literals because they occur in a text SQL file that gets read by specific VBA fuction, and the literal characters aren't properly read. In the face this, the only reason I need the literal character is to find out the code so that I can refer to it using chr(). My current solution is asc() in the VBA editor's Immediate window. – user36800 Sep 1 '17 at 19:52
  • Vim’s encoding latin1 is actually the same as cp1252, as mentioned at stackoverflow.com/a/38372767/4896941 . – Lithis Sep 1 '17 at 20:15
  • Thanks. If I set encoding=latin1, I can't properly paste the odd tick mark into the buffer from register * (the Windows buffer). It shows up as an upside-down question mark with decimal code 191. Setting back to utf-8 allows me to paste the odd tick mark, decimal code 8217. – user36800 Sep 1 '17 at 21:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.