You can test the 'encoding' option. For example I have this in my .vimrc
if &encoding == "utf-8"
if !exists("$PUTTY_TERM") && $OS isnot# "Windows_NT"
exe "set listchars=nbsp:\u2423,conceal:\u22ef,tab:\u2595\u2014,trail:\u02d1,precedes:\u2026,extends:\u2026,eol:\ub6"
exe "set sbr=\u21b3"
else
" Putty can't display all nice utf-8 chars
exe "set listchars=conceal:\u00b7,tab:>\u2014,trail:\u02d1,precedes:\u2026,extends:\u2026,eol:\ub6,nbsp:\u03c7"
exe "set sbr=\u2500"
endif
exe "set fillchars=vert:\u2502,fold:\u2500,diff:\u2014"
else
" Special characters that will be shown, when set list is on
set listchars=eol:$,trail:-,tab:>-,extends:>,precedes:<,conceal:+
" Display a `+' for wrapped lines
set sbr=+
endif
Note also, you shouldn't change the encoding option, once it has been set, because it will invalidate all character data stored in memory in Vim.