I've tried this
au BufWinEnter * if &l:buftype != 'nofile' | map <buffer> <CR> <Plug>(easymotion-prefix) | endif
but somehow this isn't working (easymotion isn't working). While this works
au BufWinEnter * map <buffer> <CR> <Plug>(easymotion-prefix)