On profiling, I found that
<SNR>101_parse_screen() is taking above 6 seconds. This problem goes away when I comment out the plugin vim-polyglot.
This is most evident for vue files. Not sure if it's a bug, so posting this here.
vim-polyglot is a possible solution, but it seems to work best for me, and don't want to abandon it.
Will add more details if needed.