I found the ultimate solution.
vimtex with language tools using YaLafi .
vim section of Yalafi's github page!!
Unlike above solution, you can also perform grammar check using
LanguageTools, which is the solution which I wanted in the first place.
It rips the equations and characters from the current tex file and performs spell and grammar check using standalone language tools.
You can even set your own replacing rules such that non-trivial names or sets of text can be replaced to common names like "John" or "Law" so that Language tools won't get confused with field-specific terms that aren't supported.