Building Vim from the source with the right flags is the general solution.
git clone https://github.com/vim/vim.git
./configure --with-x=yes --disable-gui
The last command will probably raise an error message, which means
you need to install dependencies. Experienced users know how to proceed, but
if it's your first time on this, bad news is ...