This is not possible directly from Vim. This is something that has to be done in the calling shell.
You could work with a shell function:
# Vim and optional exit
vim "$@" || exit
If Vim returns a non-zero exit code, the command
exit is called to exit the shell and hence close the terminal.
To produce a non-zero exit code use
:cq instead of
:q. If Vim would return a non-zero exit code for any other reason, the terminal would also close.
:!exit can't work, as it is not executed in the shell started by the terminal emulator, but in a shell started by vim.