I was toying around with my
.vimrc editing it, and then checking in another file whether my changes worked or not.
But now I ended up in a state where opening one file with
vim file1.tex (this was the one I used for testing) doesn't load syntax highlighting and using
vim file2.tex loads the syntax highlighting.
My thought was that maybe I accidentally set something specific for
file1.tex which persists, but the only info I found was
.viminfo and deleting that did not help.