The slowness of delimiter matching is not caused by vim itself but by the matchparen plugin, which is part of the vim distribution and loaded by default in
/etc/vim/vimrc in Debian. (This is why I still had the problem without a vimrc.)
Fortunately the matchparen plugin supports a timeout when looking for matches, so the slowness can be avoided by setting those timeout to lower values. I added the following to my vimrc:
let g:matchparen_timeout = 2
let g:matchparen_insert_timeout = 2
2 milliseconds is plenty of time on modern machines to find a relevant delimiter, and when scrolling down in this file on my machine, display is no longer slugging and CPU usage for vim stays at 15% or so.
Edit: actually I increased it to 20, otherwise I sometimes see it fail. This means there is still some sluggishness, but it's better than without the setting.