hardenedapple has given an excellent explanation for this, and a proposed fix, but I will describe what is happening in more non-technical terms:
Once a vim mapping "times out," vim does not like to wait again. Instead, at any point after timeout, it will always immediately expand the shortest valid mapping at that point. Here is another example that displays this behavior:
nnoremap aaaa :echom 'aaaa mapping'<CR>
nnoremap bb :echom 'just to ensure timeout matters'<CR>
nmap b aaa
Once b times out, vim decides on aaa (i.e., append aa) and will not wait to see if aaaa would have been possible.
In the original <nop> case this is a bigger problem because <nop> expands to '' (i.e., empty string). So, after timeout, when pressing \`` vim gets into an unfortunate loop, every `` is consumed and expanded to '' empty string immediately. It nearly impossible to press \`` quickly enough.
Whether or not this is a bug is a matter of taste. It appears to be a problem only when dealing with maps to <nop>. Bram has stated that there is not sufficient justification for a change. However, it is possible that he hasn't been made aware of this particular case.
In support of this being called a "bug," there are very few "do nothing" commands in insert mode- the most benign map I know of is <c-g><c-g> (really there are many nonsense commands starting with <c-g>) actually causes beeps, if beeps are enabled. An alternative fix to vim may be to introduce a new i_<c-g>-prefix mapping which is documented to do nothing and cause no beeps.