Here's an alternative solution which will only hide the numbers, but won't actually change the margin on the left so you text doesn't jump around so much when entering/leaving insert mode:
autocmd InsertEnter * highlight LineNr ctermfg=white guifg=white
autocmd InsertLeave * highlight LineNr ctermfg=130 guifg=Brown
When entering insert mode this will set the text colour to the background colour. In my case this is white, but this may be different in your case. When we leave insert mode it resets it to the default for my colour scheme.
You can find out the default for your colour scheme with:
LineNr xxx term=underline ctermfg=130 guifg=Brown