Is there a way to disable all mappings for vim-go? So I can only have the mappings that I explicitly set myself?
Yes, there are settings for that, see
:help go-settings for a list of all settings.
In particular, the ones you're probably interested in are:
let g:go_def_mapping_enabled = 0
let g:go_doc_keywordprg_enabled = 0
And perhaps also:
let g:go_textobj_enabled = 0
(Not sure if these were the mappings you had in mind... Since they're essentially overriding default Vim keybindings in a way to make them more effective when editing Go source code.)
You can also check the Vimscript source code to check which mappings exactly are being set (controlled by these options) in case you want to create separate mappings using different key combinations for them.