You can do it in a single substitute command like this
:s/(\(.*\))/[\1]/
The \(
and \)
mark the regex subexpression that matches everything inside the ( and ) in the input line. TheIn the replacement, the \1
stands for whatever the first (in this case the only) marked subexpression matched.
In words you might say,describe the substitute command as: look for a "(" followed by anything followed by ")". Replace that by "[" followed by the same "anything" followed by "]".