You can do it in a single substitute command like this

    :s/(\(.*\))/[\1]/

The `\(` and `\)` mark the regex subexpression that matches everything *inside* the ( and ) in the input line. The `\1` stands for whatever the *first* (in this case the *only*) marked subexpression matched.

In words you might say, look for a "(" followed by anything followed by ")". Replace that by "[" followed by the same "anything" followed by "]".