You can not use regexp's to match infinitely nested patterns in Vim.
Regular expressions are modeled off a formal language theory, regular language. Some regular expression libraries even go a step further and use an NFA (Nondeterministic finite automaton are regular) as the actual underlying model their regular expression engine. Vim has support for a NFA based regex engine for example. See
Why is it important that Regexp are modeled off of Regular Languages?
First and foremost nested matching parenthesis is not regular. Although regex engines are often not purely regular in what they can match, they must support recursion to match nested patterns. So the question becomes can the given regex engine use recursion to overcome this limit? Perl's regex engine can do this for example. Sadly, Vim does not have a regex engine that can use recursion.
You can anticipate the max depth and use branches to sort of accomplish this affect:
That pattern will match
(foo) as well as
(foo(bar)foo). It will not match
(foo(bar(baz)bar)foo) as it is too deep. It will also fail if parens are not properly matched.
You may want to use something other than
:substitute. Like use the
% command or text objects.
You may be able to use Tim Pope's surround.vim for this use case:
This would "change the surrounding"
( pairs with
More advanced techniques would be to use
searchpair() and/or sniffing Vim's syntax highlighting for a given word. See