So first things first, what is this "ex-mode"? Ex is an ancient text editor written by Bill Joy (the original author of vi). It is very primitive. Ex is a lot like vim, but without normal or insert modes. The only thing you can do is commands that start with a colon (known as ex-commands).
vi was written, a lot of people were used to
ex and didn't want to learn this fancy-shmancy new editor, so
vi was designed to be backwards compatible with
ex. Because of this, vi has an ex-mode, where you can only use ex-commands. (Vim keeps it around for similar stupid reasons)
To enter ex-mode, type
Q in normal mode. To get out of this annoying mode, you can type the command
vi as a shortcut). This will always put your cursor back where you started.
open is very similar to visual, except that it takes a number as an argument, and puts your cursor on that line. This makes sense since
ex was line-based and could only show a line at a time. So the idea behind the command was "open line 'n' for me to look at/read".
Now, what is the difference between the three when you ignore ex-mode? Not really very much. As Josh explains in this wonderful answer,
:edit are implemented in the exact same place, so they're essentially the same (when you ignore
visual being used to leave ex-mode). This goes well with the help docs, where it states:
:vi[sual][!] [++opt] [+cmd] [file]
When used in Ex mode: Leave |Ex-mode|, go back to
Normal mode. Otherwise same as |:edit|.
Otherwise same as :edit! However,
:open is slightly different because 1) the ex-command is different (accepting a range) and 2) the normal command is different (accepting an optional pattern). So
:open has it's own separate implementation.
Now there's only one question remaining. Why does
:open not tab-complete? I'm not Bram Moolenaar, so I could be wrong, but it's probably because the open command takes a pattern as an argument. It makes sense for it to behave more like
:s/ when the user could be typing a regex or a file.
If you want to edit a file, use
:e. It's standard, well-supported, and convenient (because of tab-complete). You could use
:visual if you really want too, but it's an unusual practice and confusing.
Generally avoid ex-mode. If you want to leave ex-mode, either
open are fine ways to exit it (depending on which behavior you want)
I have no idea why Bram chose to make it so that using the right command in the wrong context (e.g.
open from normal mode) defaults to pretending to be