Wrong encoding while calling shell

Question

I was experimenting with a DOT diagram and tried to do the following:

:! dot -Tpng -oFab.png %

I got an error because my filename has a special character ("ó" in "Fabricación"):

C:\windows\system32\cmd.exe /c ( dot -Tpng -oFab.png Fabricaci├│n.gv)
Error: dot: can't open Fabricaci├│n.gv
shell returned 2
Hit any key to close this window...

As you can see, the special character is being changed for "├│". This is with vim and gVim 7.4 under Win7 and NTFS, so I assume the filename is in UTF16. I also assume that when invoking the shell/cmd the filename is being interpreted as some other encoding (thanks to Carpetsmoker for pointing out it defaults to code page 850).

How can I fix this?

Sure, I can simply rename the file, but I'd like to know why this happens and how to correct it.

Update: I just found this question in superuser.SE (thanks to feedback by @ChristianBrabandt), but it doesn't seem to help either.

Answer 1

Short answer

The problem lies in dot.exe. GraphViz can open files with Unicode paths in Linux but not Windows, unless (maybe) if compiled with Visual Studio 2005.

Research

The code page is set to 850, Vim encoding to UTF-8.

It doesn't give the exact same error, but the dot.exe seems to receive a wrong argument. I tried passing the same filename to the other program.

And it worked just right. Executing both dot.exe and typedirectly from cmd.exe gives the same result, so neither the Windows Console nor Vim are the problem. The next thing that could cause that error was dot.exe itself. My suspicion was that it just doesn't know how to handle the Unicode coded arguments properly, as not even all console commands do:

https://ss64.com/nt/chcp.html

If you need full Unicode support use PowerShell. There is still VERY limited support for Unicode in the CMD shell, piping, redirection and most commands are still ANSI only. The only commands that work are DIR, FOR /F and TYPE, this allows reading and writing (UTF-16LE / BOM) files and filenames but not much else.

I searched on the web if there is support for Unicode in GraphViz and found that it does support Unicode files but nothing about Unicode support for the filenames. Neither I found any reports on the GraphViz bug tracker nor posts on the forum about anyone else being interested in reading a Unicode named file. So I looked it up in the source. Here what is dot.exe entry point looks like:

graphviz-2.40.1\cmd\dot\dot.c

int main(int argc, char **argv)
{
    . . .

/* --------------------> ARGS ARE BEING PASSED HERE */
    gvParseArgs(Gvc, argc, argv);

    . . .

Following the argv down the rabbit hole: graphviz-2.40.1\lib\common\args.c

int gvParseArgs(GVC_t *gvc, int argc, char** argv)
{
    int rv;
    if ((argc = neato_extra_args(gvc, argc, argv)) < 0)    return (1-argc);
    if ((argc = fdp_extra_args(gvc, argc, argv)) < 0)      return (1-argc);
    if ((argc = memtest_extra_args(gvc, argc, argv)) < 0)  return (1-argc);
    if ((argc = config_extra_args(gvc, argc, argv)) < 0)   return (1-argc);

/* -------------------->  HERE GO ALL NON-FLAG ARTUMENTS */
    if ((rv = dotneato_args_initialize(gvc, argc, argv)))  return rv;

    if (Verbose) gvplugin_write_status(gvc);
    return 0;
}

graphviz-2.40.1\lib\common\input.c

int dotneato_args_initialize(GVC_t * gvc, int argc, char **argv)
{
    for (i = 1; i < argc; i++) {
        if (argv[i] && argv[i][0] == '-') {
            
            . . .

/* -------------------->  JUST CASUALLY COPYING CHAR POINTERS */
        } else if (argv[i])
            gvc->input_filenames[nfiles++] = argv[i];
    }

And finaly graphviz-2.40.1\lib\common\input.c

graph_t *gvNextInputGraph(GVC_t *gvc)
{
    . . . .

/* -------------------->  OPENING THE FILES FOR READ WITH FOPEN */
    while ((fn = gvc->input_filenames[fidx++]) && !(fp = fopen(fn, "r")))  {

        . . .

    }

As the MDSN states:

The fopen function opens the file specified by filename. _wfopen is a wide-character version of fopen; the arguments to _wfopen are wide-character strings. _wfopen and fopen behave identically otherwise. Simply using _wfopen has no effect on the coded character set used in the file stream.

In Visual C++ 2005, fopen supports Unicode file streams.

Sadly, the only option there is to rename the file.