Cancelling formatting from formatprg when not successfully exited?

Question

Some format programs refuse to format malformed code, but instead sends an error message to stderr and exits with non-zero status. Vim does not seem to care about that, and simply puts on the error message in place of the original code when gqed. Before going to reinvent the wheel of formatting, I wonder if there's a way to hint Vim to pay some attention to the exit status or stderr, and not to change the original content upon failing a format.

Answer 1

Here are two methods.

A simple one:

augroup FILTER_ERROR
    autocmd!
    autocmd ShellFilterPost * if v:shell_error | silent undo | endif
augroup END

and a more involved one:

function! Format() range
    " generate a temporary file name
    let tempfile = tempname()
    " write the lines in the given range to that temporary file
    call getline(a:firstline, a:lastline)->writefile(tempfile)
    " pass the temporary file to the external command and grab the output
    let output = systemlist(&formatprg .. ' ' .. tempfile)
    if v:shell_error
        " if there is an error, print the error and leave the buffer alone
        echohl WarningMsg | echo 'Shell error: ' .. v:shell_error | echohl None
    else
        " if there is none, delete the given range
        execute a:firstline .. ',' .. a:lastline .. 'd_'
        " and replace it with the output of the external command
        execute a:firstline - 1 .. 'put=output'
    endif
endfunction

The former method is superficially OK and it has the advantage of improving one's current workflow without adding new things to remember. But it is a bit too simple, as it leaves a trail of unnecessary undo/change/jump points and messes up with automatic marks, which is… not optimal. "Doing X and then undoing it" is not the same thing as "not doing X".

The latter method only changes the given lines if the filter is successful, which is much cleaner. It is one more thing to map and remember, though.

IMO, an even better method consists in doing what the latter method does, but as a :help 'formatexpr:

function! FormatExpr()
    let l1 = v:lnum
    let l2 = (v:lnum + v:count) - 1
    let tempfile = tempname()
    call getline(l1, l2)->writefile(tempfile)
    let output = systemlist(&formatprg .. ' ' .. tempfile)
    if v:shell_error
        echohl WarningMsg | echo 'Shell error: ' .. v:shell_error | echohl None
    else
        execute l1 .. ',' .. l2 .. 'd_'
        execute (l1 - 1) .. 'put=output'
    endif
    return 0
endfunction
set formatexpr=FormatExpr()

Note that gq priorities formatexpr over :help 'formatprg', so this approach actually makes formatprg redundant, which can be useful with formatters that don't work as filters. That is what I use in my custom ftplugins.