How to start an async function in Vim 8?

Question

I've known that Vim 8 provides us an async mechanism job_start so that we can start a command asynchronously and get its return with the help of channel.

However, it seems that job_start can only execute the OS command, such as make, grep, /bin/sh etc.

Is it possible to call a vim script function with the help of job_start?

Here is my vimrc:

function! CustomGrep(...)
    // do something
endfunction
command! -nargs=? Grep :call CustomGrep(<f-args>)

As you see, I can :Grep xxx in my vim to call the function CustomGrep. Now I want to make this async, something like this: command! -nargs=? Grep :job_start(...), but I don't know how to achieve this.

I've also tested like this: command! -nargs=? Grep : call job_start('echo 111'), but I can't see 111 in my vim when I execute Grep xxx.

Here is my real custom grep: https://github.com/YvesZHI/Camel-vim/blob/vim8/Camel.vim. You can read this from the line 240 to the end.

In a word, as I start to use vim 8, which provides some kind of async-job mechanism, I'm thinking if it's possible to make my custom grep async. For now my custom grep could work as expected. Sometimes it does take a while to complete but normally it's just 2 or 3 seconds, that's not a big deal. But what if I'm working with a very huge project? So it's gonna be great if I can make my custom grep async.

Answer 1

From discussion in comments I learned that OP has a custom vimgrep call and would like to run it asynchronously. The utility of this is pretty clear, especially if you run a lot of searches that take a while: by moving the execution into the "background" you free up Vim's UI so you can continue working without waiting for the result.

As I mentioned in one of my comments above, a job is used to asynchronously manage a process. Calling a function in Vim is at a different level of granularity. It all takes place in a single, already-running process...Vim's...so jobs aren't going to help here.

Still, there is a relatively lightweight way to do what we want and that's by using a "timer". Timers are usually used to do one or both of the following:

• Start a task after a delay.
• Do a task repeatedly

A sample call makes this pretty clear

let timerid = timer_start(2000, 'RepeatMePlease', { repeat: 5 })

The timer_start() call will return immediately with the assigned timer's id. Meanwhile, in the background, a delay of 2 seconds will occur after which Vim will run the callback function RepeatMePlease() as soon as it starts waiting for input from the user¹. The delay and call will be repeated four more times after that.

What is the timer id for? Well, you can stop, pause, or get bookkeeping info about an existing timer. To do so you obviously need some way to reference it. The timer id.

Anyways, what does that have to do with making a simple background call? Well, if you don't do any repeats and you don't have a delay what's left? A function running in the background.

The one complication here is that, as you may have noticed, there are no parameters specified for RepeatMePlease(). That's because there's nowhere to put them. In fact, when you pass a function name that function will be called by Vim with a single parameter: the timer id.

That's pretty restricting. What ever shall we do?

Fortunately, there's an alternative: we can define a function inline. This is known as a lambda expression in Vim (and elsewhere but sometimes it's called an "anonymous function"). These have a property that is very useful: they act as closures. Huh? All you need to know here is that this allows us to access local variables and parameters. I'm not going to go into much more detail than that. That's what help and google are for. Instead, let's just see it in action. (It looks like a lot but it's mostly comments. There are only three lines of executed code.)

" This is the entry point for the custom vimgrep. Let's say that you
" just pass in a pattern that was entered by the user.
function! PretendCustomGrep(patt) abort
    " Here you can do whatever you need to prepare the grep call then...

    " I've added a 3 second delay so you can prove to your self that this
    " function returns immediately and you can use Vim without interruption.
    " For OP's real world use case the first param would be 0.

    call timer_start(3000, {-> execute("call LongRunningFunc('" . a:patt . "')")})

endfunc

" This does the actual grep stuff
function! LongRunningFunc(patt) abort
    " When you have your result display it, open it, open quickfix, or
    " whatever it is you want to do with it. Here I'm just showing something
    " in a short-lived popup.

    let l:msg = "Search for " . a:patt . " found: A RESULT!"
    call popup_notification(l:msg , #{ line: 5, col:10, highlight: 'Warning', } )

endfunction

Update: You can accomplish the same thing without lambdas and closures by using a Funcref variable. You get a Funcref from function()² and its parameters are a function name and a list. The list contains whatever parameters you need to pass to the callback function. Here's an example of what creation of a Funcref looks like:

    let funcref = function('MyCallback', [p1, p2]) 

When using this with timer_start() you can just inline the function() call as shown in this revised version of the example code above...

function! PretendCustomGrep(patt) abort
    call timer_start(3000, function('LongRunningFunc', [a:patt]))
endfunc

function! LongRunningFunc(patt, timerid) abort
    " Same code as the original, above
endfunction

You probably noticed an important difference between this approach and the original one. In this case Vim appends a new parameter to those that you specify. It contains the timer id. You don't have to use it but you do have to add it to the callback function's signature as I've done with LongRunningFunc().

_{1. Some people (including myself until I learned otherwise) think that the function will be run in a separate thread. This is not the case and explains why execution doesn't start until Vim is idling subsequent to the specified delay time elapsing.}

_{2. There are a couple other ways to create a Funcref so take a look at the documentation if you're interested.}

Answer 2

Is it possible to call a vim script function with the help of job_start?

Not directly. I guess you would need Vim to be multithreaded, which might be possible in the future.

In fact I've used the command :vimgrep to make my custom grep. It worked. Now I starts to use vim 8 so I'm thinking if it's possible to make my custom grep async.

FWIW, I wrote some code with the same purpose a long time ago:

com -nargs=* Vim call s:vimgrep(<q-args>)
com -nargs=* Lvim call s:vimgrep(<q-args>, v:true)

fu s:vimgrep(args, loclist = v:false) abort
    let tempqfl = tempname()
    let tempvimrc = tempname()
    let get_tempfile =<< trim END
        let tempqfl = expand('%:p')
        if tempqfl !~# '^/tmp/'
            finish
        endif
    END
    let cdcmd = 'cd ' .. getcwd()->fnameescape()
    let setcmd = printf('set wildignore=%s suffixes=%s %signorecase %ssmartcase',
        \ &wildignore, &suffixes, &ignorecase ? '' : 'no', &smartcase ? '' : 'no')
    let args = s:expandargs(a:args)
    let vimgrepcmd = 'noa vim ' .. args
    let getqfl =<< trim END
        eval getqflist()
           \ ->map({_, v -> printf('%s:%d:%d:%s',
           \     bufname(v.bufnr)->fnamemodify(':p'),
           \     v.lnum,
           \     v.col,
           \     substitute(v.text, '[^[:print:]]', {m -> strtrans(m[0])}, 'g')
           \ )})
           \ ->writefile(tempqfl, 's')
        qa!
    END
    call writefile(['vim9script']
        \ + get_tempfile
        \ + [cdcmd, setcmd, vimgrepcmd]
        \ + getqfl,
        \ tempvimrc, 's')
    let vimcmd = printf('vim -es -Nu NONE -U NONE -i NONE -S %s %s', tempvimrc, tempqfl)
    let title = (a:loclist ? ':Lvim ' : ':Vim ') .. args
    let arglist = [a:loclist, tempqfl, title]
    let opts = #{exit_cb: function('s:callback', arglist)}
    call split(vimcmd)->job_start(opts)
endfu

fu s:callback(loclist, tempqfl, title, _j, _e) abort
    let efm_save = &l:efm
    let bufnr = bufnr('%')
    try
        setl efm=%f:%l:%c:%m
        if a:loclist
            exe 'lgetfile ' .. a:tempqfl
            lw
            call setloclist(0, [], 'a', #{title: a:title})
        else
            exe 'cgetfile ' .. a:tempqfl
            cw
            call setqflist([], 'a', #{title: a:title})
        endif
    finally
        call setbufvar(bufnr, '&efm', efm_save)
    endtry
    redraw!
endfu

fu s:expandargs(args) abort
    let pat = '^\(\i\@!.\)\1\ze[gj]\{,2}\s\+'
    let rep = '/' .. escape(@/, '\/') .. '/'
    return substitute(a:args, pat, rep, '')
        \ ->substitute('\s\+\zs%\s*$', expand('%:p')->fnameescape(), '')
        \ ->substitute('\s\+\zs##\s*$', argv()
        \ ->map({_, v -> fnamemodify(v, ':p')->fnameescape()})
        \ ->join(), '')
endfu

The code installs 2 custom commands :Vim and :Lvim which are some kind of asynchronous wrappers around the builtin :vimgrep and :lvimgrep. As an example, you can test it by running this command:

:Vim /pattern/gj $VIMRUNTIME/**

This should asynchronously populate a quickfix list with all the lines where pattern is written in any file in your $VIMRUNTIME directory.

It seems to work, but I don't use it regularly, so I don't know how reliable it is.

---

Re-written in Vim9 script:

vim9script

com -nargs=* Vim Vimgrep(<q-args>)
com -nargs=* Lvim Vimgrep(<q-args>, true)

def Vimgrep(args: string, loclist = false)
    var tempvimrc: string = tempname()
    var tempqfl: string = tempname()
    var get_tempfile: list<string> =<< trim END
        var tempqfl = expand('%:p')
        if tempqfl !~ '^/tmp/'
            finish
        endif
    END
    var cdcmd: string = 'cd ' .. getcwd()->fnameescape()
    var setcmd: string = printf('set wildignore=%s suffixes=%s %signorecase %ssmartcase',
        &wildignore, &suffixes, &ignorecase ? '' : 'no', &smartcase ? '' : 'no')
    var _args: string = Expandargs(args)
    var vimgrepcmd: string = 'noa vim ' .. _args
    var getqfl: list<string> =<< trim END
        getqflist()
           ->mapnew((_, v: dict<any>): string => printf('%s:%d:%d:%s',
               bufname(v.bufnr)->fnamemodify(':p'),
               v.lnum,
               v.col,
               substitute(v.text, '[^[:print:]]', (m: list<string>): string => strtrans(m[0]), 'g')
               ))
           ->writefile(tempqfl, 's')
        qa!
    END
    writefile(['vim9script']
        + get_tempfile
        + [cdcmd, setcmd, vimgrepcmd]
        + getqfl,
        tempvimrc, 's')
    var vimcmd: string = printf('vim -es -Nu NONE -U NONE -i NONE -S %s %s', tempvimrc, tempqfl)
    var title: string = (loclist ? ':Lvim ' : ':Vim ') .. _args
    var arglist: list<any> = [loclist, tempqfl, title]
    var opts: dict<func> = {exit_cb: function(Callback, arglist)}
    split(vimcmd)->job_start(opts)
enddef

def Callback(loclist: bool, tempqfl: string, title: string, _j: job, _e: number)
    var efm_save: string = &l:efm
    var bufnr: number = bufnr('%')
    try
        setl efm=%f:%l:%c:%m
        if loclist
            exe 'lgetfile ' .. tempqfl
            lw
            setloclist(0, [], 'a', {title: title})
        else
            exe 'cgetfile ' .. tempqfl
            cw
            setqflist([], 'a', {title: title})
        endif
    finally
        setbufvar(bufnr, '&efm', efm_save)
    endtry
    redraw!
enddef

def Expandargs(args: string): string
    var pat: string = '^\(\i\@!.\)\1\ze[gj]\{,2}\s\+'
    var rep: string = '/' .. escape(@/, '\/') .. '/'
    return substitute(args, pat, rep, '')
        ->substitute('\s\+\zs%\s*$', expand('%:p')->fnameescape(), '')
        ->substitute('\s\+\zs##\s*$', argv()
        ->map((_, v: string): string => fnamemodify(v, ':p')->fnameescape())
        ->join(), '')
enddef

This requires a recent Vim version. It works on 8.2.2332.