Neovim: How to start developing python plugins which run long-blocking code?

Question

EDIT2: A user suggested that I may be going about this in a fundamentally wrong way, in particular in that I am not using remote (new-style) plugins and am trying to use async python code where it might not belong. As such, I would like to briefly reframe the question here and then leave the original text below.

I am a novice attempting to develop a neovim plugin in python3 that runs some long-blocking operations on startup and then reports a result. I would like to do this asynchronously for obvious reasons. Can anyone provide a concrete, minimal example of how to do that that can get me on my feet and coding in the right direction, assuming no particular knowledge of vim plugin development?

ORIGINAL TEXT:

I have a long-running (specifically, long-blocking) plugin written in python that runs once at startup and reports its result when finished. Let's imagine it as

import vim
import time

def plugin_sync():
    time.sleep(5)
    print("done")

vim.async_call(plugin_sync)

I had hoped this wouldn't impact startup, and indeed it doesn't, if run in isolation.

However, if I add certain other plugins, say using vim-plug and an init vim with lots of work to do like

call plug#begin('~/.local/share/nvim/plugged')
Plug 'me/my-long-running-plugin'
Plug 'neoclide/coc.nvim', {'branch': 'release'}
call plug#end()

let g:coc_global_extensions=["coc-json", "coc-eslint", "coc-tsserver",
  \ "coc-snippets", "coc-html", "coc-css", "coc-python", "coc-java",
  \ "coc-highlight", "coc-yank", "coc-omnisharp", "coc-emmet",
  \ "coc-lists", "coc-marketplace", "coc-neosnippet", "coc-prettier",
  \ "coc-clangd", "coc-cmake", "coc-xml" ]

, nvim becomes unresponsive for 5 seconds. Per the docs,

Note that this code will still block the plugin host if it does long-running computations. Intensive computations should be done in a separate thread (or process), and vim.async_call can be used to send results back to Neovim.

, so I'm guessing that the unresponsiveness has to do with another plugin blocking nvim while waiting for the plugin host.

Briefly ignoring the advice in the docs (I'll get back to that), it seemed like maybe the right thing to do was just use proper async python as follows:

import vim
import asyncio

async def plugin_async():
    await asyncio.sleep(5)
    print("done")

vim.async_call(lambda: asyncio.create_task(plugin_async()))

This doesn't block, but it also doesn't print messages. I know that it's running, because I can insert some side effect like touching a file, but I would also like to be able to print a message after finishing, which is really the whole point of the plugin.

My questions, in order of specificity, are as follows:

1. How can I print messages or otherwise deliver feedback to the user from python code running in the asyncio event loop?
2. Is what I've done "valid," in the sense of not having any weird consequences I may not have imagined?
3. Circling back to the docs, what is the "standard" way to do this? I've seen suggestions like use neomake or vim-dispatch, or use job control, but I'm very novice with vim script, and I can't say I particularly want to learn a bunch of new concepts to accomplish something that it looks like I can probably just do in pure python. That said, if my way is infeasible or otherwise "wrong," it would be really helpful if responders could give some concrete examples of how to do e.g. what the docs suggest but don't elaborate on.

EDIT: I noticed that vim.eval doesn't work properly inside of async functions:

import vim
import asyncio
import warnings

async def plugin_async():
    try:
        vim.eval("g:defined_var")
    except Exception as e:
        warnings.warn(e)

vim.async_call(lambda: asyncio.create_task(plugin_async()))

Note that I'm using warnings for convenience because they are communicated back to the user, whereas uncaught exceptions are suppressed. This produces

UserWarning: 'NoneType' object has no attribute 'switch'

, which is as yet a mystery to me. The same code works just fine in a sync function, so there's something else going on here that I don't understand. Help following what's going on would be much appreciated.

Answer 1

Although this question raises some as yet unanswered questions about how to best handle doing long-running stuff in the traditional :python style plugin, it seems mostly moot now that I've spent some time understanding the new style remote plugin system. nvim-example-python-plugin was helpful (though slightly outdated), so I'd recommend anyone else finding themselves floundering start there.

The key observation is this: remote plugins are already running in their own processes (hence remote), and the vim object is used for RPC. As such, it doesn't matter at all if the plugin blocks, so long as it doesn't instruct the plugin manager to do any blocking work, which was of course exactly what I was doing using vim.async_call to do blocking work.

As for python async code, there's a line at the end of Nvim.__init__ that exposes the event loop for python 3, so if one wanted to run async code (python >= 3.5), one can schedule tasks using vim.loop.

Here's a minimal example:

import pynvim
from pynvim.api.nvim import Nvim
from time import sleep
import asyncio


@pynvim.plugin
class MyPlugin(object):
    def __init__(self, nvim: Nvim) -> None:
        self.nvim = nvim

    @pynvim.function("LongBlock")
    def long_block(self, args):
        sleep(5)
        self.nvim.command('echo "done with blocking stuff"')

    @pynvim.function("AsyncNoBlock")
    def async_no_block(self, args):
        async def no_block():
            await asyncio.sleep(5)
            self.nvim.command('echo "done with async stuff"')

        self.nvim.loop.create_task(no_block())

Weirdly, vim.async_call(print, "message") doesn't echo, which I thought it would based on @filbranden's comment. I'm going to keep this unanswered until that is resolved, and perhaps until more experienced others can point out how I'm still probably missing some other things.