1

I (very) often write python, and I use Vim's python3 backend to test my code. To perform my tests I use:

vnoremap <localleader>p y:<c-r>"<c-b>python3 <cr>

It simply takes my visual selection and runs within Vim's python backend. This is very practical because I can re-run only pieces of code (and I'm careful to remember the state the repl is in).

This works fine even with installed libraries, for example beautifulsoup4 works:

:python3 import bs4

But my issue starts when I try loading libraries that are not fully written in python. For example trying numpy:

:python3 import numpy

I get an error from the backend:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/grochmal/.local/lib/python3.5/site-packages/numpy/__init__.py", line 180, in <module>
    from . import add_newdocs
  File "/home/grochmal/.local/lib/python3.5/site-packages/numpy/add_newdocs.py", line 13, in <module>
    from numpy.lib import add_newdoc
  File "/home/grochmal/.local/lib/python3.5/site-packages/numpy/lib/__init__.py", line 8, in <module>
    from .type_check import *
  File "/home/grochmal/.local/lib/python3.5/site-packages/numpy/lib/type_check.py", line 11, in <module>
    import numpy.core.numeric as _nx
  File "/home/grochmal/.local/lib/python3.5/site-packages/numpy/core/__init__.py", line 14, in <module>
    from . import multiarray
ImportError: /home/grochmal/.local/lib/python3.5/site-packages/numpy/core/multiarray.cpython-35m-x86_64-linux-gnu.so: undefine
d symbol: PyType_GenericNew

After debugging it a good deal I got to the conclusion that the issue only happens in /home/grochmal/.local and only to libraries that have components compiled from C. In other words, if I install numpy into /usr/lib it works.

The question

The python REPL has no issue with libraries at ~/.local but Vim's python backend does. I tried with and without:

export PYTHONPATH=/home/grochmal/.local/lib/python3.5/site-packages

And the issue persists. Is there a way to use locally installed shared libraries together with Vim's python backend? i.e. Can I tell Vim's python backend to search both places (/usr/lib and ~/.local) for symbols to load?


Extra note: both bs4 and numpy are in ~/.local in the tests above. In other words libraries that are fully written in python do work in ~/.local

  • PS: I do use makeprg=python3\ %, but that is just too slow when huge libraries are being loaded or a lot of data is read from files. Yes, i'm using Vim like an ide, shame on me :) – grochmal Oct 13 '16 at 19:15
2

This is because your numpy isn't linked against the Python 3 library (no -lpython3 used). This is fine for most applications that has the library loaded into global space, but Vim uses RTLD_LOCAL so numpy's libraries don't see Python 3's symbols unless it's linked against it.

LD_PRELOAD is fine as long as you don't load Python 2 in the same Vim (or symbols may clash; that's why Vim doesn't use RTLD_GLOBAL). You can also recompile your numpy with -lpython3 (from Arch's PKGBUILD it seems that export LDFLAGS=-shared is enough to do that).

  • 1
    Whelp, yeah. I was almost there, but not quite. Tested it (recompiled my numpy with -lpython3) and it works. p2k cannot go away soon enough (I know that people are saying that for 5 years already and will probably not happen in the next 5 either). Many thanks. – grochmal Oct 16 '16 at 18:51
0

Workaround

After a good deal of debugging I found that forcing Vim to have the shared library in memory all the time sorts the problem. i.e.

alias vim='LD_PRELOAD=/usr/lib/libpython3.so vim'

Yes, it is horrible, but works on most systems (tested on CentOS, Debian and Arch). Another problem is that LD_PRELOAD is Linux specific.

After this I can load a numpy library from ~/.local/lib/python3.5/site-packages/ with :python3 import numpy.


Extra info

I'm still baffled at the issue since in Vim code the library should come from:

src/option.h: EXTERN char_u *p_py3dll;  /* 'pythonthreedll' */

Which in turn should come from -DDYNAMIC_PYTHON3_DLL, which should be a static makefile argument. Therefore, I believe that the issue lies somewhere in (src/if_python3.c):

# ifndef WIN3264
#  include <dlfcn.h>
#  define FARPROC void*
#  define HINSTANCE void*
#  if defined(PY_NO_RTLD_GLOBAL) && defined(PY3_NO_RTLD_GLOBAL)
#   define load_dll(n) dlopen((n), RTLD_LAZY)
#  else
#   define load_dll(n) dlopen((n), RTLD_LAZY|RTLD_GLOBAL)
#  endif
...

And that RTLD_LOCAL (i.e. no RTLD_GLOBAL) is possibly used. But I never used dlopen() and may be completely wrong about this.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.