How to access a custom python module inside vim?

Question

This is a Python 2.7 question. I have a custom python 2.7, that I have compiled and have been using successfully across my system. My custom python has the YAML modules installed as well.

The system wide Python 2.7 is in it's usual path on OSX: /System/Library/Frameworks/Python.framework/Versions/2.7/

function! MyPythonFunction() abort
    echo a:testString
    python << EOF
    import yaml
    print yaml
    print "PYTHON: My Python function."
    EOF
    return 1
endfunction

When I run a Python based function as the one above, I can see that the system wide Python2.7 is active. So for that reason my YAML import fails, as I do not have YAML installed under my system Python.

Is there a way to tell VIM to use my custom Python2.7 that lives inside: /Users/me/apps/Darwin64/python2.7/

or somehow call my custom YAML module?

I have done a bit of research and I think there are things that I can tweak through VIM's Python modules (import vim) and there is a Python2 rtp config. However I am not sure how to set those up.

Thanks.

Answer 1

Here is the answer to this question, however, first things first:

• I do not use macports or brew.
• The ideal workflow for me would be to download the source code of any package, module, library or app and compile it myself.

For those reasons listed above, brew was not an option for me. So here is the solution:

export CC=clang # Will be assumed automatically, most of the times
export vi_cv_path_python=/Users/me/apps/Darwin64/python2.7/bin/python
export vi_cv_dll_name_python=/Users/me/apps/Darwin64/python2.7/lib/libpython2.7.dylib
export DYNAMIC_PYTHON_DLL=/Users/me/apps/Darwin64/python2.7/lib/libpython2.7.dylib

./configure --with-features=huge \
    --enable-cscope=yes \
    --enable-pythoninterp=yes \
    --enable-rubyinterp=yes \
    --enable-perlinterp=yes \
    --enable-luainterp=yes \
    --with-lua-prefix=/Users/me/apps/Darwin64/lua-5.2.3-clang-500.2.79 \
    --with-python-config-dir=/Users/me/apps/Darwin64/python2.7/lib/python2.7/config \
    --with-compiledby=Symbolix

(Some of the ENV variables above, might not be necessary. I not sure about what I did regarding the $PYTHONHOME and $DYLD_LIBRARY_PATH variables. As far as I remember, those were not important for the above command to be successful. You might need to have your custom Python at the front of your $PATH variable though.)

Please note that I am linking against my own custom lua.

Then:

make
cd ./MacVim/build/Release/
open MacVim.app
cp -vr MacVim.app /Applications/

This worked for me on OSX 10.10.5 and VIM 7.4.1553 and I can confirm that VIM is linked properly against my own version of Python2.7. When I type: :py import yaml; print yaml I get the following:

<module 'yaml' from '/Users/me/apps/Darwin64/python2.7/lib/python2.7/site-packages/yaml/__init__.pyc'>

I am still testing this and it seems to be stable and working so far. Now, I am not depending on System Python and I can take care of my own packages etc. If something breaks, I will let you know. Also I do not need to alter any of the sys.path or in fact I do not need to do anything extra to access my own python and the modules inside.

One important aspect is that if you run into trouble compiling this, make sure that you have the latest Xcode and you have run xcode-select --install before compiling VIM from source. If things are still not working make sure to check the build log inside ./src/auto/config.log everything that you need to debug your build process is in there. Good luck.

Answer 2

Vim links against Python; it doesn't call an external Python binary.

You can see this with ldd:

$ ldd /usr/bin/vim | grep python
        libpython2.7.so.1.0 => /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 (0x00007f94673fb000)

This output is for Linux, but your OSX output should be similar.

The only way to use your custom Python is to recompile Vim and link against libpython in /Users/me/apps/Darwin64/python2.7/.

If you want to use a module from this Python, you can do so by modifying sys.path, the default looks something like:

>>> sys.path
['', '/usr/lib/python2.7', '/home/martin/.local/lib/python2.7/site-packages', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages']

This is a list which tells Python where to search for modules. You can modify it like any list. You can also use the PYTHONPATH environment variable to the same effect.

See the documentation for sys.path and import.

---

Alternatively, you can load the module using the imp module. I didn't test this, but you should be able to use load_dynamic. I don't recommend this, since a lot of this changed in Python 3 so your code won't be future-compatible.

Answer 3

On Mac OS X, if homebrew, python3, and vim are major components in your workflow then it works best if you leave the system python alone, don't install python.org Python but rather install python ( 2 and 3 ) using brew install. Then install vim using homebrew also. If you want python3 use the flags --with-python3 --without-python and voila, you will indeed be using the same python in vim and out of vim. pip3 should function as expected and leave your system python alone.

The reasoning behind why I say that you would want not to install the Python.org distribution of python alongside homebrew:

Homebrew wants to use its own python. Even if you are not installing python3 for your direct usage, chances are that something you install with homebrew will need it, and thus it will be installed.

Mac OS X wants to to use it's own inbuilt python (2.7). You're better off leaving this python alone, in terms of installing extra packages.

You want pip (and pip3) to install packages for the same python as what you would get when you go to your shell. You should ideally be able to say $ pip3 install foo at the shell and have 'foo' installed for the python you would reach at a $ python3 prompt.

And, lastly, you want vim to use that same python. If vim is installed with homebrew, your python, homebrew's python, pip's python, and vim's python are all the same, and OS X has it's own.

Now, in this scenario, the addition of Python.org's python adds a third python, greatly complicating matters.

Previously, I was accustomed to installing the Python.org pythons (2 and 3) just after a fresh install. Then, I would install Xcode so I could clone vim from source and build with dynamic python3 support against python3. Upon compiling vim, I would typically clobber the inbuilt vim7.3 of Mac OS X with my own. This was all before I had touched homebrew – I was using MacPorts prior to that. Homebrew follows a different philosophy than MacPorts, and I have found that it is much easier to go with the flow, so to speak, with homebrew and python, because the alternative is to always have to explicitly specify the full path to python whenever using pip. Instead of pip3 install foo it becomes /Library/Frameworks/Python.framework/Versions/3.x/bin/python3 -m pip3 install foo. It is not pretty or fun to have to worry about that.