Bug 645

Summary: Building the bindings for Python3 doesn't work
Product: libsigrok Reporter: Uwe Hermann <uwe>
Component: Bindings: PythonAssignee: Nobody <nobody>
Status: RESOLVED FIXED    
Severity: normal CC: daniel.kitta, martin-sigrokbugs
Priority: Normal    
Version: unreleased development snapshot   
Target Milestone: ---   
Hardware: All   
OS: All   

Description Uwe Hermann 2015-09-03 11:36:34 CEST
When building the Python bindings like this:

 $ PYTHON=python3 ./configure ...

doing "import sigrok.core" doesn't work:


Python 3.4.3+ (default, Jul 28 2015, 13:17:50) 
[GCC 4.9.3] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sigrok.core
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/tmp/py3/lib/python3.4/site-packages/libsigrok-0.4.0-py3.4-linux-x86_64.egg/sigrok/core/__init__.py", line 20, in <module>
    from . import classes
  File "/tmp/py3/lib/python3.4/site-packages/libsigrok-0.4.0-py3.4-linux-x86_64.egg/sigrok/core/classes.py", line 53, in <module>
    _classes = swig_import_helper()
  File "/tmp/py3/lib/python3.4/site-packages/libsigrok-0.4.0-py3.4-linux-x86_64.egg/sigrok/core/classes.py", line 49, in swig_import_helper
    _mod = imp.load_module('_classes', fp, pathname, description)
  File "/usr/lib/python3.4/imp.py", line 243, in load_module
    return load_dynamic(name, filename, file)
ImportError: dynamic module does not define init function (PyInit__classes)

(for a Python2 build it works fine)


This is likely due to PYTHON=python3 not being entirely sufficient to force the use of Python 3.

Example:
 $ ldd /tmp/py3/lib/python3.4/site-packages/libsigrok-0.4.0-py3.4-linux-x86_64.egg/sigrok/core/*so | grep python
        libpython2.7.so.1.0 => /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 (0x00007fda3a583000)

(this should probably be libpython3.4.so*)
Comment 1 Uwe Hermann 2015-09-03 12:59:13 CEST
Python 2 "make install":

destdir=''; /usr/bin/python ../bindings/python/setup.py VERSION='0.4.0' CC='g++' CFLAGS='-Wall -Wextra -I/usr/include/python2.7 -I/usr/include/x86_64-linux-gnu/python2.7 -I/usr/include/pygobject-3.0 -I/usr/include/glibmm-2.4 -I/usr/lib/x86_64-linux-gnu/glibmm-2.4/include -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -I/usr/include/sigc++-2.0 -I/usr/lib/x86_64-linux-gnu/sigc++-2.0/include  -g -O2 -std=c++11' LDADD='-lpython2.7 -lglibmm-2.4 -lgobject-2.0 -lglib-2.0 -lsigc-2.0 ' --quiet install ${destdir:+"--root=$destdir"} \
        --prefix "/home/uwe/sr" --exec-prefix "/home/uwe/sr"
warning: manifest_maker: standard file 'setup.py' not found

Python 3 "make install":

destdir=''; python3 ../bindings/python/setup.py VERSION='0.4.0' CC='g++' CFLAGS='-Wall -Wextra -I/usr/include/python2.7 -I/usr/include/x86_64-linux-gnu/python2.7 -I/usr/include/pygobject-3.0 -I/usr/include/glibmm-2.4 -I/usr/lib/x86_64-linux-gnu/glibmm-2.4/include -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -I/usr/include/sigc++-2.0 -I/usr/lib/x86_64-linux-gnu/sigc++-2.0/include  -g -O2 -std=c++11' LDADD='-lpython2.7 -lglibmm-2.4 -lgobject-2.0 -lglib-2.0 -lsigc-2.0 ' --quiet install ${destdir:+"--root=$destdir"} \
        --prefix "/home/uwe/sr" --exec-prefix "/home/uwe/sr"
Comment 2 Martin Ling 2015-09-03 13:30:01 CEST
This problem arises from here in configure.ac:

SR_PKG_CHECK([python_dev], [SR_PKGLIBS_PYTHON],
  [python >= 2.7], [python2 >= 2.7], [python27 >= 2.7])

That finds the python2.7 stuff from pkg-config, which then makes it into CFLAGS and CXXFLAGS and eventually into the setup.py invocations in Makefile.am:

setup_vars = VERSION='$(PACKAGE_VERSION)' CC='$(CXX)' CFLAGS='$(AM_CXXFLAGS) $(CXXFLAGS)' LDADD='$(PYSIGROK_LIBS)'
setup_quiet = --quiet
setup_py = $(PYTHON) $(srcdir)/$(PDIR)/setup.py $(setup_vars) $(setup_quiet)

So 'make python-build ends up running this:

python3 ./bindings/python/setup.py VERSION='0.4.0' CC='g++' CFLAGS='-Wall -Wextra -I/usr/include/python2.7 -I/usr/include/x86_64-linux-gnu/python2.7 -I/usr/include/pygobject-3.0 -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -I/usr/include/glibmm-2.4 -I/usr/lib/x86_64-linux-gnu/glibmm-2.4/include -I/usr/include/sigc++-2.0 -I/usr/lib/x86_64-linux-gnu/sigc++-2.0/include   -g -O2 -std=c++11' LDADD='-lpython2.7 -lglibmm-2.4 -lgobject-2.0 -lsigc-2.0 -lglib-2.0  ' --quiet clean --all 2>/dev/null

...which obviously doesn't make much sense.
Comment 3 Daniel Elstner 2015-11-01 21:17:12 CET
OK, it seems the problem is that CFLAGS, LDFLAGS, etc. are now passed in from the Makefile (which gets them from configure) to setup.py. These flags also include the Python include paths and libs obtained via pkg-config by configure.

If one disables the pkg-config check for the Python headers in configure.ac, things start to work. Apparently setup.py adds its own flags for Python independently of what was passed in. The custom flags seem to override the builtin ones, which may be due to ordering.

So, one way to make things work as they did before would be to stop getting the Python flags from pkg-config. However, that would likely break cross compiling of the Python bindings.

The other option is to select the pkg-config module based on the Python interpreter version. However, that would mean we are getting flags from two different sources (pkg-config and setuptools) and it likely comes down to ordering who's winning. (Assuming that setuptools does not attempt to strip include paths that look like python header paths, and just always adds its own. Whether that is actually true needs to be verified.)
Comment 4 Uwe Hermann 2015-11-02 19:27:15 CET
Fixed in 3d8691326666a32925aa555f3088185ccd5d846b, thanks!

This seems to work fine for me. I'm using "PYTHON=python3 ./configure" for the Python 3 case, and "./configure" for the (default) Python 2 case. Tested with sigrok-meter against both versions (using the demo driver only, though).

Only tested on Linux, no other OSes, no cross-compile setups. More work may or may not be needed for any of those.