26.0.3 [BUG]: pytest fails in zmq/backend/cython/_zmq.py in cython.compiled import

[kloczek]

This is a pyzmq bug

• This is a pyzmq-specific bug, not an issue of zmq socket behavior. Don't worry if you're not sure! We'll figure it out together.

What pyzmq version?

26.0.3

What libzmq version?

4.3.5

Python version (and how it was installed)

3.11.14

OS

Linux/x86_64

What happened?

Looks like pytest fails in zmq/backend/cython/_zmq.py in cython.compiled import. below taht is another import of cython.cimports.cpython and there is no such module as well in cython 3.0.10 there is no such module.

[tkloczko@pers-jacek SPECS]$ python3
Python 3.10.14 (main, Apr  3 2024, 21:50:31) [GCC 14.0.1 20240328 (Red Hat 14.0.1-0)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cython
>>> import cython.compiled
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'cython.compiled'; 'cython' is not a package
>>>
>>> import cython.cimports.cpython
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'cython.cimports.cpython'

List of installed modules in build env:

Package                       Version
----------------------------- -----------
alabaster                     0.7.16
apeye                         1.4.1
apeye-core                    1.1.4
autodocsumm                   0.2.12
Babel                         2.15.0
beautifulsoup4                4.12.3
build                         1.2.1
CacheControl                  0.14.0
charset-normalizer            3.3.2
cssutils                      2.10.2
Cython                        3.0.10
defusedxml                    0.7.1
dict2css                      0.3.0.post1
docutils                      0.20.1
domdf_python_tools            3.8.0.post2
enum-tools                    0.12.0
exceptiongroup                1.1.3
filelock                      3.14.0
gevent                        24.2.1
greenlet                      3.0.3
html5lib                      1.1
idna                          3.7
imagesize                     1.4.1
importlib_metadata            7.1.0
iniconfig                     2.0.0
installer                     0.7.0
Jinja2                        3.1.4
jupyterlab_pygments           0.3.0
linkify-it-py                 2.0.3
markdown-it-py                3.0.0
MarkupSafe                    2.1.5
mdit-py-plugins               0.4.1
mdurl                         0.1.2
msgpack                       1.0.8
myst-parser                   3.0.1
natsort                       8.4.0
packaging                     24.0
pathspec                      0.12.1
platformdirs                  4.2.2
pluggy                        1.5.0
Pygments                      2.18.0
pyproject_hooks               1.0.0
pyproject-metadata            0.8.0
pytest                        8.2.1
pytest-asyncio                0.23.7
python-dateutil               2.9.0.post0
PyYAML                        6.0.1
requests                      2.32.2
ruamel.yaml                   0.18.5
ruamel.yaml.clib              0.2.8
scikit_build_core             0.9.4
setuptools                    69.4.0
setuptools-scm                8.1.0
snowballstemmer               2.2.0
soupsieve                     2.5
Sphinx                        7.3.7
sphinx-autodoc-typehints      2.1.0
sphinx-jinja2-compat          0.2.0.post1
sphinx-prompt                 1.6.0
sphinx_rtd_theme              2.0.0
sphinx-tabs                   3.4.5
sphinx-toolbox                3.5.0
sphinxcontrib-applehelp       1.0.8
sphinxcontrib-devhelp         1.0.6
sphinxcontrib-htmlhelp        2.0.5
sphinxcontrib-jquery          4.1
sphinxcontrib-jsmath          1.0.1
sphinxcontrib-qthelp          1.0.7
sphinxcontrib-serializinghtml 1.1.10
tabulate                      0.9.0
tokenize_rt                   5.2.0
tomli                         2.0.1
tornado                       6.3.3
typing_extensions             4.12.0
uc-micro-py                   1.0.3
urllib3                       2.2.1
webencodings                  0.5.1
wheel                         0.43.0
zipp                          3.18.2
zope.event                    5.0
zope.interface                6.4.post2

Code to reproduce bug

Just run pytest

Traceback, if applicable

+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-zmq-26.0.3-2.fc37.x86_64/usr/lib64/python3.10/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-zmq-26.0.3-2.fc37.x86_64/usr/lib/python3.10/site-packages
+ /usr/bin/pytest -ra -m 'not network' --import-mode=importlib
ImportError while loading conftest '/home/tkloczko/rpmbuild/BUILD/pyzmq-26.0.3/zmq/tests/conftest.py'.
zmq/__init__.py:52: in <module>
    from zmq import backend
zmq/backend/__init__.py:30: in <module>
    raise original_error from None
zmq/backend/__init__.py:25: in <module>
    _ns = select_backend(first)
zmq/backend/select.py:31: in select_backend
    mod = import_module(name)
zmq/backend/cython/__init__.py:6: in <module>
    from zmq.backend.cython import _zmq
zmq/backend/cython/_zmq.py:14: in <module>
    raise ImportError("zmq Cython backend has not been compiled") from None
E   ImportError: zmq Cython backend has not been compiled

More info

No response

[minrk]

As the error says, this means pyzmq is imported from the source directory and hasn't been built yet. However you build or install pyzmq, make sure that's imported.

To run from source, do:

pip install -e .

[kloczek]

pep517 based procedure does not build in-palce.
To fix that it needs to be removed use all relative imports.
I made such patch which as well unlocks "test as installed" methodology.
Here is my patch

--- a/zmq/__init__.py
+++ b/zmq/__init__.py
@@ -51,8 +51,8 @@
 with _libs_on_path():
     from zmq import backend

-from . import constants  # noqa
-from .constants import *  # noqa
+from zmq import constants  # noqa
+from zmq.constants import *  # noqa
 from zmq.backend import *  # noqa
 from zmq import sugar
 from zmq.sugar import *  # noqa
--- a/zmq/__init__.pyi
+++ b/zmq/__init__.pyi
@@ -1,6 +1,6 @@
 from typing import List

-from . import backend, sugar
+from zmq import backend, sugar

 COPY_THRESHOLD: int
 DRAFT_API: bool
@@ -11,19 +11,19 @@
 # see tools/backend_imports.py to generate this list
 # note: `x as x` is required for re-export
 # see https://github.com/python/mypy/issues/2190
-from .backend import IPC_PATH_MAX_LEN as IPC_PATH_MAX_LEN
-from .backend import curve_keypair as curve_keypair
-from .backend import curve_public as curve_public
-from .backend import device as device
-from .backend import has as has
-from .backend import proxy as proxy
-from .backend import proxy_steerable as proxy_steerable
-from .backend import strerror as strerror
-from .backend import zmq_errno as zmq_errno
-from .backend import zmq_poll as zmq_poll
-from .constants import *
-from .error import *
-from .sugar import *
+from zmq.backend import IPC_PATH_MAX_LEN as IPC_PATH_MAX_LEN
+from zmq.backend import curve_keypair as curve_keypair
+from zmq.backend import curve_public as curve_public
+from zmq.backend import device as device
+from zmq.backend import has as has
+from zmq.backend import proxy as proxy
+from zmq.backend import proxy_steerable as proxy_steerable
+from zmq.backend import strerror as strerror
+from zmq.backend import zmq_errno as zmq_errno
+from zmq.backend import zmq_poll as zmq_poll
+from zmq.constants import *
+from zmq.error import *
+from zmq.sugar import *

 def get_includes() -> list[str]: ...
 def get_library_dirs() -> list[str]: ...
--- a/zmq/backend/__init__.py
+++ b/zmq/backend/__init__.py
@@ -6,7 +6,7 @@
 import os
 import platform

-from .select import public_api, select_backend
+from zmq.backend.select import public_api, select_backend

 if 'PYZMQ_BACKEND' in os.environ:
     backend = os.environ['PYZMQ_BACKEND']
--- a/zmq/backend/__init__.pyi
+++ b/zmq/backend/__init__.pyi
@@ -4,7 +4,7 @@

 import zmq

-from .select import select_backend
+from zmq.backend.select import select_backend

 # avoid collision in Frame.bytes
 _bytestr = bytes
--- a/zmq/backend/cython/__init__.py
+++ b/zmq/backend/cython/__init__.py
@@ -3,11 +3,11 @@
 # Copyright (C) PyZMQ Developers
 # Distributed under the terms of the Modified BSD License.

-from . import _zmq
+from zmq.backend.cython import _zmq

 # mq not in __all__
-from ._zmq import *  # noqa
-from ._zmq import monitored_queue  # noqa
+from zmq.backend.cython._zmq import *  # noqa
+from zmq.backend.cython._zmq import monitored_queue  # noqa

 Message = _zmq.Frame

I'm using typical procedure used on package python modules as rpm package consisting from:
I'm packaging your module as an rpm package so I'm using the typical PEP517 based build, install and test cycle used on building packages from non-root account.

• python3 -sBm build -w --no-isolation
• because I'm calling build with --no-isolation I'm using during all processes only locally installed modules
• install .whl file in </install/prefix> using installer module
• run pytest with $PYTHONPATH pointing to sitearch and sitelib inside </install/prefix>

pep517 stores all temporary files in build/ and not in source tree.
All files before forming .whl are stored in build/lib*/ directory.
Tere is no any problems with building module using pep517 procedure.

Please have look one more time what is imported in zmq/backend/cython/_zmq.py.
Message mases not on possibility to import some DSOs of pyzmq but on possibility to import cython submodule which does not exist.

[minrk]

I think I need to work on the error message, because this is precisely the situation it is for. If _zmq.py is imported as a Python file, that means that uncompiled Cython is being imported, which will not work. This will generally be because pyzmq is being imported from the zmq source tree without an in-place build (e.g. pip install -e .). If you have done a regular install of pyzmq, the source tree is not importable.

To load tests from an installed package instead of a source tree with pytest (not specific to pyzmq), use --pyargs. For pyzmq, that would be pytest --pyargs zmq.tests. This is, ultimately, the same as #1550 with the same solution, just a different error message when trying to import uncompiled Cython.

Now that BaseZMQTestCase has been deprecated for a while, I can probably safely move the tests outside of the package, which I think should also avoid this issue.

[kloczek]

Moving test outside module tree definitely it is good move 👍

Please consider remove as well relative imports.
Each such import forces additional syscall to obtain current directory and as I wrote breaks typical test procedure used used on packaging from non-root account without tox or venv.

Default pytest args woild be good be good to put in pyproject.toml or pytest.ini.

[minrk]

The only thing required to test against an install is to tell pytest to find the tests in the package:

pytest --pyargs zmq.tests

No patches or anything. Note that pytest zmq/tests (the default test paths) cannot be expected to work except with a development install. --pyargs must be used and is the only way to run tests against the installed package.

I think #1994 resolves all of this, removing the --pyargs issue. I'm not sure what you mean by relative imports "breaking typical test procedure" because anything that is broken by relative imports in an installed package is not typical, since by definition it means imports have been messed up.