Further problems with locale

[pp-mo]

As reported here in the context of iris-esmf-regrid,
the current save - modify - restore code for the locale setting it seems does not always work correctly.
This was reported in #435, fixed by #436, introduced in v3.3.0

Not clear yet what is actually wrong, but for iris-esmf-regrid it seems to occur specifically in readthedocs builds.
The case given should hopefully point to it.

Detail

ReadTheDocs log captured here from a ReadTheDocs build test (i.e. its error log) on commit SciTools/iris-esmf-regrid@b5edcd7 , in SciTools/iris-esmf-regrid#431.
Since we can't expect that record to persist

python -m sphinx -T -b html -d _build/doctrees -D language=en . $READTHEDOCS_OUTPUT/html Running Sphinx v8.1.3

Traceback (most recent call last):
File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/esmf_regrid-0.12.dev0-py3.10.egg/esmf_regrid/init.py", line 11, in
import iris.mesh as _imesh
ModuleNotFoundError: No module named 'iris.mesh'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/sphinx/config.py", line 529, in eval_config_file
exec(code, namespace) # NoQA: S102
File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/checkouts/431/docs/src/conf.py", line 34, in
from esmf_regrid import version as esmf_r_version
File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/esmf_regrid-0.12.dev0-py3.10.egg/esmf_regrid/init.py", line 14, in
import iris.experimental.ugrid as _imesh
File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/iris/experimental/ugrid/init.py", line 18, in
from .load import PARSE_UGRID_ON_LOAD, load_mesh, load_meshes
File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/iris/experimental/ugrid/load.py", line 22, in
from ...coords import AuxCoord
File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/iris/coords.py", line 23, in
from iris.common import (
File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/iris/common/init.py", line 9, in
from .mixin import *
File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/iris/common/mixin.py", line 10, in
import cf_units
File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/cf_units/init.py", line 198, in
with suppress_errors(), c_locale():
File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/contextlib.py", line 142, in exit
next(self.gen)
File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/cf_units/init.py", line 191, in c_locale
locale.setlocale(locale.LC_NUMERIC, lc_numeric)
File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/locale.py", line 620, in setlocale
return _setlocale(category, locale)
locale.Error: unsupported locale setting

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/sphinx/cmd/build.py", line 496, in build_main
app = Sphinx(
File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/sphinx/application.py", line 238, in init
self.config = Config.read(self.confdir, confoverrides or {}, self.tags)
File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/sphinx/config.py", line 316, in read
namespace = eval_config_file(filename, tags)
File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/sphinx/config.py", line 542, in eval_config_file
raise ConfigError(msg % traceback.format_exc()) from exc
sphinx.errors.ConfigError: There is a programmable error in your configuration file:

Traceback (most recent call last):
File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/esmf_regrid-0.12.dev0-py3.10.egg/esmf_regrid/init.py", line 11, in
import iris.mesh as _imesh
ModuleNotFoundError: No module named 'iris.mesh'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/sphinx/config.py", line 529, in eval_config_file
exec(code, namespace) # NoQA: S102
File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/checkouts/431/docs/src/conf.py", line 34, in
from esmf_regrid import version as esmf_r_version
File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/esmf_regrid-0.12.dev0-py3.10.egg/esmf_regrid/init.py", line 14, in
import iris.experimental.ugrid as _imesh
File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/iris/experimental/ugrid/init.py", line 18, in
from .load import PARSE_UGRID_ON_LOAD, load_mesh, load_meshes
File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/iris/experimental/ugrid/load.py", line 22, in
from ...coords import AuxCoord
File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/iris/coords.py", line 23, in
from iris.common import (
File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/iris/common/init.py", line 9, in
from .mixin import *
File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/iris/common/mixin.py", line 10, in
import cf_units
File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/cf_units/init.py", line 198, in
with suppress_errors(), c_locale():
File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/contextlib.py", line 142, in exit
next(self.gen)
File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/cf_units/init.py", line 191, in c_locale
locale.setlocale(locale.LC_NUMERIC, lc_numeric)
File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/locale.py", line 620, in setlocale
return _setlocale(category, locale)
locale.Error: unsupported locale setting

Configuration error:
There is a programmable error in your configuration file:

Traceback (most recent call last):
File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/esmf_regrid-0.12.dev0-py3.10.egg/esmf_regrid/init.py", line 11, in
import iris.mesh as _imesh
ModuleNotFoundError: No module named 'iris.mesh'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/sphinx/config.py", line 529, in eval_config_file
exec(code, namespace) # NoQA: S102
File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/checkouts/431/docs/src/conf.py", line 34, in
from esmf_regrid import version as esmf_r_version
File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/esmf_regrid-0.12.dev0-py3.10.egg/esmf_regrid/init.py", line 14, in
import iris.experimental.ugrid as _imesh
File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/iris/experimental/ugrid/init.py", line 18, in
from .load import PARSE_UGRID_ON_LOAD, load_mesh, load_meshes
File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/iris/experimental/ugrid/load.py", line 22, in
from ...coords import AuxCoord
File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/iris/coords.py", line 23, in
from iris.common import (
File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/iris/common/init.py", line 9, in
from .mixin import *
File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/iris/common/mixin.py", line 10, in
import cf_units
File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/cf_units/init.py", line 198, in
with suppress_errors(), c_locale():
File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/contextlib.py", line 142, in exit
next(self.gen)
File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/cf_units/init.py", line 191, in c_locale
locale.setlocale(locale.LC_NUMERIC, lc_numeric)
File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/locale.py", line 620, in setlocale
return _setlocale(category, locale)
locale.Error: unsupported locale setting

[bouweandela]

To understand what is going on, it would be useful to print the value that is causing the error. Do you think you could modify the build so it prints that out?

[ESadek-MO]

from @SciTools/peloton:

AFAIK: For locale, setting LC_ALL=C should set sensible defaults for all the local settings...(such as LC_NUMERIC)

[valeriupredoi]

@ESadek-MO I am actually doing that in here ESMValGroup/ESMValCore#2706 and that doesn't really help much; note that we are seeing flakiness in this fail, it's about 80% fail 20% pass

[valeriupredoi]

also note these rather relevant Sphinx issues: sphinx-doc/sphinx#13485 (very recent) and sphinx-doc/sphinx#11739 (probably marginally related)

[valeriupredoi]

woop: actually setting LC_ALL=C may actually do the trick, two test passes out of two so far
EDIT: that IS the solution, consistent pass rate, no fails; do you have an idea why the locale gets altered so that it needs restting to C? I thought the C locale was the default for Linux machines

[stephenworsley]

From @SciTools/peloton: We got this info from @ukmo-ccbunney who isn't available right now. Locale stuff is a bit of a mystery to us, but we're happy that this workaround "just works".

[valeriupredoi]

@stephenworsley it'd be rather useful uf we could fibd out if something in iris is changing the locale settings. My hunch is it's nothing in iris though, since a) it started happening (to us, at least) all of a sudden, and b) others are reporting similar issues to Sphinx.

[bjlittle]

it'd be rather useful uf we could fibd out if something in iris is changing the locale settings.

We know that it's not iris because we've observed this issue here in cf-units (and that's not dependent on iris)

Not sure how to progress or investigate this ...

[bouweandela]

If you could print out the value of lc_numeric in this code when the crash happens, that would be helpful:
https://github.com/SciTools/cf-units/blob/main/cf_units/__init__.py#L180-L191

Somehow locale.getlocale(locale.LC_NUMERIC) seems to return a locale that is invalid or not installed.

[valeriupredoi]

@bjlittle some info from ESMValGroup/ESMValCore#2706 (comment) - I tested with cf-units==3.3.0 and all is fine, unless one has iris>=3.11 and Python==3.13.3, there was that change in 3.13.3 that I quote in the comment; we don't see any issues with Python<3.13.3; there may be multiple problems playing up at the same time, but whatever it hits us in the docs build is not happening without iris>=3.11 and Python==3.13.3 🍺

[valeriupredoi]

also, as a kluge we found out that by resetting the locale in our sphinx conf.py (see https://github.com/ESMValGroup/ESMValCore/pull/2708/files) with:

import locale
locale.setlocale(locale.LC_ALL, 'C')

that fixes the problem, but the cause is yet to be identified