Further problems with locale #519

pp-mo · 2024-11-12T14:57:40Z

As reported here in the context of iris-esmf-regrid,
the current save - modify - restore code for the locale setting it seems does not always work correctly.
This was reported in #435, fixed by #436, introduced in v3.3.0

Not clear yet what is actually wrong, but for iris-esmf-regrid it seems to occur specifically in readthedocs builds.
The case given should hopefully point to it.

Detail

ReadTheDocs log captured here from a ReadTheDocs build test (i.e. its error log) on commit SciTools/iris-esmf-regrid@b5edcd7 , in SciTools/iris-esmf-regrid#431.
Since we can't expect that record to persist


python -m sphinx -T -b html -d _build/doctrees -D language=en . $READTHEDOCS_OUTPUT/html
Running Sphinx v8.1.3
Traceback (most recent call last):

File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/esmf_regrid-0.12.dev0-py3.10.egg/esmf_regrid/init.py", line 11, in 

import iris.mesh as _imesh

ModuleNotFoundError: No module named 'iris.mesh'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):

File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/sphinx/config.py", line 529, in eval_config_file

exec(code, namespace)  # NoQA: S102

File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/checkouts/431/docs/src/conf.py", line 34, in 

from esmf_regrid import version as esmf_r_version

File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/esmf_regrid-0.12.dev0-py3.10.egg/esmf_regrid/init.py", line 14, in 

import iris.experimental.ugrid as _imesh

File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/iris/experimental/ugrid/init.py", line 18, in 

from .load import PARSE_UGRID_ON_LOAD, load_mesh, load_meshes

File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/iris/experimental/ugrid/load.py", line 22, in 

from ...coords import AuxCoord

File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/iris/coords.py", line 23, in 

from iris.common import (

File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/iris/common/init.py", line 9, in 

from .mixin import *

File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/iris/common/mixin.py", line 10, in 

import cf_units

File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/cf_units/init.py", line 198, in 

with suppress_errors(), c_locale():

File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/contextlib.py", line 142, in exit

next(self.gen)

File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/cf_units/init.py", line 191, in c_locale

locale.setlocale(locale.LC_NUMERIC, lc_numeric)

File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/locale.py", line 620, in setlocale

return _setlocale(category, locale)

locale.Error: unsupported locale setting
The above exception was the direct cause of the following exception:
Traceback (most recent call last):

File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/sphinx/cmd/build.py", line 496, in build_main

app = Sphinx(

File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/sphinx/application.py", line 238, in init

self.config = Config.read(self.confdir, confoverrides or {}, self.tags)

File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/sphinx/config.py", line 316, in read

namespace = eval_config_file(filename, tags)

File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/sphinx/config.py", line 542, in eval_config_file

raise ConfigError(msg % traceback.format_exc()) from exc

sphinx.errors.ConfigError: There is a programmable error in your configuration file:
Traceback (most recent call last):

File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/esmf_regrid-0.12.dev0-py3.10.egg/esmf_regrid/init.py", line 11, in 

import iris.mesh as _imesh

ModuleNotFoundError: No module named 'iris.mesh'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):

File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/sphinx/config.py", line 529, in eval_config_file

exec(code, namespace)  # NoQA: S102

File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/checkouts/431/docs/src/conf.py", line 34, in 

from esmf_regrid import version as esmf_r_version

File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/esmf_regrid-0.12.dev0-py3.10.egg/esmf_regrid/init.py", line 14, in 

import iris.experimental.ugrid as _imesh

File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/iris/experimental/ugrid/init.py", line 18, in 

from .load import PARSE_UGRID_ON_LOAD, load_mesh, load_meshes

File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/iris/experimental/ugrid/load.py", line 22, in 

from ...coords import AuxCoord

File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/iris/coords.py", line 23, in 

from iris.common import (

File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/iris/common/init.py", line 9, in 

from .mixin import *

File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/iris/common/mixin.py", line 10, in 

import cf_units

File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/cf_units/init.py", line 198, in 

with suppress_errors(), c_locale():

File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/contextlib.py", line 142, in exit

next(self.gen)

File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/cf_units/init.py", line 191, in c_locale

locale.setlocale(locale.LC_NUMERIC, lc_numeric)

File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/locale.py", line 620, in setlocale

return _setlocale(category, locale)

locale.Error: unsupported locale setting
Configuration error:

There is a programmable error in your configuration file:
Traceback (most recent call last):

File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/esmf_regrid-0.12.dev0-py3.10.egg/esmf_regrid/init.py", line 11, in 

import iris.mesh as _imesh

ModuleNotFoundError: No module named 'iris.mesh'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):

File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/sphinx/config.py", line 529, in eval_config_file

exec(code, namespace)  # NoQA: S102

File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/checkouts/431/docs/src/conf.py", line 34, in 

from esmf_regrid import version as esmf_r_version

File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/esmf_regrid-0.12.dev0-py3.10.egg/esmf_regrid/init.py", line 14, in 

import iris.experimental.ugrid as _imesh

File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/iris/experimental/ugrid/init.py", line 18, in 

from .load import PARSE_UGRID_ON_LOAD, load_mesh, load_meshes

File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/iris/experimental/ugrid/load.py", line 22, in 

from ...coords import AuxCoord

File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/iris/coords.py", line 23, in 

from iris.common import (

File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/iris/common/init.py", line 9, in 

from .mixin import *

File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/iris/common/mixin.py", line 10, in 

import cf_units

File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/cf_units/init.py", line 198, in 

with suppress_errors(), c_locale():

File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/contextlib.py", line 142, in exit

next(self.gen)

File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/site-packages/cf_units/init.py", line 191, in c_locale

locale.setlocale(locale.LC_NUMERIC, lc_numeric)

File "/home/docs/checkouts/readthedocs.org/user_builds/iris-esmf-regrid/conda/431/lib/python3.10/locale.py", line 620, in setlocale

return _setlocale(category, locale)

locale.Error: unsupported locale setting

The text was updated successfully, but these errors were encountered:

bouweandela · 2024-12-20T15:35:54Z

To understand what is going on, it would be useful to print the value that is causing the error. Do you think you could modify the build so it prints that out?

ESadek-MO · 2025-03-19T11:00:51Z

from @SciTools/peloton:

AFAIK: For locale, setting LC_ALL=C should set sensible defaults for all the local settings...(such as LC_NUMERIC)

valeriupredoi · 2025-04-15T16:02:05Z

@ESadek-MO I am actually doing that in here ESMValGroup/ESMValCore#2706 and that doesn't really help much; note that we are seeing flakiness in this fail, it's about 80% fail 20% pass

valeriupredoi · 2025-04-15T16:11:57Z

also note these rather relevant Sphinx issues: sphinx-doc/sphinx#13485 (very recent) and sphinx-doc/sphinx#11739 (probably marginally related)

valeriupredoi · 2025-04-15T16:13:08Z

woop: actually setting LC_ALL=C may actually do the trick, two test passes out of two so far
EDIT: that IS the solution, consistent pass rate, no fails; do you have an idea why the locale gets altered so that it needs restting to C? I thought the C locale was the default for Linux machines

stephenworsley · 2025-04-16T09:22:11Z

From @SciTools/peloton: We got this info from @ukmo-ccbunney who isn't available right now. Locale stuff is a bit of a mystery to us, but we're happy that this workaround "just works".

valeriupredoi · 2025-04-16T09:40:56Z

@stephenworsley it'd be rather useful uf we could fibd out if something in iris is changing the locale settings. My hunch is it's nothing in iris though, since a) it started happening (to us, at least) all of a sudden, and b) others are reporting similar issues to Sphinx.

bjlittle · 2025-04-23T09:50:11Z

it'd be rather useful uf we could fibd out if something in iris is changing the locale settings.

We know that it's not iris because we've observed this issue here in cf-units (and that's not dependent on iris)

Not sure how to progress or investigate this ...

bouweandela · 2025-04-23T11:33:11Z

If you could print out the value of lc_numeric in this code when the crash happens, that would be helpful:
https://github.com/SciTools/cf-units/blob/main/cf_units/__init__.py#L180-L191

Somehow locale.getlocale(locale.LC_NUMERIC) seems to return a locale that is invalid or not installed.

valeriupredoi · 2025-05-08T15:10:30Z

@bjlittle some info from ESMValGroup/ESMValCore#2706 (comment) - I tested with cf-units==3.3.0 and all is fine, unless one has iris>=3.11 and Python==3.13.3, there was that change in 3.13.3 that I quote in the comment; we don't see any issues with Python<3.13.3; there may be multiple problems playing up at the same time, but whatever it hits us in the docs build is not happening without iris>=3.11 and Python==3.13.3 🍺

valeriupredoi · 2025-05-08T15:18:25Z

also, as a kluge we found out that by resetting the locale in our sphinx conf.py (see https://github.com/ESMValGroup/ESMValCore/pull/2708/files) with:

import locale
locale.setlocale(locale.LC_ALL, 'C')

that fixes the problem, but the cause is yet to be identified

bouweandela · 2025-05-13T12:31:34Z

Some further debugging results of the recent issue: restoring the locale to the previous setting works fine with Python 3.13.2 but does not work anymore with Python 3.13.3. Possibly this is caused by the changes in python/cpython#129647.

Python 3.13.3 example:

$ docker run -it python:3.13.3 
Python 3.13.3 (main, May  9 2025, 23:49:05) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os, locale
>>> os.environ
environ({'PATH': '/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'HOSTNAME': '2e43b79a7135', 'TERM': 'xterm', 'GPG_KEY': '7169605F62C751356D054A26A821E680E5FA6305', 'PYTHON_VERSION': '3.13.3', 'PYTHON_SHA256': '40f868bcbdeb8149a3149580bb9bfd407b3321cd48f0be631af955ac92c0e041', 'HOME': '/root', 'LC_CTYPE': 'C.UTF-8'})
>>> os.environ["LC_ALL"] = "C.UTF-8"
>>> locale.setlocale(locale.LC_ALL, "")
'C.UTF-8'
>>> locale.getlocale(locale.LC_NUMERIC)
('en_US', 'UTF-8')
>>> locale.setlocale(locale.LC_NUMERIC, locale.getlocale(locale.LC_NUMERIC))
Traceback (most recent call last):
  File "<python-input-5>", line 1, in <module>
    locale.setlocale(locale.LC_NUMERIC, locale.getlocale(locale.LC_NUMERIC))
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/locale.py", line 615, in setlocale
    return _setlocale(category, locale)
locale.Error: unsupported locale setting

Python 3.13.2 example:

$ docker run -it python:3.13.2 
Python 3.13.2 (main, Apr  8 2025, 04:27:11) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os, locale
>>> os.environ
environ({'PATH': '/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'HOSTNAME': '64800af765e4', 'TERM': 'xterm', 'GPG_KEY': '7169605F62C751356D054A26A821E680E5FA6305', 'PYTHON_VERSION': '3.13.2', 'PYTHON_SHA256': 'd984bcc57cd67caab26f7def42e523b1c015bbc5dc07836cf4f0b63fa159eb56', 'HOME': '/root', 'LC_CTYPE': 'C.UTF-8'})
>>> os.environ["LC_ALL"] = "C.UTF-8"
>>> locale.setlocale(locale.LC_ALL, "")
'C.UTF-8'
>>> locale.getlocale(locale.LC_NUMERIC)
('C', 'UTF-8')
>>> locale.setlocale(locale.LC_NUMERIC, locale.getlocale(locale.LC_NUMERIC))
'C.UTF-8'

bouweandela · 2025-05-13T12:43:20Z

Opened python/cpython#133967 to ask if this is a bug or intended behaviour.

pp-mo added Type: Bug New: Issue Highlight a new community raised "generic" issue labels Nov 12, 2024

trexfeathers added this to 🚴 Peloton Dec 4, 2024

bouweandela mentioned this issue Apr 11, 2025

Build on readthedocs failing ESMValGroup/ESMValCore#2705

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Further problems with locale #519

Further problems with locale #519

pp-mo commented Nov 12, 2024 •

edited

Loading

bouweandela commented Dec 20, 2024

Uh oh!

ESadek-MO commented Mar 19, 2025

Uh oh!

valeriupredoi commented Apr 15, 2025

Uh oh!

valeriupredoi commented Apr 15, 2025

Uh oh!

valeriupredoi commented Apr 15, 2025 •

edited

Loading

Uh oh!

stephenworsley commented Apr 16, 2025

Uh oh!

valeriupredoi commented Apr 16, 2025

Uh oh!

bjlittle commented Apr 23, 2025

Uh oh!

bouweandela commented Apr 23, 2025

Uh oh!

valeriupredoi commented May 8, 2025

Uh oh!

valeriupredoi commented May 8, 2025

Uh oh!

bouweandela commented May 13, 2025

Uh oh!

bouweandela commented May 13, 2025

Uh oh!

Further problems with locale #519

Further problems with locale #519

Comments

pp-mo commented Nov 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Detail

bouweandela commented Dec 20, 2024

Uh oh!

ESadek-MO commented Mar 19, 2025

Uh oh!

valeriupredoi commented Apr 15, 2025

Uh oh!

valeriupredoi commented Apr 15, 2025

Uh oh!

valeriupredoi commented Apr 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

stephenworsley commented Apr 16, 2025

Uh oh!

valeriupredoi commented Apr 16, 2025

Uh oh!

bjlittle commented Apr 23, 2025

Uh oh!

bouweandela commented Apr 23, 2025

Uh oh!

valeriupredoi commented May 8, 2025

Uh oh!

valeriupredoi commented May 8, 2025

Uh oh!

bouweandela commented May 13, 2025

Uh oh!

bouweandela commented May 13, 2025

Uh oh!

pp-mo commented Nov 12, 2024 •

edited

Loading

valeriupredoi commented Apr 15, 2025 •

edited

Loading