Skip to content

Creating dataframe from Python dict with array as value ignores/loses values when starting with None #22747

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
2 tasks done
aostermeier-100 opened this issue May 14, 2025 · 2 comments
Labels
bug Something isn't working needs triage Awaiting prioritization by a maintainer python Related to Python Polars

Comments

@aostermeier-100
Copy link

Checks

  • I have checked that this issue has not already been reported.
  • I have confirmed this bug exists on the latest version of Polars.

Reproducible example

    data = [
        {
            "measurement_location": "um03",
            "date": datetime.datetime(2025, 2, 27, tzinfo=TZ).date(),
            "consumption": [None] * 95 + [327.0],
        },
        {
            "measurement_location": "um04",
            "date": datetime.datetime(2025, 2, 28, tzinfo=TZ).date(),
            "consumption": [327.0] + [None] * 95,
        },
    ]

      return pl.DataFrame(data)

Log output

Issue description

When creating test data like above I have noticed that the elements in data are not converted into the dataframe as expected, for cases where they array does start with None. As soon as I have one leading value, which is not None the whole initialization is working properly.

um03 ┆ 2025-02-27 ┆ [null, null, … null]
um04 ┆ 2025-02-28 ┆ [327.0, null, … null]

Expected behavior

Independent of what data I am converting into a dataframe with polars I would expect that the values are not being dropped or getting lost in this conversion.

um03 ┆ 2025-02-27 ┆ [null, null, … 327.0]
um04 ┆ 2025-02-28 ┆ [327.0, null, … null]

Installed versions

Polars:              1.29.0
Index type:          UInt32
Platform:            Linux-5.15.167.4-microsoft-standard-WSL2-x86_64-with-glibc2.36
Python:              3.12.9 (main, Mar 18 2025, 03:28:39) [GCC 12.2.0]
LTS CPU:             False
@aostermeier-100 aostermeier-100 added bug Something isn't working python Related to Python Polars needs triage Awaiting prioritization by a maintainer labels May 14, 2025
@aostermeier-100
Copy link
Author

we have conducted some more tests and it seems that 25 might be the magic number here - as you can see the on the screenshot the first array contains the value and the second one does not.

arr_broken = [{"key1": "value_1", "datetime": datetime.datetime.now(), "list": [None]24 + 72 [1.0]}, {"key1": "value_2", "datetime": datetime.datetime.now(), "list": [None]25 + 71[2.0]}

Image

@cmdlineluser
Copy link
Contributor

#22302 (comment) may serve as an alternative approach.

shape: (2, 3)
┌──────────────────────┬────────────┬───────────────────────┐
│ measurement_location ┆ date       ┆ consumption           │
│ ---                  ┆ ---        ┆ ---                   │
│ str                  ┆ str        ┆ list[f64]             │
╞══════════════════════╪════════════╪═══════════════════════╡
│ um03                 ┆ 2025-02-27 ┆ [null, null, … 327.0] │
│ um04                 ┆ 2025-02-28 ┆ [327.0, null, … null] │
└──────────────────────┴────────────┴───────────────────────┘

@nameexhaustion nameexhaustion changed the title Creating dataframe from JSON data with array as value ignores/loses values when starting with None Creating dataframe from Python dict with array as value ignores/loses values when starting with None May 14, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working needs triage Awaiting prioritization by a maintainer python Related to Python Polars
Projects
None yet
Development

No branches or pull requests

2 participants