Skip to content

Panic when combining .list.explode() with .over(mapping_strategy="join") #22770

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
2 tasks done
atombear opened this issue May 16, 2025 · 1 comment
Open
2 tasks done
Labels
A-panic Area: code that results in panic exceptions accepted Ready for implementation bug Something isn't working P-medium Priority: medium python Related to Python Polars

Comments

@atombear
Copy link

Checks

  • I have checked that this issue has not already been reported.
  • I have confirmed this bug exists on the latest version of Polars.

Reproducible example

on polars 1.29.0

df = pl.DataFrame({"x": [[1.0], [2.0]]}).with_row_index("idx")
e = pl.col("x").list.explode().over("idx", mapping_strategy="join")
df.with_columns(e).equals(df)  # True
df.select(pl.col("x").list.diff()) # no problem
df.select(e.list.diff())  # panic

Log output

group_by keys are sorted; running sorted key fast path

thread '<unnamed>' panicked at crates/polars-plan/src/dsl/function_expr/list.rs:94:55:
called `Option::unwrap()` on a `None` value
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
---------------------------------------------------------------------------
PanicException                            Traceback (most recent call last)
Cell In[2], line 5
      3 df.with_columns(e).equals(df)  # True
      4 df.select(pl.col("x").list.diff()) # no problem
----> 5 df.select(e.list.diff())

File ~/miniforge3/envs/impressionism-env/lib/python3.10/site-packages/polars/dataframe/frame.py:9657, in DataFrame.select(self, *exprs, **named_exprs)
   9557 def select(
   9558     self, *exprs: IntoExpr | Iterable[IntoExpr], **named_exprs: IntoExpr
   9559 ) -> DataFrame:
   9560     """
   9561     Select columns from this DataFrame.
   9562 
   (...)
   9655     └──────────────┘
   9656     """
-> 9657     return self.lazy().select(*exprs, **named_exprs).collect(_eager=True)

File ~/miniforge3/envs/impressionism-env/lib/python3.10/site-packages/polars/_utils/deprecation.py:93, in deprecate_streaming_parameter.<locals>.decorate.<locals>.wrapper(*args, **kwargs)
     89         kwargs["engine"] = "in-memory"
     91     del kwargs["streaming"]
---> 93 return function(*args, **kwargs)

File ~/miniforge3/envs/impressionism-env/lib/python3.10/site-packages/polars/lazyframe/frame.py:2224, in LazyFrame.collect(self, type_coercion, _type_check, predicate_pushdown, projection_pushdown, simplify_expression, slice_pushdown, comm_subplan_elim, comm_subexpr_elim, cluster_with_columns, collapse_joins, no_optimization, engine, background, _check_order, _eager, **_kwargs)
   2222 # Only for testing purposes
   2223 callback = _kwargs.get("post_opt_callback", callback)
-> 2224 return wrap_df(ldf.collect(engine, callback))

PanicException: called `Option::unwrap()` on a `None` value

Issue description

this worked in 1.20

Expected behavior

pl.col("x") and pl.col("x").list.explode().over("idx", mapping_strategy="join") should be equivalent in this case.

Installed versions

--------Version info---------
Polars:              1.29.0
Index type:          UInt32
Platform:            macOS-15.4.1-x86_64-i386-64bit
Python:              3.10.5 | packaged by conda-forge | (main, Jun 14 2022, 07:03:09) [Clang 13.0.1 ]
LTS CPU:             False

i see this on linux as well

@atombear atombear added bug Something isn't working python Related to Python Polars needs triage Awaiting prioritization by a maintainer labels May 16, 2025
@coastalwhite coastalwhite self-assigned this May 16, 2025
@coastalwhite
Copy link
Collaborator

Doing some initial investigation, this seems to be a regression from the recent fixes to .explode() + .over(). It is a mismatch between the type check result from .over and what is actually done in the engines. I need to have it a bit more clear what the signature of .over should be in this case.

@coastalwhite coastalwhite removed their assignment May 16, 2025
@coastalwhite coastalwhite changed the title panic with list api Panic when combining .list.explode() with .over(mapping_strategy="join") May 16, 2025
@coastalwhite coastalwhite added accepted Ready for implementation P-medium Priority: medium A-panic Area: code that results in panic exceptions and removed needs triage Awaiting prioritization by a maintainer labels May 16, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-panic Area: code that results in panic exceptions accepted Ready for implementation bug Something isn't working P-medium Priority: medium python Related to Python Polars
Projects
None yet
Development

No branches or pull requests

2 participants