Add fill_null(exprs, strategy='inverse')

Description

I have always want functions to support skip nan, because in the financial field, null values appear very frequently.
Recently I finally find a way, which is to use fill_null to fill back.

How to implement?
before call function,first drop_nulls, then call function, finally create array and fill it with results from tail to head, if meet null, skip it.

Here's how to use it:

import polars as pl

df = pl.DataFrame(
    {
        "A": [5, None, 3, 2, 1],
        "B": [5, 3, None, 2, 1],
        "C": [None, None, None, None, None],
    }
)
print(df)
"""
┌──────┬──────┬──────┐
│ A    ┆ B    ┆ C    │
│ ---  ┆ ---  ┆ ---  │
│ i64  ┆ i64  ┆ f32  │
╞══════╪══════╪══════╡
│ 5    ┆ 5    ┆ null │
│ null ┆ 3    ┆ null │
│ 3    ┆ null ┆ null │
│ 2    ┆ 2    ┆ null │
│ 1    ┆ 1    ┆ null │
└──────┴──────┴──────┘
"""

df1 = df.select([
    pl.col("A", "B", "C").rolling_mean(2),
])
print(df1)
"""
┌──────┬──────┬──────┐
│ A    ┆ B    ┆ C    │
│ ---  ┆ ---  ┆ ---  │
│ f64  ┆ f64  ┆ f32  │
╞══════╪══════╪══════╡
│ null ┆ null ┆ null │
│ null ┆ 4.0  ┆ null │
│ null ┆ null ┆ null │
│ 2.5  ┆ null ┆ null │
│ 1.5  ┆ 1.5  ┆ null │
└──────┴──────┴──────┘
"""

# Implemented the function of ignoring nan, but the position and length need to be adjusted
df2 = df.select([
    pl.col("A", "B").drop_nulls().rolling_mean(2),
])
print(df2)
df2 = df.select([
    pl.col("C").drop_nulls().rolling_mean(2),
])
print(df2)
"""
┌──────┬──────┐
│ A    ┆ B    │
│ ---  ┆ ---  │
│ f64  ┆ f64  │
╞══════╪══════╡
│ null ┆ null │
│ 4.0  ┆ 4.0  │
│ 2.5  ┆ 2.5  │
│ 1.5  ┆ 1.5  │
└──────┴──────┘
shape: (0, 1)
┌─────┐
│ C   │
│ --- │
│ f32 │
╞═════╡
└─────┘
"""

#  This should be exactly the same as the original data
df3 = df.with_columns([
    pl.col("A", "B", "C").drop_nulls().fill_null(pl.col("A", "B", "C"), strategy='inverse'),
])
print(df3)
"""
┌──────┬──────┬──────┐
│ A    ┆ B    ┆ C    │
│ ---  ┆ ---  ┆ ---  │
│ i64  ┆ i64  ┆ f32  │
╞══════╪══════╪══════╡
│ 5    ┆ 5    ┆ null │
│ null ┆ 3    ┆ null │
│ 3    ┆ null ┆ null │
│ 2    ┆ 2    ┆ null │
│ 1    ┆ 1    ┆ null │
└──────┴──────┴──────┘
"""

# Implement skip nan
df4 = df.with_columns([
    pl.col("A", "B", "C").drop_nulls().rolling_mean(2).fill_null(pl.col("A", "B", "C"), strategy='inverse'),
])
print(df4)
"""
┌──────┬──────┬──────┐
│ A    ┆ B    ┆ C    │
│ ---  ┆ ---  ┆ ---  │
│ i64  ┆ i64  ┆ f32  │
╞══════╪══════╪══════╡
│ null ┆ null ┆ null │
│ null ┆ 4.0  ┆ null │
│ 4.0  ┆ null ┆ null │
│ 2.5  ┆ 2.5  ┆ null │
│ 1.5  ┆ 1.5  ┆ null │
└──────┴──────┴──────┘
"""

Maybe the follow code is better

pl.col("A", "B", "C").drop_nulls().rolling_mean(2).fill_null(pl.col("A", "B", "C").is_nan(), strategy='inverse')
pl.col("A", "B", "C").drop_nulls().rolling_mean(2).fill_null(pl.col("A", "B", "C").is_null(), strategy='inverse')

I don't understand what you're trying to do here.

From what I understand, you're trying to use fill_null to add null values. We will not support that. Though we may implement a different method to do what you're trying to do.

Please try to describe more clearly what you're trying to do and give a simple example.

@stinodego

Hi @stinodego

My purpose is to simplify handling of nan/null when calculating time series indicators.
The possible situations where null values may appear are:

Stock trading suspended,
Trading days are different. For example, S&P 500 is traded 5 days a week, and BTCUSDT is traded 7 days a week.

indicators such as rolling_* will have problems, and talib will also have problems. So I want to be able to skip null values. see
#9131

It is unrealistic to implement the skip function for every time series function, it will be simplified if drop nan first and then backfill.

from datetime import datetime

import numpy as np
import polars as pl
import polars.selectors as cs

prices = pl.DataFrame({'date': pl.datetime_range(start=datetime(2023, 1, 1), end=datetime(2023, 12, 13), eager=True)})
prices = prices.with_columns([
    # 7 days a week
    pl.col('date').dt.weekday().alias('BTCUSDT'),
    # 5 days a week
    pl.when(pl.col('date').dt.weekday() <= 5).then(pl.col('date').dt.weekday()).otherwise(np.nan).alias('S&P 500'),
])

out = prices.with_columns(cs.numeric().rolling_mean(6).name.prefix('rolling_mean_'))
print(out.tail())
"""
shape: (5, 5)
┌─────────────────────┬─────────┬─────────┬──────────────────────┬──────────────────────┐
│ date                ┆ BTCUSDT ┆ S&P 500 ┆ rolling_mean_BTCUSDT ┆ rolling_mean_S&P 500 │
│ ---                 ┆ ---     ┆ ---     ┆ ---                  ┆ ---                  │
│ datetime[μs]        ┆ u32     ┆ f64     ┆ f64                  ┆ f64                  │
╞═════════════════════╪═════════╪═════════╪══════════════════════╪══════════════════════╡
│ 2023-12-09 00:00:00 ┆ 6       ┆ NaN     ┆ 3.5                  ┆ NaN                  │
│ 2023-12-10 00:00:00 ┆ 7       ┆ NaN     ┆ 4.5                  ┆ NaN                  │
│ 2023-12-11 00:00:00 ┆ 1       ┆ 1.0     ┆ 4.333333             ┆ NaN                  │
│ 2023-12-12 00:00:00 ┆ 2       ┆ 2.0     ┆ 4.166667             ┆ NaN                  │
│ 2023-12-13 00:00:00 ┆ 3       ┆ 3.0     ┆ 4.0                  ┆ NaN                  │
└─────────────────────┴─────────┴─────────┴──────────────────────┴──────────────────────┘
"""

rolling_mean_S&P 500 is all nan/null and does not meet the requirements. Because the results I actually need to calculate are as follows.

out1 = prices.select(pl.col('S&P 500').drop_nans().rolling_mean(6).name.prefix('rolling_mean_'))
print(out1.tail())
"""
shape: (5, 1)
┌──────────────────────┐
│ rolling_mean_S&P 500 │
│ ---                  │
│ f64                  │
╞══════════════════════╡
│ 3.166667             │
│ 3.333333             │
│ 2.666667             │
│ 2.833333             │
│ 3.0                  │
└──────────────────────┘

I wrote a demo code for reference

import numpy as np

def resize(price, is_nan):
    if len(price) >= len(is_nan):
        return price

    out = np.full_like(is_nan, fill_value=np.nan, dtype=price.dtype)

    pos_i = len(price) - 1
    pos_j = len(out) - 1
    for i in range(pos_i, -1, -1):
        for j in range(pos_j, -1, -1):
            if is_nan[j]:
                pos_j -= 1
                continue
            else:
                out[pos_j] = price[i]
                pos_j -= 1
                break
    return out


def nan_or_none(x):
    return x != x or x is None


price0 = np.array([1, 2, 3, 4, 5, np.nan, np.nan, 1, 2, 3])
is_nan = np.isnan(price0)  # [False False False False False  True  True False False False]


# ------------------------
price1 = price0[~is_nan]  # [1. 2. 3. 4. 5. 1. 2. 3.]
out1 = resize(price1, is_nan)
print(out1)  # [1, 2, 3, 4, 5, nan, nan, 1, 2, 3]



# ------------------------
timeperiod = 3
# rolling_mean
price2 = np.convolve(price1, np.ones((timeperiod,)) / timeperiod, mode='valid')
print(price2)
"""
[2.         3.         4.         3.33333333 2.66666667 2.        ]
"""


out2 = resize(price2, is_nan)
print(out2)
"""
[       nan        nan 2.         3.         4.                nan
        nan 3.33333333 2.66666667 2.        ]
"""

@wukan1986

Hi @wukan1986

thanks for the request

As far as I can tell (please correct me if I'm wrong), you'd like a way to skip nulls in some rolling functions. I think we're confusing nan and null here, so let's stick to just skipping nulls (you can always fill nans with nulls if you want to skip both).

If so, would the following work for you?

In [26]: (
    ...:     df.with_row_count("idx")
    ...:     .rolling(pl.col("idx").cast(pl.Int64), period="2i")
    ...:     .agg(pl.col("A", "B", "C").drop_nulls().mean())
    ...: )
    ...:
Out[26]:
shape: (5, 4)
┌─────┬─────┬─────┬──────┐
│ idx ┆ A   ┆ B   ┆ C    │
│ --- ┆ --- ┆ --- ┆ ---  │
│ i64 ┆ f64 ┆ f64 ┆ f32  │
╞═════╪═════╪═════╪══════╡
│ 0   ┆ 5.0 ┆ 5.0 ┆ null │
│ 1   ┆ 5.0 ┆ 4.0 ┆ null │
│ 2   ┆ 3.0 ┆ 3.0 ┆ null │
│ 3   ┆ 2.5 ┆ 2.0 ┆ null │
│ 4   ┆ 1.5 ┆ 1.5 ┆ null │
└─────┴─────┴─────┴──────┘

If so, then for now I think this is probably OK, and doesn't warrant extra arguments?

@MarcoGorelli

Hi, @MarcoGorelli

Thank you for reply, I get a PanicException, it is caused by pl.col("C")

import polars as pl

df = pl.DataFrame(
    {
        "A": [5, None, 3, 2, 1],
        "B": [5, 3, None, 2, 1],
        "C": [None, None, None, None, None],
    }
)

df = (
    df.with_row_count("idx")
    .rolling(pl.col("idx").cast(pl.Int64), period="2i")
    .agg(pl.col("A", "B", "C").drop_nulls().mean())
)
print(df)

thread '<unnamed>' panicked at crates\polars-lazy\src\physical_plan\expressions\apply.rs:168:22:
called `Result::unwrap()` on an `Err` value: SchemaMismatch(ErrString("invalid series dtype: expected `List`, got `null`"))
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Traceback (most recent call last):
  File "t7.py", line 14, in <module>
    .agg(pl.col("A", "B", "C").drop_nulls().mean())
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Users\Kan\miniconda3\envs\py311_dagster\Lib\site-packages\polars\dataframe\group_by.py", line 866, in agg
    .collect(no_optimization=True)
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Users\Kan\miniconda3\envs\py311_dagster\Lib\site-packages\polars\lazyframe\frame.py", line 1706, in collect
    return wrap_df(ldf.collect())
                   ^^^^^^^^^^^^^
pyo3_runtime.PanicException: called `Result::unwrap()` on an `Err` value: SchemaMismatch(ErrString("invalid series dtype: expected `List`, got `null`"))

--------Version info---------
Polars:               0.20.0
Index type:           UInt32
Platform:             Windows-10-10.0.22631-SP0
Python:               3.11.6 | packaged by conda-forge | (main, Oct  3 2023, 10:29:11) [MSC v.1935 64 bit (AMD64)]

----Optional dependencies----
adbc_driver_manager:  <not installed>
cloudpickle:          <not installed>
connectorx:           <not installed>
deltalake:            <not installed>
fsspec:               2023.10.0
gevent:               <not installed>
matplotlib:           <not installed>
numpy:                1.26.1
openpyxl:             <not installed>
pandas:               2.1.3
pyarrow:              14.0.1
pydantic:             2.4.2
pyiceberg:            <not installed>
pyxlsb:               <not installed>
sqlalchemy:           2.0.23
xlsx2csv:             <not installed>
xlsxwriter:           <not installed>

@MarcoGorelli

Hi, @MarcoGorelli

There are two requirements

Skip null values when calculating
The result should keep the null value of the original position.

from datetime import datetime

import polars as pl

df = pl.DataFrame(
    {
        "date": pl.datetime_range(datetime(2022, 1, 1), datetime(2022, 1, 5), eager=True),
        "A": [5, None, 3, 2, 1],
        "B": [1, None, None, 1, None],
        # "C": [None, None, None, None, None],
    }
)

df = (
    df.with_row_count("idx")
    .rolling(pl.col("idx").cast(pl.Int64), period="2i")
    .agg(pl.col("date").last(),
         pl.col("B").drop_nulls().mean().alias('mean'),
         pl.col("B").drop_nulls().sum().alias('sum'), )
)
print(df)

"""
shape: (5, 4)
┌─────┬─────────────────────┬──────┬─────┐
│ idx ┆ date                ┆ mean ┆ sum │
│ --- ┆ ---                 ┆ ---  ┆ --- │
│ i64 ┆ datetime[μs]        ┆ f64  ┆ i64 │
╞═════╪═════════════════════╪══════╪═════╡
│ 0   ┆ 2022-01-01 00:00:00 ┆ 1.0  ┆ 1   │
│ 1   ┆ 2022-01-02 00:00:00 ┆ 1.0  ┆ 1   │
│ 2   ┆ 2022-01-03 00:00:00 ┆ null ┆ 0   │
│ 3   ┆ 2022-01-04 00:00:00 ┆ 1.0  ┆ 1   │
│ 4   ┆ 2022-01-05 00:00:00 ┆ 1.0  ┆ 1   │
└─────┴─────────────────────┴──────┴─────┘
"""

but my expectation is

"""
shape: (5, 4)
┌─────┬─────────────────────┬──────┬─────┐
│ idx ┆ date                ┆ mean ┆ sum │
│ --- ┆ ---                 ┆ ---  ┆ --- │
│ i64 ┆ datetime[μs]        ┆ f64  ┆ i64 │
╞═════╪═════════════════════╪══════╪═════╡
│ 0   ┆ 2022-01-01 00:00:00 ┆ null ┆ null│
│ 1   ┆ 2022-01-02 00:00:00 ┆ null ┆ null│
│ 2   ┆ 2022-01-03 00:00:00 ┆ null ┆ null│
│ 3   ┆ 2022-01-04 00:00:00 ┆ 1.0  ┆ 2   │
│ 4   ┆ 2022-01-05 00:00:00 ┆ null ┆ null│
└─────┴─────────────────────┴──────┴─────┘
"""

rolling(period="2i") is not enough

@wukan1986

Thanks @wukan1986 for explaining, that's a clear example

So I think you also have a third requirement:

there need to be at least 2 elements in the window, else return null
, right?

How about

df = pl.DataFrame(
    {
        "date": pl.datetime_range(
            datetime(2022, 1, 1), datetime(2022, 1, 5), eager=True
        ),
        "A": [5, None, 3, 2, 1],
        "B": [1, None, None, 1, None],
        "C": [None, None, None, None, None],
    },
    schema_overrides={"C": pl.Int64},
)


def agg_func(col, aggregation, min_periods):
    return (
        pl.when(
            col.last().is_null() | (col.len() < (min_periods)),
        )
        .then(pl.lit(None))
        .otherwise(getattr(col.drop_nulls(), aggregation)())
    )


result = (
    df.with_row_count("idx")
    .rolling(pl.col("idx").cast(pl.Int64), period="2i")
    .agg(
        pl.col("date").last(),
        agg_func(pl.col("B"), 'mean', 2).alias("mean"),
        agg_func(pl.col("B"), 'sum', 2).alias("sum"),
    )
)
print(result)

, I think this would give you your expected output?

Yes, it seems you can do this with a combination of min_periods and when(col.is_not_null()).then(rolling_expr).

import polars as pl

df = pl.DataFrame({"a": [1, 2, None, None, 5, 6]})

rolling = pl.col("a").rolling_mean(3, min_periods=0)
r = df.select(pl.when(pl.col("a").is_not_null()).then(rolling).otherwise(None))
print(r)

I don't think we need to add anything new here. Thanks for opening an issue though! Closing this now unless I see a good reason to re-open.

not sure that's quite to same - if I you try applying it to

df = pl.DataFrame({"a": [1, None, None, 1, None]})

you'll get

rolling = pl.col("a").rolling_mean(2, min_periods=0)
r = df.select(pl.when(pl.col("a").is_not_null()).then(rolling).otherwise(None))

shape: (5, 1)
┌──────┐
│ a    │
│ ---  │
│ f64  │
╞══════╡
│ 1.0  │
│ null │
│ null │
│ 1.0  │
│ null │
└──────┘

, whereas min_periods=2 would give you all-nulls

But the expect output is

shape: (5, 1)
┌──────┐
│ a    │
│ ---  │
│ f64  │
╞══════╡
│ null │
│ null │
│ null │
│ 1.0  │
│ null │
└──────┘

I think being able to express

rolling = pl.col("a").rolling_mean(2, min_periods=2, skip_nulls=True)
r = df.select(pl.when(pl.col("a").is_not_null()).then(rolling).otherwise(None))

would be quite nice

It is possible to work around skip_nulls in rolling aggregations, it's just not that easy

So, to summarise, I think this boils down to a feature request of skip_nulls in rolling_* functions

If True, then ignore the null values when aggregating
If False, then return null if a null value is present (just like happens now - so this would probably be a good default)

@wukan1986

Thanks @wukan1986 for explaining, that's a clear example

So I think you also have a third requirement:

there need to be at least 2 elements in the window, else return null
, right?

How about

df = pl.DataFrame(
    {
        "date": pl.datetime_range(
            datetime(2022, 1, 1), datetime(2022, 1, 5), eager=True
        ),
        "A": [5, None, 3, 2, 1],
        "B": [1, None, None, 1, None],
        "C": [None, None, None, None, None],
    },
    schema_overrides={"C": pl.Int64},
)


def agg_func(col, aggregation, min_periods):
    return (
        pl.when(
            col.last().is_null() | (col.len() < (min_periods)),
        )
        .then(pl.lit(None))
        .otherwise(getattr(col.drop_nulls(), aggregation)())
    )


result = (
    df.with_row_count("idx")
    .rolling(pl.col("idx").cast(pl.Int64), period="2i")
    .agg(
        pl.col("date").last(),
        agg_func(pl.col("B"), 'mean', 2).alias("mean"),
        agg_func(pl.col("B"), 'sum', 2).alias("sum"),
    )
)
print(result)

, I think this would give you your expected output?

shape: (5, 4)
┌─────┬─────────────────────┬──────┬──────┐
│ idx ┆ date                ┆ mean ┆ sum  │
│ --- ┆ ---                 ┆ ---  ┆ ---  │
│ i64 ┆ datetime[μs]        ┆ f64  ┆ i64  │
╞═════╪═════════════════════╪══════╪══════╡
│ 0   ┆ 2022-01-01 00:00:00 ┆ null ┆ null │
│ 1   ┆ 2022-01-02 00:00:00 ┆ null ┆ null │
│ 2   ┆ 2022-01-03 00:00:00 ┆ null ┆ null │
│ 3   ┆ 2022-01-04 00:00:00 ┆ 1.0  ┆ 1    │  # this should be 2
│ 4   ┆ 2022-01-05 00:00:00 ┆ null ┆ null │
└─────┴─────────────────────┴──────┴──────┘

the output is not my want.

┌─────┬─────────────────────┬──────┬──────┐
│ idx ┆ date                ┆ mean ┆ sum  │
│ --- ┆ ---                 ┆ ---  ┆ ---  │
│ i64 ┆ datetime[μs]        ┆ f64  ┆ i64  │
╞═════╪═════════════════════╪══════╪══════╡
│ 0   ┆ 2022-01-01 00:00:00 ┆ null ┆ null │
│ 1   ┆ 2022-01-02 00:00:00 ┆ null ┆ null │
│ 2   ┆ 2022-01-03 00:00:00 ┆ null ┆ null │
│ 3   ┆ 2022-01-04 00:00:00 ┆ 1.0  ┆ 2    │  # this should be 2
│ 4   ┆ 2022-01-05 00:00:00 ┆ null ┆ null │
└─────┴─────────────────────┴──────┴──────┘

So, to summarise, I think this boils down to a feature request of skip_nulls in rolling_* functions

If True, then ignore the null values when aggregating

If False, then return null if a null value is present (just like happens now - so this would probably be a good default)

rolling_* and map_batches add skip_nulls is a lot of work.
so i suggest to add a function fill_null or resize

see this #12925 (comment) function resize

Right, so just checking I've understood, what you want is:

if a given row is null, output null
if a given row is not null, then:
- if there are n-1 previous non-null values, then aggregate them
- if there are fewer than n-1 previous non-null values, then output null
  ?

Right, so just checking I've understood, what you want is:

if a given row is null, output null

if a given row is not null, then:

if there are n-1 previous non-null values, then aggregate them

if there are fewer than n-1 previous non-null values, then output null
?

I think you are overcomplicating the issue
In fact, it is just a function that moves the elements of a short array into a long array as required.
The aggregation operation is done elsewhere, so that all rolling functions do not need to be modified.

for an example

not_nan    = [True, True, True,   True, False]
short_arr  = [ null,    1,    2,     3]
new_arr    = [ null,    1,    2,     3, null]

The pseudo code is as follows

pl.col('A').drop_nulls().rolling_xxx().resize(pl.col('A').is_null())
pl.col('A').drop_nulls().resize(pl.col('A').is_null())

We do not consider aggregation, aggregation is just one usage scenario

pl.col('A').drop_nulls().rolling_mean(skip_null=True).drop_nulls().rolling_mean(skip_null=True)  # not good
pl.col('A').drop_nulls().rolling_mean().rolling_mean().resize(pl.col('A)) # better

I think you just need scatter then?

not_nan = pl.Series([True, True, True,   True, False])
short_arr = pl.Series([ None,    1,    2,     3])
result = pl.Series([None]*len(not_nan), dtype=short_arr.dtype)
result.scatter(not_nan.arg_true(), short_arr)

This gives

shape: (5,)
Series: '' [i64]
[
        null
        1
        2
        3
        null
]

I think you just need scatter then?

not_nan = pl.Series([True, True, True,   True, False])
short_arr = pl.Series([ None,    1,    2,     3])
result = pl.Series([None]*len(not_nan), dtype=short_arr.dtype)
result.scatter(not_nan.arg_true(), short_arr)

This gives

shape: (5,)
Series: '' [i64]
[
        null
        1
        2
        3
        null
]

That's great. This is what I want. It would be better if it could support Expr.

pl.col('A').drop_nulls().scatter(pl.col('A').arg_not_null())

agree - fancy opening a new issue in which you just request Expr.scatter?

Not sure if it's of any use, but this is how it looks to me using the current API:

def rolling_no_null(expr, period):             
    return (
        expr.cum_count().filter(expr.is_not_null()).alias("row_nr"),
        expr.drop_nulls().rolling_mean(period).name.suffix(".mean"),
        expr.drop_nulls().rolling_sum(period).name.suffix(".sum")
    )
    
lazy = df.with_row_count().lazy()

frames = [lazy]
frames.extend(
    lazy.select(rolling_no_null(pl.col(col), 2)) 
    for col in ["A", "B", "C"]
)

print(
    pl.concat(frames, how="align").collect()
)

# shape: (5, 11)
# ┌────────┬─────────────────────┬──────┬──────┬──────┬────────┬───────┬────────┬───────┬────────┬───────┐
# │ row_nr ┆ date                ┆ A    ┆ B    ┆ C    ┆ A.mean ┆ A.sum ┆ B.mean ┆ B.sum ┆ C.mean ┆ C.sum │
# │ ---    ┆ ---                 ┆ ---  ┆ ---  ┆ ---  ┆ ---    ┆ ---   ┆ ---    ┆ ---   ┆ ---    ┆ ---   │
# │ u32    ┆ datetime[μs]        ┆ i64  ┆ i64  ┆ i64  ┆ f64    ┆ i64   ┆ f64    ┆ i64   ┆ f64    ┆ i64   │
# ╞════════╪═════════════════════╪══════╪══════╪══════╪════════╪═══════╪════════╪═══════╪════════╪═══════╡
# │ 0      ┆ 2022-01-01 00:00:00 ┆ 5    ┆ 1    ┆ null ┆ null   ┆ null  ┆ null   ┆ null  ┆ null   ┆ null  │
# │ 1      ┆ 2022-01-02 00:00:00 ┆ null ┆ null ┆ null ┆ null   ┆ null  ┆ null   ┆ null  ┆ null   ┆ null  │
# │ 2      ┆ 2022-01-03 00:00:00 ┆ 3    ┆ null ┆ null ┆ 4.0    ┆ 8     ┆ null   ┆ null  ┆ null   ┆ null  │
# │ 3      ┆ 2022-01-04 00:00:00 ┆ 2    ┆ 1    ┆ null ┆ 2.5    ┆ 5     ┆ 1.0    ┆ 2     ┆ null   ┆ null  │
# │ 4      ┆ 2022-01-05 00:00:00 ┆ 1    ┆ null ┆ null ┆ 1.5    ┆ 3     ┆ null   ┆ null  ┆ null   ┆ null  │
# └────────┴─────────────────────┴──────┴──────┴──────┴────────┴───────┴────────┴───────┴────────┴───────┘

@cmdlineluser

Hi, @cmdlineluser

Your expr.cum_count().filter(expr.is_not_null()) is very creative

You can use .over() to do this:

import polars as pl

df = pl.DataFrame(
    {
        "A": [5, None, 3, 2, 1],
        "B": [5, 3, None, 2, 1],
        "C": [None, None, None, None, None],
    }
)

df_res = df.with_columns(
    pl.col("A", "B", "C")
      .rolling_mean(2)
      .over(pl.col("A", "B", "C").is_null())
      .name.suffix('.mean')
)
print(df_res)

shape: (5, 6)
┌──────┬──────┬──────┬────────┬────────┬────────┐
│ A    ┆ B    ┆ C    ┆ A.mean ┆ B.mean ┆ C.mean │
│ ---  ┆ ---  ┆ ---  ┆ ---    ┆ ---    ┆ ---    │
│ i64  ┆ i64  ┆ null ┆ f64    ┆ f64    ┆ f64    │
╞══════╪══════╪══════╪════════╪════════╪════════╡
│ 5    ┆ 5    ┆ null ┆ null   ┆ null   ┆ null   │
│ null ┆ 3    ┆ null ┆ null   ┆ 4.0    ┆ null   │
│ 3    ┆ null ┆ null ┆ 4.0    ┆ null   ┆ null   │
│ 2    ┆ 2    ┆ null ┆ 2.5    ┆ 2.5    ┆ null   │
│ 1    ┆ 1    ┆ null ┆ 1.5    ┆ 1.5    ┆ null   │
└──────┴──────┴──────┴────────┴────────┴────────┘

@ruoyu0088

@ruoyu0088 Thank you very much, your idea is very good

Navigation Menu

Repository navigation

Add `fill_null(exprs, strategy='inverse')` #12925

Description

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Add fill_null(exprs, strategy='inverse') #12925

Description

Description

Activity

wukan1986 commented on Dec 8, 2023

stinodego commented on Dec 15, 2023

wukan1986 commented on Dec 16, 2023

MarcoGorelli commented on Dec 16, 2023

wukan1986 commented on Dec 17, 2023

wukan1986 commented on Dec 17, 2023

MarcoGorelli commented on Dec 17, 2023

stinodego commented on Dec 17, 2023

MarcoGorelli commented on Dec 17, 2023

MarcoGorelli commented on Dec 17, 2023

wukan1986 commented on Dec 17, 2023

wukan1986 commented on Dec 17, 2023

MarcoGorelli commented on Dec 17, 2023

wukan1986 commented on Dec 17, 2023

MarcoGorelli commented on Dec 17, 2023

wukan1986 commented on Dec 17, 2023

MarcoGorelli commented on Dec 17, 2023

cmdlineluser commented on Dec 17, 2023

wukan1986 commented on Dec 18, 2023

ruoyu0088 commented on Dec 19, 2024

wukan1986 commented on Dec 19, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Add `fill_null(exprs, strategy='inverse')` #12925