Skip to content

Filter and combine result columns#1278

Draft
gnn wants to merge 23 commits into
devfrom
features/filter-and-combine-result-columns
Draft

Filter and combine result columns#1278
gnn wants to merge 23 commits into
devfrom
features/filter-and-combine-result-columns

Conversation

@gnn

@gnn gnn commented Mar 26, 2026

Copy link
Copy Markdown
Member

Implement a proposal for an API that allows filtering the columns of DataFrames in oemof's Results objects. Having a stable API would also allow predefining filters enabled by default or filters which are expected to be used often.

Also implement a proposal for an API that allows specifying callback functions that manipulate the DataFrames in different ways than just filtering.

gnn and others added 8 commits August 25, 2025 03:31
When calling `results.to_df(variable)`, look for a `filters` attribute
on `results`, then look up `results.filters[variable]`, which should be
a function, and use that function to filter the `columns` of the
`variable`'s results `DataFrame` before returning it. More specifically,
the function should accept entries of the `DataFrame`'s `columns`
attribute and return something that evaluates to `True` if the column
should be included in the results and to `False` if the column should be
excluded.
Retain all columns, if `results.filters` does not contain the key
`variable` and provide a `filters` class attribute on `Results` in order
to provide default filters.
Add a `filters` argument to the `Results` constructor to allow setting
the `filters` attribute on construction.
Don't overwrite the `variables` argument to `Results.to_df` within the
method body with another computed value.
Instead of using a `filters` attribute to filter result `DataFrame`
columns, use an additional `filters` argument to `to_df` to achieve the
same result. Replace `functools.cache` on `to_df` with only caching
unfiltered results manually because `functools.cache` would interact
badly with the new `filters` argument.
This new implementation allows easy usage of different custom filters
without instantiating multiple `Results` objects and without constantly
recomputing the results `DataFrame`.
Note that default filtering is still done via a `filters` attribute.
Create as special `dict` subclass for `Filters` for `Results`. The
subclass as a special `updating` method, which is functionally
equivalent to `dict`'s `update` method, except that it doesn't update in
place, but creates a shallow clone and returns the `update`d clone.
The idea is that this enables combining the default `Results.filters`
easily with custom ones by doing e.g.:

```python
def nosinks(column):
    return not isinstance(column[1], Sink)

results.filters = Results.filters.updating({"flow": nosinks})
flows = results["flow"]
```

Note that the code making `Results.filters` a `Filters` instance instead
of a regular `dict` is not properly formatted yet. This is intentional,
in order to make it easier to spot that the code didn't change
semantically beyond adding the `Filters` call. Formatting will be done
in the next commit.
No semantic changes. Just formatting.
Not applying any filtering at all as a default is a special feature of
the `Filters` class, so the implementation should be placed there,
instead of making it the call site's responsibility to supply that
default.
)


def main(optimize=True):

Check notice

Code scanning / CodeQL

Explicit returns mixed with implicit (fall through) returns Note

Mixing implicit and explicit returns may indicate an error, as implicit returns always return None.
Comment thread examples/subnetwork/optimization_and_results_display.py Fixed

# -------------- Investment Costs ---------------------------

invest = results.get("invest")

Check notice

Code scanning / CodeQL

Unused local variable Note

Variable invest is not used.
gnn and others added 7 commits April 30, 2026 03:05
And manually shorten some overly long lines, too.
Quench a `flake8` complaint.
Helps when comparing them between script runs.
The `Filters` class is intended to be a special dictionary holding
filter functions as values, so it's `__getitem__` function should return
such functions and, consequently, have an appropriate return type.
I added a function "hide_and_rename" to the examples to show what I
need. For me it is already fine to have this function but we should
think about a way to make the functionality of this function available
for others in a simple way (callbacks, filters etc.)
@uvchik

uvchik commented Apr 30, 2026

Copy link
Copy Markdown
Member

I added a function "hide_and_rename" to the examples to show what I need. For me it is already fine to have this function but we should think about a way to make the functionality of this function available for others in a simple way (callbacks, filters etc.)

)
return filtered

res_flow = results["flow"]

Check notice

Code scanning / CodeQL

Unused local variable Note

Variable res_flow is not used.
gnn and others added 8 commits May 21, 2026 02:42
Since the function is analogous to Python's builtin `sorted`, this
naming is more consistent.
Using `DataFrame.loc` instead of `[]` squelches the
`SettingWithCopyWarning`s.
It's IMHO also a lot more readable than making the reader figure out
which of the many behaviours of `DataFrame`'s or `Series`'s overloaded
`[]` implementation actually gets used.
It's consistent with `Callbacks.noop` and uses context to avoid
redundant information. It should be clear that a `noop` in the context
of `Filters` means using no filter, so there's no need to repeat the
context in the name of the function.
Cast `Results.get`'s `filters` and `callbacks` arguments to `Filters`
and `Callbacks` before using them, thus allowing normal dictionaries to
be used when supplying values for these arguments. Among other things,
this allows the usage of empty `dict`s as an easy way to disable
defaults.
At least in one example. Leave the other one be for now, to illustrate
the difference.
This method could be defined (just `pass` inside the method) also with
the Node class of oemof.network
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants