[ENH] Is there any interest in adding an error handling option within EnsembleForecaster?
#5729
Replies: 2 comments 5 replies
-
|
Very nice idea! I see two options that could be of use:
I would suggest though to start with the first option, it is less general and programmatic, but the 80 in the 80/20 here imo. |
Beta Was this translation helpful? Give feedback.
-
|
I agree that option 1 provides a quick and straightforward implementation, focusing on immediate needs. However, option 2 offers a more comprehensive and flexible solution, potentially benefiting other modules as well, but it's a bigger lift with more requirements and testing needs. I really like the idea of a forecaster = FallbackForecaster(
forecasters=[
# First try: EnsembleForecaster with seasonality-sensitive components
EnsembleForecaster([
("ets", AutoETS(auto=True, n_jobs=1, additive_only=True, sp=sp)),
("arima", AutoARIMA(sp=sp)) # Seasonality often causes exceptions
]),
# Second try: EnsembleForecaster without explicit seasonality
EnsembleForecaster([
("ets", AutoETS(auto=True, n_jobs=1, additive_only=True, sp=None)),
("sarima", AutoARIMA(sp=None)),
]),
# Final fallback: Simple ensemble with a naive forecaster
EnsembleForecaster([
("ets", AutoETS(auto=True, n_jobs=1, additive_only=True, sp=None)),
("naive", NaiveForecaster(strategy="last")),
]),
]
)There could be additional parameters for logging, performance tracking, customizable fallback logic, etc. |
Beta Was this translation helpful? Give feedback.
-
|
yes, exactly! There's some thinking for the parameters to do, e.g., when we consider an estimator as "failed" - a failure could occur in |
Beta Was this translation helpful? Give feedback.
-
|
I've got some starter code EDIT: On second thought, I can just write a dummy forecaster that fails intentionally at the respective steps. |
Beta Was this translation helpful? Give feedback.
-
|
Another side note, @ninedigits, if you are already digging into This is a completely separate thinng though. Relevant issue and recipe is here: #5380 |
Beta Was this translation helpful? Give feedback.
-
|
Do you mean replace this parallel class: from joblib import Parallel, delayed
...
self.forecasters_ = Parallel(n_jobs=self.n_jobs)(
delayed(_fit_forecaster)(forecaster.clone(), y, X, fh)
for forecaster in forecasters
)with this? from sktime.utils.parallel import parallelize
...
out = parallelize(
fun=_fit_and_score,
iter=candidate_params,
backend=backend,
backend_params=backend_params,
) |
Beta Was this translation helpful? Give feedback.
-
|
exactly! Of course you need to ensue that |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
In the
EnsembleForecaster, when an exception occurs, the code halts. This is often intentional, allowing for adjustments in preprocessing or model substitution. However, it becomes challenging in tasks where I forecast hundreds or thousands of time series. Specifically, tailoring a model to each individual time series is often impractical. Using the sameEnsembleForecaster(fit to each time series) not only reduces the risk of overfitting but also simplifies maintenance.Previously, I've handled this with a try-except block::
As a potential enhancement, I propose adding an error handling parameter to
EnsembleForecaster, such aserrors="ignore",errors="raise", orerrors="warn". This would enable the forecaster to continue producing forecasts even if one component fails, unless all or a specified number of models fail. Example:Would this feature be of interest to others? Happy to work on a PR to develop it if there's an interest in it.
Beta Was this translation helpful? Give feedback.
All reactions