How good is your forecast really?
A methodically fair forecast benchmark on your data – with metrics agreed in advance, an identical cut-off and a documented results report.
aimpera v3 versus the existing forecast
We are convinced that the best forecast only proves itself in comparison. That is why benchmarking is central to us. Get in touch – together we will find the best forecast for your use case, transparently and fairly.
aimpera v3 vs. comparison forecast
· Weekend · Public holiday (Rhineland-Palatinate) · Click lines to show or hide them · quarter-hourly time series provided, loaded dynamically on page view.
Everything for a fair forecast comparison
The Benchmarking Suite bundles everything a robust comparison needs – from a fair comparison environment to the results workshop.
Fair comparison environment
Identical actuals, timestamps, horizon and cut-off – the foundation of every robust comparison.
Baseline & challenger
Existing forecast, a suitable baseline and the aimpera v3 forecast model – calculated out-of-sample.
Metrics set
Technical quality and economic impact – reported separately and transparently.
Segment & portfolio analysis
Errors by segment, period, horizon and portfolio level – including the offsetting effects between individual time series.
Results report & scorecard
Executive summary and forecast scorecard as a basis for decision-making.
Results workshop
Joint interpretation of the results and a clear recommendation for the next step.
We do not want to claim that our model is always the best – we want to prove it. In a fair comparison, under equal conditions. What matters to us is to find – or even develop – the best forecast for each use case together.
Same data. Same timestamps. Same horizon.
Forecast quality can only be compared meaningfully under identical conditions.
- same actuals and same evaluation period
- same forecast cut-off and same horizon
- same temporal resolution
- same information state at the cut-off (incl. weather data state)
- same aggregation and missing-data rules
- temporally separated out-of-sample evaluation
- no information that was not known at the time of the original forecast
Six dimensions of forecast quality
Forecast quality emerges from several perspectives: error level, systematic deviations, critical forecast errors, temporal structures, special situations and portfolio effects. Depending on the forecast type, RMSE, probabilistic scores and other suitable metrics complement the assessment.
Five steps to a basis for decision-making
From clarifying the data through a fair comparison to the joint results workshop – with a clearly defined, transparent process.
The individual use case is at the centre. Together, we take into account the relevant markets and trading logic, the available data basis and the existing portfolio structure. This creates the conditions to optimize forecast quality specifically for the concrete application – instead of evaluating a standardized model detached from its actual use.
The benchmarking is deliberately iterative: right after the initial forecasting phase – about two weeks after the start – we review first observations and findings together in the Results Preview. This feedback flows directly into the further analysis. Midway through the benchmark, a Benchmark Update follows, in which we jointly interpret interim results, dig deeper into open questions and define the next analysis steps.
Finally, the results are brought together comprehensively in the results workshop: we assess forecast quality and impact, prioritize the potential identified and derive concrete next steps for the respective use case.
Define data and scope
Historical actuals, existing forecast, metadata, cut-offs and portfolio mapping.
Reproduce the baseline
Ensure comparable conditions and map the starting point reliably.
Run the challenger
The aimpera forecast model is calculated out-of-sample over a historical period.
Results Preview (approx. 2 weeks after start)
Interpret first observations together and sharpen the focus for the further analysis.
Benchmark Update
Reflect interim results, integrate feedback and define the next analysis steps.
Assess quality and impact
Technical and – where possible – economic assessment, clearly separated.
Results workshop
Consolidate results, prioritize potential and derive next steps.
The benchmark in detail
Do you need access to our production data?
Ideally, we work with the most recent production data possible, because it reflects real data quality and operational particularities. Direct access to your production system is not required for this: you can provide the data offline as an Excel or CSV file, or connect it online via our documented interfaces.
Do you need our existing forecast?
No. An existing forecast is helpful for a direct comparison, but not a prerequisite. If it cannot be provided, we compare aimpera under identical conditions against suitable, transparent baselines.
Are the data or forecasts we provide passed on to third parties?
No. The data and forecasts you provide are treated confidentially, processed exclusively for the agreed benchmark and never passed on to third parties.
How long should the historical period be?
That depends on the forecast type and objective. We define the evaluation period together so that the result is robust and out-of-sample. A good period should, if possible, cover at least one month and ideally include heterogeneous influences such as changing weather.
Which metrics are used?
Average error (e.g. MAE, normalized), bias, large deviations (tail errors), temporal patterns and portfolio effects. Further metrics depending on the forecast type. Which metrics are most meaningful for your use case is defined together with you and our experts.
Can an external provider also be compared?
Yes, absolutely! It only becomes interesting and meaningful when there is something to compare – so, if possible, the existing forecast should always be included as well (this can also be done directly via our platform). An independent challenger can show where an existing external provider is strong or weak.
What happens after the benchmark?
You receive a transparent results report and a results workshop with a recommendation – ranging from "no relevant advantage" to parallel operation or productive delivery. It is very important to us to assess the results together with you and derive a good, sustainable decision from them.
Does our forecast have to be fully reproducible?
Not necessarily. What matters is a fair comparison under identical conditions – same actuals, timestamps, horizon and cut-off.
What happens with incomplete forecast histories?
We flag gaps transparently and evaluate only what is robust under comparable conditions.
Is the evaluation done on individual time series or at portfolio level?
Both are possible – from individual market locations (MaLo) through clusters to the relevant total portfolio. At portfolio level, we take into account that over- and underforecasts of individual time series can partly offset each other.
How can you tell whether an AI load forecast is actually better than an existing forecast?
Whether an AI load forecast is better can only be determined through a methodologically fair comparison. To do this, the forecasts must be evaluated on identical actuals, forecast times, horizons and cut-offs.
Important metrics include the mean absolute error (MAE), systematic forecast deviations (bias) and particularly large forecast errors (tail errors). Quality at portfolio level and possible economic effects should also be considered.
In the aimpera Forecast Benchmark, existing forecasts are compared with the aimpera v3 forecast model under comparable conditions. This shows for which time series, customer groups or sub-portfolios improvements are actually achieved – and where an existing method already works well.
Request a forecast benchmark
Tell us the forecast type, portfolio scope and evaluation period – we will give you a clearly defined proposal.