Why prediction algorithms fail because of Covid-19

The main components for Machine Learning

Like common sense expectations, machine learning methods for forecasting are based on what happened in the past. These methods have four main components that contribute to the predictions:

1. the current level (average during last week, month or year)
2. the recent trend (going up or down, at a certain speed)
3. seasonality (impact of season or day of week)
4. other factors (partially treated as unexplained noise)

It is possible to approach the consequences of covid-19 as a change in level: figures are higher or lower than expected, and this deviation may continue for some time. Apart from the challenge of not knowing the duration, the direction of the effect may also change. For example, online retailers have seen an initial decline in sales that was followed by a steep recovery and increase. For short term predictions, modeling covid-19 effects as a change in level (or trend) may be the best available approach. The data so far provides a way to estimate the impact, and a scenario to work with.

‘Unexplained noise’

In the long run, it might make more sense to treat covid-19 as a part of ‘other factors’. The effects are present for some time, and not at other times, at least in the past. For machine learning, it is a problem that there are no historical examples to learn what effects can be expected. Without additional assumptions about how the effects develop, covid-19 effects are part of unexplained noise – with a considerable impact.

Lessons learned

To conclude, the bad news is that events (or changes) like covid-19 pose a challenge to forecasting that has a large impact and is really hard to model. The good news, if any, is that not just sophisticated forecasts but also layman expectations are off. And we learn a humbling lesson that any forecast, no matter how sophisticated, only succeeds to deliver good predictions as long as the other factors are not changing too much.