Viewing a single comment thread. View all comments

Fancy_Traffic7435 t1_izkikl5 wrote

Can you? Sure. Should you? Not sure about that.

If you think of neural networks as underlying probability distributions, then estimating the error of your regression as single values can seem misleading, as the true error actually lies within a distribution of predicted errors. Going off of that, it would be better to analyze the distribution of your errors in relation to your predictions.

16

Alex-S-S OP t1_izkirr6 wrote

I can do that during tests on labeled data but what about unlabeled data?

2

Fancy_Traffic7435 t1_izkksyd wrote

If you have the assumption that your test data and forecasting data are from the same distributions, which they should be unless data drift is occurring, then you can create the distribution of your errors when your model has been trained by looking at the errors for your test set. From there you could simply say, "This is my prediction and this is the general distribution of error, so therefore this is the distribution of my potential values," or you could take it a step further and look at the errors associated with the test data that are most similar.

There may be other techniques that work better, so further reading may help.

4

PandaMomentum t1_izlhb57 wrote

I feel like you're trying to re-invent regression residuals analysis in a NN setting? Where you're comparing your data points to the predicted values or class? There are a lot of tools on the regression diagnostics side, most of which are looking for things that don't really matter in an arbitrary non-linear curve fitting process like NN. So depends on what your need is for the error analysis.

1