Viewing a single comment thread. View all comments

LeN3rd t1_jcgzk3c wrote

Reply to comment by ilrazziatore in [D] Simple Questions Thread by AutoModerator

If it is model uncertainty, the bnn should only assume distributions only for the model parameters, no? If you make the samples a distribution, you assume data uncertainty. Also I do not know exactly what you other model gives you, but as long as you get variances, I would just compare those at first. If the models give vastly different means, you should take that into account. There is probably some nice way to add this ensemble uncertainty with the uncertainty of the models. Also this strongly means that one model is biased and does jot give you a correct estimate of the model uncertainty.

1

ilrazziatore t1_jch3vpu wrote

Uhm..... the bnn are built assuming distribution both on th parameters( ie the value assumed by the neurons weights) and on the data (the last layer has 2 outputs : the predicted mean and the predicted variance. Those 2 values are then used to model the loss function which is the likelihood and is a product of gaussians. I think its both model and data uncertainty.

Let's say I compare the variances and the mean values predicted.

Do I have to set the same calibration and test dataset apart for both models or use the entire dataset? The mcmc model can use the entire dataset without the risk of overfitting but for the bnn it will be like cheating

1

LeN3rd t1_jchht71 wrote

Than I would just use a completely different test dataset. In a paper I would also expect this.

1

ilrazziatore t1_jciyif4 wrote

Eh data are scarce, I have only this dataset ( it's composed by astrophysical measures, I cannot ask them to produce more data).

1