Viewing a single comment thread. View all comments

DreamyPen OP t1_ivvm1k0 wrote

Reply to comment by Ulfgardleo in [Discussion] Can we train with multiple sources of data, some very reliable, others less so? by DreamyPen

  1. Yes I did mean outputs/targets. The features are always known, they correspond to testing conditions (a certain temperature, a certain processing speed, etc.) Given these testing conditions (inputs / labels), can we predict the material properties (outputs/targets) Experimental measurements are very reliable.

  2. The physics based model can always output a prediction for any given labels (testing conditions). But it is not always reliable. We would still like to include them because it allows us to augment the small experimental data set, and, often times, it is quite good approximation from the ground truth. This will also answer 4. Indeed, since the physics based model can always make predictions, we will have in some instances both reliable and unreliable data.

  3. Correct! :)

  4. We do indeed.

  5. Hopefully my response to 1. clarified it.

Let me know if the goal is clearer, and thank you for your help.

1

Ulfgardleo t1_ivxodj0 wrote

  1. okay you completely confuse everyone in the ML community when you call inputs "labels". lets keep with inputs/outputs.

  2. This is good, because it allows you to estimate some crude measure for the quality of the physics model.

So, label noise is a broad field. I am mostly knowledgeable in the classification setting, where label noise has different effects. Moreover, you are not in the standard noisy label setting, because the noise is not independent of the label, so just using weights will be difficult. Similarly, if you have more than one output to predict, a single weight is difficult to compute.

The standard way to derive all of these methods is by noting that the MSE can be derived as the log-probability of the normal distribution p(y|f) where y is the ground truth and f is the mean, and variance is some fixed value. For the mse, the value of the variance does not matter, as long as it remains fixed, but with fairly little effort you can show that as soon as you give samples individual variances, this amounts to weighting the MSE.

So, the cheapest approach would be to give outcomes from the different sources a different variance and if you have more than one output, you will also have more than one variance. How do you guess the parameters? well, make them learnable parameters and train them together with your model parameters.

Of course you can make it arbitrarily complicated. Since your cheap labels come from a physics simulation, errors are likely correlated so you can learn a full covariance matrix. And from there you can make it as complex as you like by making the error distribution more complex, but you will likely not have enough data to do so.

1