LurkAroundLurkAround t1_ivup7pc wrote on November 10, 2022 at 7:13 PM

By far the easiest thing to do is to feed in the data source as a feature. This should allow the model to generalize across datasets as much as possible, while accounting for different inherent properties of the data

DreamyPen OP t1_ivvdy75 wrote on November 10, 2022 at 9:58 PM

I like this idea! Definitely worth trying, thank you.

farmingvillein t1_ivvkfko wrote on November 10, 2022 at 10:44 PM

This is the right 80-20 starting answer.

ObjectManagerManager t1_ivwf1f0 wrote on November 11, 2022 at 2:38 AM

Alternatively, feed the data source as an output. i.e., have your model output two values. For data sourced from dataset A, minimize loss against the first output. For data sourced from dataset B, minimize loss against the second output.

I don't remember who, but someone wrote a thesis on how it often works better in practice to incorporate additional / auxiliary information in the form of outputs rather than inputs. It's also a very clean solution since you can usually just remove the unnecessary output heads after training, which might decrease your model size for inference (albeit a small amount, unless you have a lot of auxiliary information).