Not sure what you mean by specific data. Here is a good read - When Does Label Smoothing Help? (https://arxiv.org/pdf/1906.02629.pdf)

I think you are looking for federated learning which is complete research field on its own. It digs into combining weights of two neural networks such that both tasks can still be accomplished. Existing approaches should apply to transformers too.

Use scikit learn mlp classifier if you have to use mlp.

Transfer learning. With the availability of these large models, it will be in demand to adapt them to smaller/use case specific datasets.

Another approach can be cyclegan or pix2pix, both are gan based model. Pix2pix works on paired data and cyclegan works can use unpaired data.