someone always screams when you take it out abruptly in public

my daytoday tasks are more with classical ML as opposed to DL but we take offline DL model score as a features. for classical boosted models, we normally use a tool developed over shap decision plots. where you can visualise plots on sample values for each features. In those plots for each sample you can see that how much contribution ( in +ve or -ve direction) that feature value makes to the final output.

For DL/computer vision specific tasks it wouldn't be practical since each pixel is your feature however shap packages provides a way to generate heatmaps for image classification as far as i can recall.

That being said I'm also keen to know how other DL/ML practitioners do their model debugging and especially if someone has done it for NLP domain, then i would really like to hear their experience.