Viewing a single comment thread. View all comments

p0p4ks t1_jcppzf4 wrote

I get these confusions all the time. But then I remember we are back propagating the errors. Imagine your case happening and the model output was incorrect, the backprop will take care of fixing the key value being too big or small and fix the output.

1

FunQuarter3511 OP t1_jcpstej wrote

Fully agree!

I think my issue was that because of the terms query, key, value, I was trying to relate them in a database or hashtable context. But in reality, those terms seem to be misnomers, and backprop will set the key/query pair to whatever is needed such that the dot product for important context will be large and be weighted appropriately.

I was over complicating it.

1