> implement a gradient descent optimization process at inference time

Could you expand on what this means? At inference time, I thought all weights were frozen, so how could the attention layers be somehow performing gradient descent?

Edit: I read the paper in detail and understood it (walk through the math in Section 3). Basically, the sentence itself X has some weights that go through the attention layer (recall how attention works: it embeds the sentence, then multiplies it by key, value, query matrices). If you give it some examples, X', to learn from, well, of course there are going to be weights for both X, and X'. Turns out those weights for X' end up being equivalent to stepping in gradient descent.

> And integrating good-old-fashioned-ai > (GOFAI) with more modern ML is becoming an area of increasing research interest.

Any papers you recommend on this topic?

Could you elaborate what you mean by undertrained?

Btw, one should take care not to implement the worst-case O(n) algorithm (which is Quickselect + Median of Medians), because it has high constant factors in the time complexity which slow it down in the average case. QuickSelect + Random Partitioning, or Introselect (the C++ standard library function mentioned) have good average time complexities and rarely hit the worst case.

I think the one in the standard library is introselect, which is a hybrid of QuickSelect

Curious - what are the primary differences between this, and the "information correction system" in LaMDA? https://arxiv.org/pdf/2201.08239.pdf figure 3