Imo, transformers are significantly less simple and more "hand-crafted" than lstm.

The point of the bitter lesson, I think, is that trying to be clever ends up biting you and eventually compute will reach a point when you can just learn it. Cross attention and all this special architecture to help a model capture intraseries information is definitely being clever when compared LSTM (or rnns in general) which just give a way for the network to keep some information around when presented with things in series.

But this bit:

>"This is a fairly new model and I do not know the details"

is hard! I understand having anxiety about being The ML Guy and not being able to immediately answer questions.

I kinda get where OP is coming from, though. With all the pop-sci ML stuff and big press releases for popular consumption hitting really shortly after actual publication, there's always a risk that some manager will be like "hey I just read about stable diffusion on Twitter, can we use it to do this?" and then you're a deer in headlights bc you weren't at the press conference where they introduced it and you have no idea what the manager is even talking about.

Generative image models probably need to fork off sometime soon, especially text-guided versions. It's a pet peeve of mine that we're calling it "vision". Vision, at least to me, implies seeing/making sense of what is actually there.