Viewing a single comment thread. View all comments

mike94025 t1_je5ojaw wrote

Reply to comment by tripple13 in [D] PyTorch 2.0 Native Flash Attention 32k Context Window by super_deap

It is. Follow the call tree into F.multi_head_attention_forward

1

tripple13 t1_je5seed wrote

Is that right? I some how end up here when trying to assess what the F.multi_head_attention call does in the Class definition.

But I trust you're right, it would only make sense, I just couldn't identify the calls myself.

1