Viewing a single comment thread. View all comments

Spoffort t1_itbjrj0 wrote

Reply to comment by FirstOrderCat in U-PaLM 540B by xutw21

I know what do you mean, look at the x axis where compute is. The model is not 2 times better (your point with y axis) but 2 times less compute for given outcome (x axis). If you want i can explain it further 😄

3

FirstOrderCat t1_itc6pne wrote

It looks like they had point of diminishing return somewhere at 0.5*1e25 FLOPS.

After that model trains much slower. They could continue training farther, and say they "saved" another 20M TPU hours.

1