To anyone wondering, this is largely an iterative work over facebook’s prior work. As far as I’ve been able to tell there is little going on that’s groundbreaking, think of it as the difference between a small GPT model and a large one.