When you said 'avoid high level frameworks', I thought you were going to ask about building matrix operations with for loops in C++...

I'm here to say that compilers aren't as good as you imagine, and the built in intrinsics in pytorch will perform far better than anything you will write in C/C++/assembly (without months of effort), even before you get to GPU stuff...