Ultimately AI will become a big boys club, where big corporate will hold all the cards.

OpenAI just made the first leap towards that dystopian near future.

No they didn't.

Actually I've been sort of looking at ML computers (kind of of browsing and dreaming one day I would have one, but it's always going to be out of my means and needs anyway). Anyway, they can put two PSUs in a box, obviously it's made by companies, so the total cost is twice or 3 times the cost of the parts alone (e.g. building yourself would be 2-3x cheaper) but it could inspire you for picking your parts.https://bizon-tech.com/amd-ryzen-threadripper-up-to-64-cores-workstation-pc

https://shop.lambdalabs.com/gpu-workstations/vector/customize

I suppose one PSU will take care of motherboard + CPU + some GPUs and the other one will take care of remaining GPUs.

So if you get 4x 3090, that's 350W x4 = 1400W just for GPUs, +300 watts for CPU, +powering the rest of the components, drives, etc... So let's say we round that up to 2000W, then add at least 10% margin, that's 2200W total.

So maybe 1600W PSU for mobo and some GPUs, and another 1000W or more for the remaining GPUs. Note, if you go with 3090TI, it's more like 450-500W per card, so you have to do the maths.

Or if you want to go future proof, just put two 1600W PSUs, and then you can just swap your 3090 with 4090 in the future and not worry about upgrading PSUs.

Do you have some numbers or a link because all benchmarks I've seen point to the contrary? I'm happy to update my opinion if things have changed and there's data to support it.

I don't know if that's going to work well to have 16 PCIe lane, everyone here I've seen making 4 GPUs machines uses the CPUs that have 48 or 16 PCIe lanes.

Also you'll need a lot of watts to power that monster, not to mention you need a 10-20% margin if you don't want fry the PSU.

Each time someone asks this question, someone repeats this misinformed answer.

This is incorrect, NVLink doesn't make much difference.

https://www.pugetsystems.com/labs/hpc/rtx-2080ti-with-nvlink-tensorflow-performance-includes-comparison-with-gtx-1080ti-rtx-2070-2080-2080ti-and-titan-v-1267/

Thanks that makes sense, I thought they were a startup in the business of building computers, I was completely confused!!!

I'm not following: you're doing start-up on infrastructure build and you have to ask for advice on reddit to scale past 1 machine? That gives a terrible image of your startup. To the average person like me it sounds like you don't know what you're doing.

Wow, that's a nice extension/replacement. Very cool work!

Correlation is not causation.

Just get two 4090 and power limit them to 350 watts using "nvidia-smi", there's almost no performance loss and you don't need NVLink anyway to do multi-gpu training.

Those $10,000 won't last long on AWS. There's also LambdaLabs, their cloud prices are a lot more affordable. They also make dedicated servers for machine learning.

Maybe it was a pain years ago, I don't really know but nowadays you just click install nVidia drivers in the software management and it works. There's nothing painful or difficult about it.

If I was Schmidhuber, instead of whining every time someone "stole my idea that didn't really work" and made it work, I'd just revisit my old ideas and make them work myself.

It's like inventing the airplane, there's a big difference between the general idea and actually making one that flies.

Really cool stuff!

This is amazing, the code is so clear I feel I can finally understand RL!

Bookmarked.

Looks like a lot of sprinting...

The performance is not better: the results are the same within the margin of error for standard (not super-deep networks). Here I copied from their table:

Cifar10

ZerO Init 5.13 ± 0.08
Kaiming Init 5.15 ± 0.13

Imagenet

ZerO Init 23.43 ± 0.04
Kaiming Init 23.46 ± 0.07

Like what? Training ultra-deep neural networks without batchnorm? But in their experiments the accuracy gets worse with deeper networks, what's the point of going deeper to get worse results?

Seems useless, why not simply fix the seed of the random generator for reproducibility?

Oh I didn't realize you saw it as a simplification because all it does is remove the input size but then you need to create input placeholders. I thought it was meant to reimplement Keras on PyTorch.

I am not understanding the point of rewriting Keras on top of PyTorch? Is it just for your own fun?

Why not use Keras directly? Keras has better deployment capabilities as a bonus.