Viewing a single comment thread. View all comments

mgwizdala t1_j5tyf1f wrote

If you are willing to trade time for batch size you can try with gradient accumulation

8

RaptorDotCpp t1_j5u0yxq wrote

Gradient accumulation is tricky for contrastive methods that rely on having lots of negatives in a batch.

13

altmly t1_j5uglpx wrote

I'm confused. Gradient accumulation is exactly equivalent to batching as long as the data is the same, unless you use things like batch norm (you shouldn't).

1

Paedor t1_j5ur6tx wrote

The trouble is that contrastive methods often compare elements from the same batch, instead of treating elements as independent like pretty much all other ML (except batchnorm).

As a simple example with a really weird version of contrastive learning: with a batch of 2N, contrastive learning might use the 4N^2 distances between batch elements to calculate a loss, while with two accumulated batches of N, contrastive learning could only use 2N^2 pairs for loss.

11

satireplusplus t1_j5v24u2 wrote

If you don't have 8 GPUs you can always run the same computation 8x in series on one GPU. Then you merge the results the same way the parallel implementation would do it. In most cases that's probably gonna end up being a form of gradient accumulation. Think of it this way: you basically compute your distances on a subset of n, but since there are much fewer pairs of distances, the gradient would be noisy. So you just run it a couple of times and average the result to get an approximation of the real thing. Very likely that this is what the parallel implementation does too.

1

koolaidman123 t1_j5ujfpv wrote

contrastive methods require in-batch negatives, you can't replicate that with grad accumulation

7

shingekichan1996 OP t1_j5u22zn wrote

Curious about this, I have not read any paper related. What is its effect on the performance (accuracy, etc) ?

1

mgwizdala t1_j5u2mgr wrote

It depends on implementation. Naive gradient accumulation will probably give better results than small batches, but as u/RaptorDotCpp mentioned, if you relay on many negative samples inside one batch, it will still be worse than a large batch training.

There is also a cool paper about gradient caching, which somehow solves this issue, but again with an additional penalty on training speed. https://arxiv.org/pdf/2101.06983v2.pdf

1