[D] Pytorch or TensorFlow for development and deployment?

Submitted by CodaholicCorgi t3_z50vvq in MachineLearning

Hi folks,

I know this question is kept being asked for many times, but as time change, and Pytorch is getting more favorable everywhere, I want to bring this one on the table again.

Assuming that you are building AI products (deployment is a must!), do you prefer TensorFlow or Pytorch in your codebase and why?

For me (and most likely a lot of people out there), I prefer training and developing my models in Pytorch (ease of debugging and customizing). But if that's a case, is there any option to deploy the .pt models? Even if we need .tflite for mobile deployment?

Thanks in advance!

66

Comments

You must log in or register to comment.

Deep-Station-1746 t1_ixu2bxb wrote

First, PlayTorch is a thing. That could replace tflite for mobile.

Second, you can use ONNX to transfer models from pt to tf, then to tflite.

Last but not least, if you are a REAL MAN, you can try to translate the model weights to tf by hand, using numpy as an intermediary.

71

parabellum630 t1_ixuno8d wrote

Oh no! I had no idea about ONNX and just transfered a huge model from tf to pyt manually.

18

JustOneAvailableName t1_ixv2bz6 wrote

ONNX tracers are decent, but aren't that good, I had to rewrite Wav2vec to get both batch size and input length as dynamic axis.

8

parabellum630 t1_ixvapdu wrote

I see. But can they be used as a good starting point to build upon?

2

agupta12 t1_ixwfbow wrote

I have been trying to convert wav2vec2 model to onnx without much success

1

CodaholicCorgi OP t1_iy1rgxx wrote

Agreed with all the folks, if your model is far beyond ordinary CNN and RNN then ONNX will give you a bunch of warnings that confuses you.

2

Background_Thanks604 t1_ixu3170 wrote

How do you mean - translate model weights by hand?

4

Deep-Station-1746 t1_ixu6ych wrote

Load pt model with torch, get all weights with state_dict. Make them all into channels last format with permute. Load all the arrays into numpy. Load all the arrays into tensorflow. Write the network forward logic in tensorflow. Plug in weights. Run.

31

Background_Thanks604 t1_ixuccby wrote

Thanks for clearification! Do you know if there is an tutorial/blog post for this approach?

4

Deep-Station-1746 t1_ixugp6t wrote

Nope. I don't think so. If you need help and are willing to wait a bit (A bit busy right now), DM me and I'll take a look at your problem.

6

Background_Thanks604 t1_ixukvcg wrote

Thx - appreciate it! I dont have a problem i just want to learn/try this approach because i never heard of it.

7

jobeta t1_ixum5yx wrote

How complex is the model you want to translate?

1

Background_Thanks604 t1_ixun76x wrote

I dont have a model to translate - i read about this approach in the comments and i want to learn about it.

2

ApeForHire t1_ixwveuz wrote

I was actually able to do this once by relying heavily on Github's Copilot, ie just giving functions very specific names and commenting things like "this function converts a pytorch model into a vector of weights" etc. It worked pretty well and was simple for basic network architectures, though I imagine it could get more complicated.

1

CodaholicCorgi OP t1_iy1s4ws wrote

I did it once in one of my projects. It's basically hand-picking weights from Pytorch model's layers to TensorFlow model's layers, but it feels more reliable than relying on ONNX with a bunch of warnings.

Though there are not much tutorials or blog posts about this, I will try creating a github repo for this later (just examples with simple layers), so many more people will know that this technique exists.

2

sarmientoj24 t1_ixukgwd wrote

Not a big TF guy but thr SOTA models nowadays will usually be in Pytorch. So if you need those, then Pytorch.

38

a1_jakesauce_ t1_ixvn6jf wrote

I don’t know much about this topic but to me, it seems there is a difference between a SOTA model developed for a paper and a model ready for deployment. Seems torch is easier to dev while tf uses lazy evaluation and is therefore lighter. Could be wrong

2

sarmientoj24 t1_ixvouw5 wrote

The thing is, either you wait for the TF support or you code it yourself. Research is fast paced so researchers want a boilerplate, or just to build on top of another repo.

For deployment, there's a lot of deployment tools for Pytorch nowadays. I use BentoML for deploying compvision models. If you want a lighter model, there are libraries and repositories that support sparsification, pruning, etc. of your models. A lot of newer repos for industrial use (mmdet, YOLOv5/6/7) have sparsification and pruning support. Some of them even have distillation built-in. Again, I am not a TF guy so I havent seen such rich support for model deployment than these Pytorch-based repos.

While it is true that research and deployment are different, it is up to your team how to do MLOps on those SOTA methods. New SSL method, new loss functions, new stuff -- either you borrow it from a repo or you code it yourself. In the industry, the prototyping procedure for experimentatio is critical for a fast-moving product. You dont have time to convert codes. So you test repositories, then build on top of it if it is successful.

6

erannare t1_ixvt17s wrote

TensorFlow has model optimization libraries such as, but not limited to: weight clustering, pruning, and weight quantization, as well as training support for these.

2

sarmientoj24 t1_ixw1ca2 wrote

That's great. Again, I am not used to TF. My point was that Pytorch has deployment capabilities near TF's nowadays and that most SOTA and new research are leaning towards publishing their code with Pytorch.

1

sarmientoj24 t1_ixvpd7m wrote

Also, if u want a middle ground, then go for ONNX. Just convert ur tf or pytorch model to onnx.

1

drivanova t1_ixw8w0v wrote

PyTorch.

Datapoint 1: it’s part of the Linux foundation

Datapoint 2: Jax

Disclosure: I haven’t used TensorFlow.

6

rsandler t1_ixxu6px wrote

My 2 cents: I am still the stubborn minority using TF. Mostly because of switching costs in my organization. But also, honestly, because I am decently happy with it. I think the ease-of-use gap between pytorch and TF was mostly erased with the release of TF2. I was really hoping that more of the research community would come back to TF after that, but unfortunately that has not been the case. In practice, it does suck to not have access to SOTA models. Otherwise, I think they are pretty similar.

3

nwestninja t1_ixx5ewa wrote

I usually do onnx or tensorRT for deployment and use whatever I want for deployment. Most libraries can serialize down to an ONNX model or a tensorRT model.

1

wisscool t1_ixw4fd7 wrote

Develop in whatever framework you like, then deploy with Triton inference server

−1

[deleted] t1_ixvzhu3 wrote

[deleted]

−6

[deleted] t1_ixtv8dj wrote

[deleted]

−19

a1_jakesauce_ t1_ixvnevj wrote

The ecosystems are very dissimilar so it’s not easy to deploy a model in both

4

kaskoosek t1_ixvnka0 wrote

Yeah i might be wrong based on the downvotes.

;)

1

a1_jakesauce_ t1_ixvnw93 wrote

I don’t know much about this either but it seems you’re right since there’s not a one size fits all solution for anything. But at the same time, it’s not very helpful advice

1

kaskoosek t1_ixvor9u wrote

Definitely not helpful advise. Anyway it feels like the question caters to fanboys of each enviroment.

Both pytorch and tensorflow are pretty good, though each has their own strengths.

1