Viewing a single comment thread. View all comments

BestSentence4868 t1_iu8fluq wrote

Love love love Triton inference server for the framework flexibility. Super mature with so much stuff I never thought I'd need like model warmup etc. ORT and TensorRT are cool, but if all else fails Python backend is awesome.

6

big_dog_2k OP t1_iu8gcr1 wrote

Great! Does Triton allow something like native pytorch models? Or is it onnx, tensorRT, torchscript?

1

BestSentence4868 t1_iu8gi68 wrote

Yep! Fire up Triton(I'd used their docker container), install pytorch via pip or just put it in the dockerfile and you're off to the races! I actually did just deploy Triton+pytorch+flask for a web app this week :)

1

big_dog_2k OP t1_iu8gxg1 wrote

Wow! I did not know that! I think I have answers to my questions now.

1