pommedeterresautee OP t1_ituya3z wrote on October 26, 2022 at 2:32 PM

Reply to comment by Lolologist in [P] Up to 12X faster GPU inference on Bert, T5 and other transformers with OpenAI Triton kernels by pommedeterresautee

I have not used Spacy since years but my understanding is that for large models they leverage Hugging Face library (https://spacy.io/universe/project/spacy-transformers), so I would say it should work out of the box, the only thing is to catch the model instance and override it with the optimized version (it will take the very same input).

Maybe a redditer with more Spacy knowledge than I have can validate the approach...

reSAMpled t1_iuj2yrm wrote on October 31, 2022 at 6:33 PM

I also haven't used spaCy in a while, but I am pretty sure there is not a way to make this work with -sm, -md or -lg models, but what Michaël says should be true for -trf models, but I don't think it will be easy. Already spacy-transformers has to wrap HF models so they have a thinc API, you would have to dig deep in there to call Kernl's optimize_model