konze t1_iu37t3g wrote on October 28, 2022 at 5:43 AM

Reply to Do companies actually care about their model's training/inference speed? by GPUaccelerated

I’m coming from academia with a lot of industry connections. Yes, there are a lot of companies that need fast DNN inference to point where they build custom ASICs just to fulfill their latency demands.

konze t1_is2i187 wrote on October 12, 2022 at 8:43 PM

Reply to comment by muunbo in Optimizing deep learning inference to run on the edge? by muunbo

No, the data already comes in the “correct” format as a data stream from a sensor. The DNN models are trained to work on that data stream, on an edge device you usually don’t have the performance to pre process anything, except for an FFT which is a fixed function accelerator that can process the data in real time.

konze t1_irzfxlz wrote on October 12, 2022 at 4:37 AM

Reply to Optimizing deep learning inference to run on the edge? by muunbo

I contribute to our group who is working exactly on this. Currently, it is quite a mess because each HW vendor provides its own tooling for deploying on their device which leads to a lot of problems (e.g. missing support for certain layers). One of the most promising tools for edge deployment is TVM together with Network Architecture Search (NAS) where the network is tailored for a specific use case and the available resources.