3D morphable models are the way to go for image of face to 3d model. I have previously done a lot of research work on this. The deep learning model takes an image as input and outputs 3d model as an obj or fbx file. This can be used in any 3d editing software. Take a look at DECA on github.

I see. But can they be used as a good starting point to build upon?

Oh no! I had no idea about ONNX and just transfered a huge model from tf to pyt manually.

Oh my God. I used to do this too! I am happy I am not the only one!! But my monkey brain learned not to do this eventually. I have managed to get it to GRU performance by applying more warmup steps, learning rate scheduling, decreasing model size, using Pre-LN, doubling the batch size, and reducing the sequence length.

Interesting, I will try this out as well.

Thank you!! I was experimenting with off-the-shelf implementation with little customization. I am using the transformer in an encoder fashion with 800 hidden dimensions due to the constraints of other models surrounding it. I will try out varying all these hyper parameters. Looks like it's going to be a long week.

Thank you so much for these insights!! I will try these out.

I worked on eyes and upper head shape while the paper I built upon (DECA) tackles personalized fine details. But I do agree hair modeling and finer details and texturing is better with volumetric approaches. Maybe a fusion of the two would be a good research direction? Like the end goal is not only the modeling but also using it for real tasks.

What are your thoughts about 3DMM based models. I am doing research on avatar generation from monocular images using 3DMM based approaches and found them to be better in terms of inference speed, animatability and ease of integration into existing creative workflows.