Viewing a single comment thread. View all comments

trikortreat123 OP t1_itmkk63 wrote

Reply to comment by parabellum630 in [D] Neural Avatar Community by trikortreat123

3DMM you're embedding 3D head shape into several low-dimensional PCA spaces so you lose the refined details necessary. If you use a mesh based 3DMM you can not represent personalized facial details due to its limited representation ability. Hair, eyes, and teeth are also huge problems to resolve. The approach that's most convincing is volumetric based representation texturing on top of some mesh structure or canonical face model. The problem with Nerf/volumetric representations is either the amount of data required, the number of angles required for Nerf, or the time it takes to train/inference speed. There's been good work specifically in https://arxiv.org/abs/2210.06108 where inference takes 20 seconds (still a much longer time than i would like), but it's improving and highly accurate especially for expression transfer

3

parabellum630 t1_itmm0ei wrote

I worked on eyes and upper head shape while the paper I built upon (DECA) tackles personalized fine details. But I do agree hair modeling and finer details and texturing is better with volumetric approaches. Maybe a fusion of the two would be a good research direction? Like the end goal is not only the modeling but also using it for real tasks.

2

trikortreat123 OP t1_itmninz wrote

For avatars to have any functionality in the real world and for the technology to truly be adapted there has to be realism beyond the uncanny valley. You cannot implement a really fast model that accomplishes tasks if you don't have a great sense of depth, emotion, and detail. Speed and functionality is very important but the core problem lies in people not wanting to use a C-grade model which doesn't have any realism. Over the past 2-3 years Nerf training speed has gone down from a few days to just a few minutes with Instant-NGP so the functionality is improving.

1