[D] Neural Avatar Community

Submitted by trikortreat123 t3_ychg1t in MachineLearning

Avatars are going to be one of the most critical components in the next decade. I have not found any community which has deep insight beyond just a subsection into this brilliant field. Meta Reality Labs, Disney Research, and Microsoft Research are just a few companies developing neural avatar representations of the face and body with very realistic results. I've created a discord so that we can share the latest research and datasets which i will update constantly anytime I find anything fascinating and work on building a realistic neural avatar as well. Hopefully we can build a community big enough similar to even stable diffusion where we can create the most realistic avatars possible.

Here are some references that Meta, Samsung, and University of Science and Technology of China have achieved

https://arxiv.org/abs/2103.01954

https://arxiv.org/pdf/2207.07621.pdf

https://arxiv.org/abs/2210.06108

0

Comments

You must log in or register to comment.

jobeta t1_itootcr wrote

Why avatars would be so important?

3

parabellum630 t1_itmihvd wrote

What are your thoughts about 3DMM based models. I am doing research on avatar generation from monocular images using 3DMM based approaches and found them to be better in terms of inference speed, animatability and ease of integration into existing creative workflows.

2

trikortreat123 OP t1_itmkk63 wrote

3DMM you're embedding 3D head shape into several low-dimensional PCA spaces so you lose the refined details necessary. If you use a mesh based 3DMM you can not represent personalized facial details due to its limited representation ability. Hair, eyes, and teeth are also huge problems to resolve. The approach that's most convincing is volumetric based representation texturing on top of some mesh structure or canonical face model. The problem with Nerf/volumetric representations is either the amount of data required, the number of angles required for Nerf, or the time it takes to train/inference speed. There's been good work specifically in https://arxiv.org/abs/2210.06108 where inference takes 20 seconds (still a much longer time than i would like), but it's improving and highly accurate especially for expression transfer

3

parabellum630 t1_itmm0ei wrote

I worked on eyes and upper head shape while the paper I built upon (DECA) tackles personalized fine details. But I do agree hair modeling and finer details and texturing is better with volumetric approaches. Maybe a fusion of the two would be a good research direction? Like the end goal is not only the modeling but also using it for real tasks.

2

trikortreat123 OP t1_itmninz wrote

For avatars to have any functionality in the real world and for the technology to truly be adapted there has to be realism beyond the uncanny valley. You cannot implement a really fast model that accomplishes tasks if you don't have a great sense of depth, emotion, and detail. Speed and functionality is very important but the core problem lies in people not wanting to use a C-grade model which doesn't have any realism. Over the past 2-3 years Nerf training speed has gone down from a few days to just a few minutes with Instant-NGP so the functionality is improving.

1