8 research outputs found
AVFace: Towards Detailed Audio-Visual 4D Face Reconstruction
No full textIn this work, we present a multimodal solution to the problem of 4D face
reconstruction from monocular videos. 3D face reconstruction from 2D images is
an under-constrained problem due to the ambiguity of depth. State-of-the-art
methods try to solve this problem by leveraging visual information from a
single image or video, whereas 3D mesh animation approaches rely more on audio.
However, in most cases (e.g. AR/VR applications), videos include both visual
and speech information. We propose AVFace that incorporates both modalities and
accurately reconstructs the 4D facial and lip motion of any speaker, without
requiring any 3D ground truth for training. A coarse stage estimates the
per-frame parameters of a 3D morphable model, followed by a lip refinement, and
then a fine stage recovers facial geometric details. Due to the temporal audio
and video information captured by transformer-based modules, our method is
robust in cases when either modality is insufficient (e.g. face occlusions).
Extensive qualitative and quantitative evaluation demonstrates the superiority
of our method over the current state-of-the-art.Comment: Accepted by CVPR 2023. Project page:
https://aggelinacha.github.io/AVFace
MIGS: Multi-Identity Gaussian Splatting via Tensor Decomposition
No full textWe introduce MIGS (Multi-Identity Gaussian Splatting), a novel method that learns a single neural representation for multiple identities, using only monocular videos. Recent 3D Gaussian Splatting (3DGS) approaches for human avatars require per-identity optimization. However, learning a multi-identity representation presents advantages in robustly animating humans under arbitrary poses. We propose to construct a high-order tensor that combines all the learnable 3DGS parameters for all the training identities. By assuming a low-rank structure and factorizing the tensor, we model the complex rigid and non-rigid deformations of multiple subjects in a unified network, significantly reducing the total number of parameters. Our proposed approach leverages information from all the training identities and enables robust animation under challenging unseen poses, outperforming existing approaches. It can also be extended to learn unseen identities.Accepted by ECCV 2024. Project page: https://aggelinacha.github.io/MIGS
MI-NeRF: Learning a Single Face NeRF from Multiple Identities
No full textIn this work, we introduce a method that learns a single dynamic neural
radiance field (NeRF) from monocular talking face videos of multiple
identities. NeRFs have shown remarkable results in modeling the 4D dynamics and
appearance of human faces. However, they require per-identity optimization.
Although recent approaches have proposed techniques to reduce the training and
rendering time, increasing the number of identities can be expensive. We
introduce MI-NeRF (multi-identity NeRF), a single unified network that models
complex non-rigid facial motion for multiple identities, using only monocular
videos of arbitrary length. The core premise in our method is to learn the
non-linear interactions between identity and non-identity specific information
with a multiplicative module. By training on multiple videos simultaneously,
MI-NeRF not only reduces the total training time compared to standard
single-identity NeRFs, but also demonstrates robustness in synthesizing novel
expressions for any input identity. We present results for both facial
expression transfer and talking face video synthesis. Our method can be further
personalized for a target identity given only a short video.Comment: Project page: https://aggelinacha.github.io/MI-NeRF
SIDER: Single-Image Neural Optimization for Facial Geometric Detail Recovery
No full text© 2011 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes,creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.We present SIDER (Single-Image neural optimization for facial geometric DEtail Recovery), a novel photometric optimization method that recovers detailed facial geometry from a single image in an unsupervised manner. Inspired by classical techniques of coarse-to-fine optimization and recent advances in implicit neural representations of 3D shape, SIDER combines a geometry prior based on statistical models and Signed Distance Functions (SDFs) to recover facial details from single images. First, it estimates a coarse geometry using a morphable model represented as an SDF. Next, it reconstructs facial geometry details by optimizing a photometric loss with respect to the ground truth image. In contrast to prior work, SIDER does not rely on any dataset priors and does not require additional supervision from multiple views, lighting changes or ground truth 3D shape. Extensive qualitative and quantitative evaluation demonstrates that our method achieves state-of-the-art on facial geometric detail recovery, using only a single in the-wild image.Peer ReviewedPostprint (author's final draft
JEAN: Joint Expression and Audio-guided NeRF-based Talking Face Generation
No full textWe introduce a novel method for joint expression and audio-guided talking face generation. Recent approaches either struggle to preserve the speaker identity or fail to produce faithful facial expressions. To address these challenges, we propose a NeRF-based network. Since we train our network on monocular videos without any ground truth, it is essential to learn disentangled representations for audio and expression. We first learn audio features in a self-supervised manner, given utterances from multiple subjects. By incorporating a contrastive learning technique, we ensure that the learned audio features are aligned to the lip motion and disentangled from the muscle motion of the rest of the face. We then devise a transformer-based architecture that learns expression features, capturing long-range facial expressions and disentangling them from the speech-specific mouth movements. Through quantitative and qualitative evaluation, we demonstrate that our method can synthesize high-fidelity talking face videos, achieving state-of-the-art facial expression transfer along with lip synchronization to unseen audio.Accepted by BMVC 2024. Project Page: https://starc52.github.io/publications/2024-07-19-JEA
Το Βατερλώ δύο γελοίων του Γιάννη Σκαρίμπα: όροι (ανα)σημασιοδότησης της διαφεύγουσας πραγματικότητας
No full textΑντικείμενο της παρούσας μελέτης αποτελεί η ανάδειξη του κομβικού ρόλου που διαδραμάτισε η πορεία συγγραφής του μυθιστορήματος Το Βατερλώ δυο γελοίων, στην εργογραφία του Γιάννη Σκαρίμπα. Ο βασικός άξονας διερεύνησης του ζητήματος, είναι όπως δηλώνεται ήδη από τον τίτλο, η διαρκής (ανα)σημασιοδότηση του κειμένου η οποία, έχει ως εφαλτήριο την αποσπασματικότητα και αποτελεί συνδετικό αρμό στο έργου του συγγραφέα εν συνόλω. Όπως φιλοδοξείται να αναδειχθεί μέσω της περιδιάβασης μας στα κείμενα, το σκαριμπικό έργο, συντελεί μία ενιαία και αδιάσπαστη ολότητα, επιτευχθείσας μέσω μιας πρωτοφανούς εσωτερικής διακειμενικότητας, η οποία επιτρέπει την ομαλή μετάβαση από το ένα έργο στο άλλο˙ από το ένα είδος στο άλλο. Μέσα στο σύνολο του έργου, βλέπουμε φράσεις-εικόνες, ήρωες, καθώς και ολόκληρα επεισόδια (ένθετες ιστορίες) να μεταμοσχεύονται ως αυτούσια και οργανική ύλη σε ένα νέο έργο, δημιουργώντας ως ένα είδος κολάζ, ένα καινούριο κείμενο. Αυτό το κολάζ, επιτρέπει και συνεπικουρεί στη διαρκώς μετέωρη ανασημασιοδότηση του κυρίου νοήματος το οποίο όμως καταλήγει πάντοτε να μετεωρεί. Ακόμη, θίγεται το ζήτημα των ειδολογικών προσμίξεων, μέσα από τη δυαδικότητα των διακειμένων: ενός πλαστού και ενός πραγματικού καθώς και το πως αυτά συνδυαστικά διαμορφώνουν το έργο εν προόδω. Τέλος, διερευνάται η προέκταση μοτιβικών και θεματικών αναφορών και επεισοδίων του μυθιστορήματος Το Βατερλώ δυο γελοίων στα μεταγενέστερα έργα και ειδικότερα: στη νουβέλα Η μαθητευομένη των τακουνιών και τα διηγήματα: Φιγουραζέρ Κυριών και Το Μουστάκι (του κ. Φρανσουά ντε λα Τουςς).The purpose of this study is to highlight the crucial role played the novel “Waterloo's Two Ridiculous Men” in the work of Yannis Skaribas. The main axis of investigating the issue, as already stated in the title, is the constant (re) signification of the text which, as a starting point has the fragmentation of the meaning and is an integral part of the author's work as a whole. As it is aspired to emerge through our browsing through the texts, the work of Skaribas composes a unified and unbroken wholeness, achieved through unprecedented internal intertextuality that permits a smooth transition from one work to another. Throughout his works, we see phrases, images, heroes, as well as whole episodes (inserted stories) transplanted as raw and organic material into a new work, creating as a kind of collage, a new text. This collage allows and assists in the ever-expanding resignification of the main meaning but always ends up being obscured. Moreover, it is raised the issue of generic admixtures, through the duality of the intertexts: a fake and a real one; and how they combine together, in order to shape an eternal work in progress. Finally, we explore the extension of motifs, thematic references, and episodes of the novel “Waterloo's Two Ridiculous Men” in his later works
