65 research outputs found
PD2T: Person-specific Detection, Deformable Tracking
No full textFace detection/alignment has reached a satisfactory state in static images captured under arbitrary conditions. Such methods typically perform (joint) fitting independently for each frame and are used in commercial applications; however in the majority of the real-world scenarios the dynamic scenes are of interest. Hence, we argue that generic fitting per frame is suboptimal (it discards the informative correlation of sequential frames) and propose to learn person-specific statistics from the video to improve the generic results. To that end, we introduce a meticulously studied pipeline, which we name PD\textsuperscript{2}T, that performs person-specific detection and landmark localisation. We carry out extensive experimentation with a diverse set of i) generic fitting results, ii) different objects (human faces, animal faces) that illustrate the powerful properties of our proposed pipeline and experimentally verify that PD\textsuperscript{2}T outperforms all the compared methods
Deep face deblurring
No full textBlind deblurring consists a long studied task, however the outcomes of generic methods are not effective in real world blurred images. Domain-specific methods for deblurring targeted object categories, e.g. text or faces, frequently outperform their generic counterparts, hence they are attracting an increasing amount of attention. In this work, we develop such a domain-specific method to tackle deblurring of human faces, henceforth referred to as face deblurring. Studying faces is of tremendous significance in computer vision, however face deblurring has yet to demonstrate some convincing results. This can be partly attributed to the combination of i) poor texture and ii) highly structure shape that yield the contour/gradient priors (that are typically used) sub-optimal. In our work instead of making assumptions over the prior, we adopt a learning approach by inserting weak supervision that exploits the well-documented structure of the face. Namely, we utilise a deep network to perform the deblurring and employ a face alignment technique to pre-process each face. We additionally surpass the requirement of the deep network for thousands training samples, by introducing an efficient framework that allows the generation of a large dataset. We utilised this framework to create 2MF2, a dataset of over two million frames. We conducted experiments with real world blurred facial images and report that our method returns a result close to the sharp natural latent image
Sound and complete verification of Polynomial Networks
No full textLas Redes Polinomiales (PNs) han demostrado un rendimiento prometedor en reconocimiento de imágenes y caras recientemente. No obstante, no queda clara la robustez de estas y por lo tanto, certificar la robustez es imperativo para posibilitar su implantación en aplicaciones en el mundo real. Los algoritmos de verificación existentes para Redes Neuronales (NNs) con la activación ReLU, basados en técnicas "branch and bound" (BaB), no pueden ser aplicados trivialmente para a verificación de las PNs. En este trabajo, ideamos un nuevo método de acotado (bounding), equipado con BaB para garantizar la convergencia global, llamado VPN. Una observación clave es que obtenemos cotas mucho más ajustadas que la referencia dada con propagación de intervalos. Esto permite la verificación solida y completa de las PNs con validación empírica en los conjuntos de datos MNIST, CIFAR10 y STL10. Creemos que nuestro método tiene su propio interés en la verificatión de NNs
Sound and complete verification of Polynomial Networks
No full textLas Redes Polinomiales (PNs) han demostrado un rendimiento prometedor en reconocimiento de imágenes y caras recientemente. No obstante, no queda clara la robustez de estas y por lo tanto, certificar la robustez es imperativo para posibilitar su implantación en aplicaciones en el mundo real. Los algoritmos de verificación existentes para Redes Neuronales (NNs) con la activación ReLU, basados en técnicas "branch and bound" (BaB), no pueden ser aplicados trivialmente para a verificación de las PNs. En este trabajo, ideamos un nuevo método de acotado (bounding), equipado con BaB para garantizar la convergencia global, llamado VPN. Una observación clave es que obtenemos cotas mucho más ajustadas que la referencia dada con propagación de intervalos. Esto permite la verificación solida y completa de las PNs con validación empírica en los conjuntos de datos MNIST, CIFAR10 y STL10. Creemos que nuestro método tiene su propio interés en la verificatión de NNs
Certifying Robustness in NLP Classifiers via Lipschitz Constraints
No full textThis thesis presents novel extensions of certifiable robustness techniques for characterlevel text classifiers under edit-based perturbations, specifically leveraging the ERP distance. Building upon LipsLev, a recent framework introducing deterministic robustness certificates for convolutional text classifiers, we further refine and strengthen these results by adapting algebraic Lipschitz constraints inspired by recent developments in certified robustness and residual architectures. First, we adapt Spectral SDP-based Lipschitz Layers (SLL), originally formulated for Euclidean metrics, introducing convolutional architectures that guarantee 1-Lipschitz continuity via two mappings: scaled linear projections and residual compositions. Second, we propose a novel adaptation of the Cholesky-Orthogonalized Residual Dense (CHORD) architecture, originally developed for image-based perturbations. We integrate convolutional residual blocks by maintaining global Lipschitz constraints. Experimental evaluations confirm that both approaches yield classifiers with certified robustness and demonstrate competitive performance against character-level adversarial perturbations
RoCGAN: robust conditional GAN
No full textConditional image generation lies at the heart of computer vision and conditional generative adversarial networks (cGAN) have recently become the method of choice for this task, owing to their superior performance. The focus so far has largely been on performance improvement, with little effort in making cGANs more robust to noise. However, the regression (of the generator) might lead to arbitrarily large errors in the output, which makes cGANs unreliable for real-world applications. In this work, we introduce a novel conditional GAN model, called RoCGAN, which leverages structure in the target space of the model to address the issue. Specifically, we augment the generator with an unsupervised pathway, which promotes the outputs of the generator to span the target manifold, even in the presence of intense noise. We prove that RoCGAN share similar theoretical properties as GAN and establish with both synthetic and real data the merits of our model. We perform a thorough experimental validation on large scale datasets for natural scenes and faces and observe that our model outperforms existing cGAN architectures by a large margin. We also empirically demonstrate the performance of our approach in the face of two types of noise (adversarial and Bernoulli)
MIGS: Multi-Identity Gaussian Splatting via Tensor Decomposition
No full textWe introduce MIGS (Multi-Identity Gaussian Splatting), a novel method that learns a single neural representation for multiple identities, using only monocular videos. Recent 3D Gaussian Splatting (3DGS) approaches for human avatars require per-identity optimization. However, learning a multi-identity representation presents advantages in robustly animating humans under arbitrary poses. We propose to construct a high-order tensor that combines all the learnable 3DGS parameters for all the training identities. By assuming a low-rank structure and factorizing the tensor, we model the complex rigid and non-rigid deformations of multiple subjects in a unified network, significantly reducing the total number of parameters. Our proposed approach leverages information from all the training identities and enables robust animation under challenging unseen poses, outperforming existing approaches. It can also be extended to learn unseen identities.Accepted by ECCV 2024. Project page: https://aggelinacha.github.io/MIGS
MI-NeRF: Learning a Single Face NeRF from Multiple Identities
No full textIn this work, we introduce a method that learns a single dynamic neural
radiance field (NeRF) from monocular talking face videos of multiple
identities. NeRFs have shown remarkable results in modeling the 4D dynamics and
appearance of human faces. However, they require per-identity optimization.
Although recent approaches have proposed techniques to reduce the training and
rendering time, increasing the number of identities can be expensive. We
introduce MI-NeRF (multi-identity NeRF), a single unified network that models
complex non-rigid facial motion for multiple identities, using only monocular
videos of arbitrary length. The core premise in our method is to learn the
non-linear interactions between identity and non-identity specific information
with a multiplicative module. By training on multiple videos simultaneously,
MI-NeRF not only reduces the total training time compared to standard
single-identity NeRFs, but also demonstrates robustness in synthesizing novel
expressions for any input identity. We present results for both facial
expression transfer and talking face video synthesis. Our method can be further
personalized for a target identity given only a short video.Comment: Project page: https://aggelinacha.github.io/MI-NeRF
- …
