15 research outputs found
StudioGAN: A Taxonomy and Benchmark of GANs for Image Synthesis
No full textGenerative Adversarial Network (GAN) is one of the state-of-the-art
generative models for realistic image synthesis. While training and evaluating
GAN becomes increasingly important, the current GAN research ecosystem does not
provide reliable benchmarks for which the evaluation is conducted consistently
and fairly. Furthermore, because there are few validated GAN implementations,
researchers devote considerable time to reproducing baselines. We study the
taxonomy of GAN approaches and present a new open-source library named
StudioGAN. StudioGAN supports 7 GAN architectures, 9 conditioning methods, 4
adversarial losses, 12 regularization modules, 3 differentiable augmentations,
7 evaluation metrics, and 5 evaluation backbones. With our training and
evaluation protocol, we present a large-scale benchmark using various datasets
(CIFAR10, ImageNet, AFHQv2, FFHQ, and Baby/Papa/Granpa-ImageNet) and 3
different evaluation backbones (InceptionV3, SwAV, and Swin Transformer).
Unlike other benchmarks used in the GAN community, we train representative
GANs, including BigGAN and StyleGAN series in a unified training pipeline and
quantify generation performance with 7 evaluation metrics. The benchmark
evaluates other cutting-edge generative models (e.g., StyleGAN-XL, ADM,
MaskGIT, and RQ-Transformer). StudioGAN provides GAN implementations, training,
and evaluation scripts with the pre-trained weights. StudioGAN is available at
https://github.com/POSTECH-CVLab/PyTorch-StudioGAN.Comment: 32 pages, IEEE Transactions on Pattern Analysis and Machine
Intelligence (TPAMI, 2023
StudioGAN: A Taxonomy and Benchmark of GANs for Image Synthesis
No full textGenerative Adversarial Network (GAN) is one of the state-of-the-art generative models for realistic image synthesis. While training and evaluating GAN becomes increasingly important, the current GAN research ecosystem does not provide reliable benchmarks for which the evaluation is conducted consistently and fairly. Furthermore, because there are few validated GAN implementations, researchers devote considerable time to reproducing baselines. We study the taxonomy of GAN approaches and present a new open-source library named StudioGAN. StudioGAN supports 7 GAN architectures, 9 conditioning methods, 4 adversarial losses, 12 regularization modules, 3 differentiable augmentations, 7 evaluation metrics, and 5 evaluation backbones. With our training and evaluation protocol, we present a large-scale benchmark using various datasets (CIFAR10, ImageNet, AFHQv2, FFHQ, and Baby/Papa/Granpa-ImageNet) and 3 different evaluation backbones (InceptionV3, SwAV, and Swin Transformer). Unlike other benchmarks used in the GAN community, we train representative GANs, including BigGAN and StyleGAN series in a unified training pipeline and quantify generation performance with 7 evaluation metrics. The benchmark evaluates other cutting-edge generative models (e.g., StyleGAN-XL, ADM, MaskGIT, and RQ-Transformer). StudioGAN provides GAN implementations, training, and evaluation scripts with the pre-trained weights. StudioGAN is available at https://github.com/POSTECH-CVLab/PyTorch-StudioGAN.Y
Instance-Aware Image Completion
No full textImage completion is a task that aims to fill in the missing region of a
masked image with plausible contents. However, existing image completion
methods tend to fill in the missing region with the surrounding texture instead
of hallucinating a visual instance that is suitable in accordance with the
context of the scene. In this work, we propose a novel image completion model,
dubbed ImComplete, that hallucinates the missing instance that harmonizes well
with - and thus preserves - the original context. ImComplete first adopts a
transformer architecture that considers the visible instances and the location
of the missing region. Then, ImComplete completes the semantic segmentation
masks within the missing region, providing pixel-level semantic and structural
guidance. Finally, the image synthesis blocks generate photo-realistic content.
We perform a comprehensive evaluation of the results in terms of visual quality
(LPIPS and FID) and contextual preservation scores (CLIPscore and object
detection accuracy) with COCO-panoptic and Visual Genome datasets. Experimental
results show the superiority of ImComplete on various natural images.Comment: AI for Content Creation (AI4CC) CVPR workshop, 202
Extending CLIPs Image-Text Alignment to Referring Image Segmentation
No full textReferring Image Segmentation (RIS) is a cross-modal task that aims to segment an instance described by a natural language expression. Recent methods leverage large-scale pretrained unimodal models as backbones along with fusion techniques for joint reasoning across modalities. However, the inherent cross-modal nature of RIS raises questions about the effectiveness of unimodal backbones. We propose RISCLIP, a novel framework that effectively leverages the cross-modal nature of CLIP for RIS. Observing CLIPs inherent alignment between image and text features, we capitalize on this starting point and introduce simple but strong modules that enhance unimodal feature extraction and leverage rich alignment knowledge in CLIPs image-text shared-embedding space. RISCLIP exhibits outstanding results on all three major RIS benchmarks and also outperforms previous CLIP-based methods, demonstrating the efficacy of our strategy in extending CLIPs image-text alignment to RIS.
Learning-Based Anomaly Detection and Monitoring for Swarm Drone Flights
No full textThis paper addresses anomaly detection and monitoring for swarm drone flights. While the current practice of swarm flight typically relies on the operator’s naked eyes to monitor health of the multiple vehicles, this work proposes a machine learning-based framework to enable detection of abnormal behavior of a large number of flying drones on the fly. The method works in two steps: a sequence of two unsupervised learning procedures reduces the dimensionality of the real flight test data and labels them as normal and abnormal cases; then, a deep neural network classifier with one-dimensional convolution layers followed by fully connected multi-layer perceptron extracts the associated features and distinguishes the anomaly from normal conditions. The proposed anomaly detection scheme is validated on the real flight test data, highlighting its capability of online implementation
Extending CLIP's Image-Text Alignment to Referring Image Segmentation
No full textReferring Image Segmentation (RIS) is a cross-modal task that aims to segment
an instance described by a natural language expression. Recent methods leverage
large-scale pretrained unimodal models as backbones along with fusion
techniques for joint reasoning across modalities. However, the inherent
cross-modal nature of RIS raises questions about the effectiveness of unimodal
backbones. We propose RISCLIP, a novel framework that effectively leverages the
cross-modal nature of CLIP for RIS. Observing CLIP's inherent alignment between
image and text features, we capitalize on this starting point and introduce
simple but strong modules that enhance unimodal feature extraction and leverage
rich alignment knowledge in CLIP's image-text shared-embedding space. RISCLIP
exhibits outstanding results on all three major RIS benchmarks and also
outperforms previous CLIP-based methods, demonstrating the efficacy of our
strategy in extending CLIP's image-text alignment to RIS.Comment: NAACL 202
Rebooting ACGAN: Auxiliary Classifier GANs with Stable Training
No full textConditional Generative Adversarial Networks (cGAN) generate realistic images by incorporating class information into GAN. While one of the most popular cGANs is an auxiliary classifier GAN with softmax cross-entropy loss (ACGAN), it is widely known that training ACGAN is challenging as the number of classes in the dataset increases. ACGAN also tends to generate easily classifiable samples with a lack of diversity. In this paper, we introduce two cures for ACGAN. First, we identify that gradient exploding in the classifier can cause an undesirable collapse in early training, and projecting input vectors onto a unit hypersphere can resolve the problem. Second, we propose the Data-to-Data Cross-Entropy loss (D2D-CE) to exploit relational information in the class-labeled dataset. On this foundation, we propose the Rebooted Auxiliary Classifier Generative Adversarial Network (ReACGAN). The experimental results show that ReACGAN achieves state-of-the-art generation results on CIFAR10, Tiny-ImageNet, CUB200, and ImageNet datasets. We also verify that ReACGAN benefits from differentiable augmentations and that D2D-CE harmonizes with StyleGAN2 architecture. Model weights and a software package that provides implementations of representative cGANs and all experiments in our paper are available at https://github.com/POSTECH-CVLab/PyTorch-StudioGAN.N
Scaling up GANs for Text-to-Image Synthesis
No full textThe recent success of text-to-image synthesis has taken the world by storm
and captured the general public's imagination. From a technical standpoint, it
also marked a drastic change in the favored architecture to design generative
image models. GANs used to be the de facto choice, with techniques like
StyleGAN. With DALL-E 2, auto-regressive and diffusion models became the new
standard for large-scale generative models overnight. This rapid shift raises a
fundamental question: can we scale up GANs to benefit from large datasets like
LAION? We find that na\"Ively increasing the capacity of the StyleGAN
architecture quickly becomes unstable. We introduce GigaGAN, a new GAN
architecture that far exceeds this limit, demonstrating GANs as a viable option
for text-to-image synthesis. GigaGAN offers three major advantages. First, it
is orders of magnitude faster at inference time, taking only 0.13 seconds to
synthesize a 512px image. Second, it can synthesize high-resolution images, for
example, 16-megapixel pixels in 3.66 seconds. Finally, GigaGAN supports various
latent space editing applications such as latent interpolation, style mixing,
and vector arithmetic operations.Comment: CVPR 2023. Project webpage at https://mingukkang.github.io/GigaGAN
