45 research outputs found
Your ViT is Secretly an Image Segmentation Model
Get PDFVision Transformers (ViTs) have shown remarkable performance and scalability across various computer vision tasks. To apply single-scale ViTs to image segmentation, existing methods adopt a convolutional adapter to generate multi-scale features, a pixel decoder to fuse these features, and a Transformer decoder that uses the fused features to make predictions. In this paper, we show that the inductive biases introduced by these task-specific components can instead be learned by the ViT itself, given sufficiently large models and extensive pre-training. Based on these findings, we introduce the Encoder-Only Mask Transformer (EoMT), which repurposes the plain ViT architecture to conduct image segmentation. With large-scale models and pre-training, EoMT obtains a segmentation accuracy similar to state-of-the-art models that use task-specific components. At the same time, EoMT is significantly faster than these methods due to its architectural simplicity, e.g., up to 4 × faster with ViT-L. Across a range of model sizes, EoMT demonstrates an optimal balance between segmentation accuracy and prediction speed, suggesting that compute resources are better spent on scaling the ViT itself rather than adding architectural complexity. Code: https://www.tue-mps.org/eomt/
Dealing with uncertainty through prototyping
No full textAs the COVID-19 struck, many organizations faced the challenge to rapidly adapt in order to stay relevant. Over night consumers’ needs changed, leaving the organizations in an uncertain environment. The current situation highlights that organizations need approaches that must enable them engage in search activities, in order to find out how to adapt and deal with uncertainty. This thesis was carried out in the context of a strategy design agency. They help their clients to explore how to adapt an organization by designing strategies and new business models. By a design doing approach they help their clients to engage in search activities with a focus on business model innovation. An initial business idea is considered to be a ‘guess’ that needs to be tested. During the search activities a prototyping process is followed to test these ‘guesses’, referred to as assumptions.The prototyping process is carried out by a multidisciplinary team, consisting of employees of the organization from various departments and business designers from a strategic agency. These (validation) teams aim to use prototyping to reduce uncertainty. By carefully deciding what assumptions needs to be tested, an experiment (activity) is designed to do so. Prototypes in the form of artifacts are build to generate learnings. With the use of these learnings, teams can iterate their business model, value proposition and product designs.Initial observations show, validation teams face difficulty to create suitable prototypes, and setups to test these assumptions while uncertainty is only addressed to a limited extent. Underlying assumptions, which are not explicit at first, are seldom considered. To enhance the current practice, a wide range of prototypes are created which are implemented within the context. The implemented prototypes are aimed to enhance the current process, mainly through a digital manner due to the COVID-19 pandemic. Through numerous of prototypes, new ways were explored with the intention to enable teams to address uncertainty, and moreover, help to create suitable prototypes. These efforts manifested in enabling teams to utilize prototypes as a straw man within the team or use prototypes to trigger customers in an early stage. These activities helped the team to go beyond testing assumptions, and enabling them to reveal assumptions. Through the practice of revealing assumptions, the prototyping process enables to drive continuous learning, since the newly revealed assumption can influence learnings goals for the next steps.The project also identified obstacles that could hinder a successful use of prototyping, such as a risk avoidance mindset and lacking skills of team members.Eventually, based on the developed prototypes and generated learnings, a boundary object in the form of a digital warroom was designed and presented. So the teams are being enabled to go through the prototyping process to address uncertainty. The embodied approach of the warroom should help teams to drive changes by developing new business models, value propositions with product & services supporting these.Strategic Product Desig
Intra-Batch Supervision for Panoptic Segmentation on High-Resolution Images
No full textUnified panoptic segmentation methods are achieving state-of-the-art results on several datasets. To achieve these results on high-resolution datasets, these methods apply crop-based training. In this work, we find that, although crop-based training is advantageous in general, it also has a harmful side-effect. Specifically, it limits the ability of unified networks to discriminate between large object instances, causing them to make predictions that are confused between multiple instances. To solve this, we propose Intra-Batch Supervision (IBS), which improves a network’s ability to discriminate between instances by introducing additional supervision using multiple images from the same batch. We show that, with our IBS, we successfully address the confusion problem and consistently improve the performance of unified networks. For the high-resolution Cityscapes and Mapillary Vistas datasets, we achieve improvements of up to +2.5 on the Panoptic Quality for thing classes, and even more considerable gains of up to +5.8 on both the pixel accuracy and pixel precision, which we identify as better metrics to capture the confusion problem
Encyclopedie van de menselijkheid : begrippen, gebeurtenissen, personen en organisaties uit het werk voor mensenrechten, vrede, veiligheid, asiel, ontwikkeling en milieu /
No full textTask-aligned Part-aware Panoptic Segmentation through Joint Object-Part Representations
Get PDFPart-aware panoptic segmentation (PPS) requires (a) that each foreground
object and background region in an image is segmented and classified, and (b)
that all parts within foreground objects are segmented, classified and linked
to their parent object. Existing methods approach PPS by separately conducting
object-level and part-level segmentation. However, their part-level predictions
are not linked to individual parent objects. Therefore, their learning
objective is not aligned with the PPS task objective, which harms the PPS
performance. To solve this, and make more accurate PPS predictions, we propose
Task-Aligned Part-aware Panoptic Segmentation (TAPPS). This method uses a set
of shared queries to jointly predict (a) object-level segments, and (b) the
part-level segments within those same objects. As a result, TAPPS learns to
predict part-level segments that are linked to individual parent objects,
aligning the learning objective with the task objective, and allowing TAPPS to
leverage joint object-part representations. With experiments, we show that
TAPPS considerably outperforms methods that predict objects and parts
separately, and achieves new state-of-the-art PPS results.Comment: CVPR 2024. Project page and code: https://tue-mps.github.io/tapps
Coordinating Fully-Cooperative Agents Using Hierarchical Learning Anticipation
Get PDFLearning anticipation is a reasoning paradigm in multi-agent reinforcement learning, where agents, during learning, consider the anticipated learning of other agents. There has been substantial research into the role of learning anticipation in improving cooperation among self-interested agents in general-sum games. Two primary examples are Learning with Opponent-Learning Awareness (LOLA), which anticipates and shapes the opponent's learning process to ensure cooperation among self-interested agents in various games such as iterated prisoner's dilemma, and Look-Ahead (LA), which uses learning anticipation to guarantee convergence in games with cyclic behaviors. So far, the effectiveness of applying learning anticipation to fully-cooperative games has not been explored. In this study, we aim to research the influence of learning anticipation on coordination among common-interested agents. We first illustrate that both LOLA and LA, when applied to fully-cooperative games, degrade coordination among agents, causing worst-case outcomes. Subsequently, to overcome this miscoordination behavior, we propose Hierarchical Learning Anticipation (HLA), where agents anticipate the learning of other agents in a hierarchical fashion. Specifically, HLA assigns agents to several hierarchy levels to properly regulate their reasonings. Our theoretical and empirical findings confirm that HLA can significantly improve coordination among common-interested agents in fully-cooperative normal-form games. With HLA, to the best of our knowledge, we are the first to unlock the benefits of learning anticipation for fully-cooperative games
Empirical Generalization Study: Unsupervised Domain Adaptation vs. Domain Generalization Methods for Semantic Segmentation in the Wild
No full textFor autonomous vehicles and mobile robots to safely operate in the real world, i.e., the wild, scene understanding models should perform well in the many different scenarios that can be encountered. In reality, these scenarios are not all represented in the model’s training data, leading to poor performance. To tackle this, current training strategies attempt to either exploit additional unlabeled data with unsupervised domain adaptation (UDA), or to reduce overfitting using the limited available labeled data with domain generalization (DG). However, it is not clear from current literature which of these methods allows for better generalization to unseen data from the wild. Therefore, in this work, we present an evaluation framework in which the generalization capabilities of state-of-the-art UDA and DG methods can be compared fairly. From this evaluation, we find that UDA methods, which leverage unlabeled data, outperform DG methods in terms of generalization, and can deliver similar performance on unseen data as fully-supervised training methods that require all data to be labeled. We show that semantic segmentation performance can be increased up to 30% for a priori unknown data without using any extra labeled data
Task-Aligned Part-Aware Panoptic Segmentation Through Joint Object-Part Representations
Get PDFPart-aware panoptic segmentation (PPS) requires (a) that each foreground object and background region in an image is segmented and classified, and (b) that all parts within foreground objects are segmented, classified and linked to their parent object. Existing methods approach PPS by separately conducting object-level and part-level segmentation. However, their part-level predictions are not linked to individual parent objects. Therefore, their learning objective is not aligned with the PPS task objective, which harms the PPS performance. To solve this, and make more accurate PPS predictions, we propose Task-Aligned Part-aware Panoptic Segmentation (TAPPS). This method uses a set of shared queries to jointly predict (a) object-level segments, and (b) the part-level segments within those same objects. As a result, TAPPS learns to predict part-level segments that are linked to individual parent objects, aligning the learning objective with the task objective, and allowing TAPPS to leverage joint object-part representations. With experiments, we show that TAPPS considerably outperforms methods that predict objects and parts separately, and achieves new state-of-the-art PPS results
Content-aware Token Sharing for Efficient Semantic Segmentation with Vision Transformers
Get PDFThis paper introduces Content-aware Token Sharing (CTS), a token reduction
approach that improves the computational efficiency of semantic segmentation
networks that use Vision Transformers (ViTs). Existing works have proposed
token reduction approaches to improve the efficiency of ViT-based image
classification networks, but these methods are not directly applicable to
semantic segmentation, which we address in this work. We observe that, for
semantic segmentation, multiple image patches can share a token if they contain
the same semantic class, as they contain redundant information. Our approach
leverages this by employing an efficient, class-agnostic policy network that
predicts if image patches contain the same semantic class, and lets them share
a token if they do. With experiments, we explore the critical design choices of
CTS and show its effectiveness on the ADE20K, Pascal Context and Cityscapes
datasets, various ViT backbones, and different segmentation decoders. With
Content-aware Token Sharing, we are able to reduce the number of processed
tokens by up to 44%, without diminishing the segmentation quality.Comment: CVPR 2023. Project page and code: https://tue-mps.github.io/CTS
