31 research outputs found
Neural Architecture Search Survey: A Hardware Perspective
No full textWe review the problem of automating hardware-aware architectural design process of Deep Neural Networks (DNNs). The field of Convolutional Neural Network (CNN) algorithm design has led to advancements in many fields, such as computer vision, virtual reality, and autonomous driving. The end-to-end design process of a CNN is a challenging and time-consuming task, as it requires expertise in multiple areas such as signal and image processing, neural networks, and optimization. At the same time, several hardware platforms, general- and special-purpose, have equally contributed to the training and deployment of these complex networks in a different setting. Hardware-Aware Neural Architecture Search (HW-NAS) automates the architectural design process of DNNs to alleviate human effort and generate efficient models accomplishing acceptable accuracy-performance tradeoffs. The goal of this article is to provide insights and understanding of HW-NAS techniques for various hardware platforms (MCU, CPU, GPU, ASIC, FPGA, ReRAM, DSP, and VPU), followed by the co-search methodologies of neural algorithm and hardware accelerator specifications.This article is published as Chitty-Venkata, Krishna Teja, and Arun K. Somani. "Neural architecture search survey: A hardware perspective." ACM Computing Surveys 55, no. 4 (2022): 1-36.
DOI: 10.1145/3524500
Copyright 2022 Copyright held by the owner/author(s).
Attribution 4.0 International (CC BY 4.0).
Posted with permission
Neural Architecture Search Benchmarks: Insights and Survey
No full textNeural Architecture Search (NAS), a promising and fast-moving research field, aims to automate the architectural design of Deep Neural Networks (DNNs) to achieve better performance on the given task and dataset. NAS methods have been very successful in discovering efficient models for various Computer Vision, Natural Language Processing, etc. The major obstacles to the advancement of NAS techniques are the demand for large computation resources and fair evaluation of various search methods. The differences in training pipeline and setting make it challenging to compare the efficiency of two NAS algorithms. A large number of NAS Benchmarks to simulate the architecture evaluation in seconds have been released over the last few years to ease the computation burden of training neural networks and can aid in the unbiased assessment of different search methods. This paper provides an extensive review of several publicly available NAS Benchmarks in the literature. We provide technical details and a deeper understanding of each benchmark and point out future directions.This article is published as Chitty-Venkata, Krishna Teja, Murali Emani, Venkatram Vishwanath, and Arun K. Somani. "Neural Architecture Search Benchmarks: Insights and Survey." IEEE Access 11 (2023): 25217 - 25236.
DOI: 10.1109/ACCESS.2023.3253818.
Copyright 2023 The Author(s).
Attribution 4.0 International (CC BY 4.0).
Posted with permission
Efficient Design Space Exploration for Sparse Mixed Precision Neural Architectures
No full textPruning and Quantization are two effective Deep Neural Network (DNN) compression methods for efficient inference on various hardware platforms. Pruning refers to removing unimportant weights or nodes, whereas Quantization converts the floating-point parameters to low-bit fixed integer representation. The pruned and low precision models result in smaller and faster inference models on hardware platforms with almost the same accuracy as the unoptimized network. Tensor Cores in Nvidia Ampere 100 (A100) GPU supports (1) 2:4 fine-grained sparse pruning where 2 out of every 4 elements are pruned, and (2) traditional dense multiplication to achieve a good accuracy and performance trade-off. The A100 Tensor Core also takes advantage of 1-bit, 4-bit, and 8-bit multiplication to speed up the inference of a model. Hence, finding the right matrix type (dense or 2:4 sparse) along with the precision for each layer becomes a combinatorial problem. Neural Architecture Search (NAS) can alleviate such problems by automating the architecture design process instead of a brute-force search. In this paper, we propose (i) Mixed Sparse and Precision Search (MSPS), a NAS framework to search for efficient sparse and mixed-precision quantized model within the predefined search space and fixed backbone neural network (Eg. ResNet50), and (ii) Architecture, Sparse and Precision Search (ASPS) to jointly search for kernel size and number of filters, and sparse-precision combination of each layer. We illustrate the effectiveness of our methods targeting A100 Tensor Core on Nvidia GPUs by searching efficient sparse-mixed precision networks on ResNet50 and achieving better accuracy-latency trade-off models compared to the manually designed Uniform Sparse Int8 networks.This proceeding is published as Chitty-Venkata, Krishna Teja, Murali Emani, Venkatram Vishwanath, and Arun K. Somani. "Efficient Design Space Exploration for Sparse Mixed Precision Neural Architectures." In Proceedings of the 31st International Symposium on High-Performance Parallel and Distributed Computing, pp. 265-276. 2022.
DOI: 10.1145/3502181.3531463.
Copyright 2022 The Author(s).
Attribution 4.0 International (CC BY 4.0).
Posted with permission
A Survey of Techniques for Optimizing Transformer Inference
No full textRecent years have seen a phenomenal rise in performance and applications of transformer neural networks. The family of transformer networks, including Bidirectional Encoder Representations from Transformer (BERT), Generative Pretrained Transformer (GPT) and Vision Transformer (ViT), have shown their effectiveness across Natural Language Processing (NLP) and Computer Vision (CV) domains. Transformer-based networks such as ChatGPT have impacted the lives of common men. However, the quest for high predictive performance has led to an exponential increase in transformers' memory and compute footprint. Researchers have proposed techniques to optimize transformer inference at all levels of abstraction. This paper presents a comprehensive survey of techniques for optimizing the inference phase of transformer networks. We survey techniques such as knowledge distillation, pruning, quantization, neural architecture search and lightweight network design at the algorithmic level. We further review hardware-level optimization techniques and the design of novel hardware accelerators for transformers. We summarize the quantitative results on the number of parameters/FLOPs and accuracy of several models/techniques to showcase the tradeoff exercised by them. We also outline future directions in this rapidly evolving field of research. We believe that this survey will educate both novice and seasoned researchers and also spark a plethora of research efforts in this field.This preprint of Chitty-Venkata, K.T., Mittal, S., Emani, M., Vishwanath, V., Somani, A.K., A Survey of Techniques for Optimizing Transformer Inference. Is available at ArXiv (arXiv:2307.07982) https://doi.org/10.48550/arXiv.2307.07982. Posted with permission. CC BY-NC-ND 4.0 DEED Attribution-NonCommercial-NoDerivs 4.0 International. Published as Chitty-Venkata, Krishna Teja, Sparsh Mittal, Murali Emani, Venkatram Vishwanath, and Arun K. Somani. "A survey of techniques for optimizing transformer inference." Journal of Systems Architecture (2023): 102990.
doi: https://doi.org/10.1016/j.sysarc.2023.10299
Neural Architecture Search for Transformers: A Survey
No full textTransformer-based Deep Neural Network architectures have gained tremendous interest due to their effectiveness in various applications across Natural Language Processing (NLP) and Computer Vision (CV) domains. These models are the de facto choice in several language tasks, such as Sentiment Analysis and Text Summarization, replacing Long Short Term Memory (LSTM) model. Vision Transformers (ViTs) have shown better model performance than traditional Convolutional Neural Networks (CNNs) in vision applications while requiring significantly fewer parameters and training time. The design pipeline of a neural architecture for a given task and dataset is extremely challenging as it requires expertise in several interdisciplinary areas such as signal processing, image processing, optimization and allied fields. Neural Architecture Search (NAS) is a promising technique to automate the architectural design process of a Neural Network in a data-driven way using Machine Learning (ML) methods. The search method explores several architectures without requiring significant human effort, and the searched models outperform the manually built networks. In this paper, we review Neural Architecture Search techniques, targeting the Transformer model and its family of architectures such as Bidirectional Encoder Representations from Transformers (BERT) and Vision Transformers. We provide an in-depth literature review of approximately 50 state-of-the-art Neural Architecture Search methods and explore future directions in this fast-evolving class of problems.This article is published as Chitty-Venkata, Krishna Teja, Murali Emani, Venkatram Vishwanath, and Arun K. Somani. "Neural Architecture Search for Transformers: A Survey." IEEE Access 10 (2022): 108374-108412.
DOI: 10.1109/ACCESS.2022.3212767.
Copyright 2022 IEEE. Attribution 4.0 International (CC BY 4.0).
Posted with permission
Searching Architecture and Precision for U-net based Image Restoration Tasks
No full textManually architecting Deep Neural Networks (DNNs) has led to the success of Deep Learning in many domains. However, recent DNNs designed using Neural Architecture Search (NAS) have exceeded manually designed architectures and have significantly reduced the human effort to develop complex networks. Current works use NAS to identify a cell architecture constrained by a fixed order of operations that is then replicated throughout the network. The constraints potentially limit the effectiveness of NAS in converging on a more efficient DNN architecture. In the first part of our paper, we propose “Operation Search,” a search on an enlarged topological space for U-net and its variants that retain efficiency. The idea is to allow for custom cells (operations and their sequence) at various levels of the network to maximize image quality while being sensitive to computation cost. In the second part of our paper, we propose custom quantization at various levels resulting in a mixed-precision network. Additionally, we increase the search efficiency by constraining the search space to use the same precision for both weights and activations at any level. This does not result in computational inefficiency because it matches the operand precisions supported by Tensor Core enabled GPUs.This is a manuscript of an article published as Chitty-Venkata, Krishna Teja, Arun K. Somani, and Sreenivas Kothandaraman. "Searching Architecture and Precision for U-net based Image Restoration Tasks." In 2021 IEEE International Conference on Image Processing (ICIP), pp. 1989-1993. IEEE, 2021. DOI: 10.1109/ICIP42928.2021.9506777. Posted with permission.</p
Array-Aware Neural Architecture Search
No full textConvolutional Neural Networks (CNNs) have exceeded human accuracy in many Computer Vision tasks, such as Image Classification, Object Detection, Image Segmentation, etc. This advancement is due to the efficient manual design of CNNs in initially, followed by automated design through Neural Architecture Search (NAS). In parallel to neural network design, advances in Accelerator hardware design, such as Google’s Tensor Processing Unit (TPU), Eyeriss, etc., also occurred for efficient processing of CNN forward propagation. The heart of these accelerators is an array processor (Systolic Array) of a fixed dimension, that limits the amount of CNN computation that can be carried out in a single clock cycle. While NAS is able to produce efficient neural architectures, the networks need to be co-designed with respect to the underlying array dimensions to obtain the best performance. In this paper, we introduce "Array Aware Neural Architecture Search" to automatically design efficient CNNs for a fixed array-based neural network accelerator. Previous Hardware Aware NAS methods consider a fixed search space for different hardware platforms and search within its predefined space. We explore the search space based on the underlying hardware array dimensions to design a more efficient CNN architectures for optimal performance. We observe that our proposed NAS methods on the CIFAR-10 dataset produce similar accuracy as the baseline network while saving a substantial number of cycles on the Array.This is a manuscript of a proceeding published as Chitty-Venkata, Krishna Teja, and Arun K. Somani. "Array-Aware Neural Architecture Search." In 2021 IEEE 32nd International Conference on Application-Specific Systems, Architectures and Processors (ASAP), pp. 125-132. IEEE, 2021. DOI: 10.1109/ASAP52443.2021.00026. Posted with permission.</p
Differentiable Neural Architecture, Mixed Precision and Accelerator Co-Search
No full textQuantization, effective Neural Network architecture, and efficient accelerator hardware are three important design paradigms to maximize accuracy and efficiency. Mixed Precision Quantization is a process of assigning different precision to different Neural Network layers for optimized inference. Neural Architecture Search (NAS) is a process of automatically designing the neural network for a task and can also be extended to search for the precision of each weight and activation matrix. In this paper, we develop the following three methods: (i) Fast Differentiable Hardware-aware Mixed Precision Quantization Search method to find optimal precision, (ii) Joint Differentiable hardware-aware Architecture and Mixed Precision Quantization Co-search, (iii) Joint Accelerator, Architecture, and Precision triple co-search to find best possibilities in all the three worlds. We demonstrate the effectiveness of our proposed methods targeting Bitfusion accelerator by searching mixed precision models on MobilenetV2. We achieve better accuracy-latency trade-off models than the manually designed and previously proposed search methods.This article is published as Chitty-Venkata, Krishna Teja, Yiming Bian, Murali Emani, Venkatram Vishwanath, and Arun K. Somani. "Differentiable Neural Architecture, Mixed Precision and Accelerator Co-search." IEEE Access 11 (2023):106670-106687. doi: https://doi.org/10.1109/ACCESS.2023.3320133. This is an open access article distributed under a Creative Commons Attribution 4.0 License
Array Aware Training/Pruning: Methods for Efficient Forward Propagation on Array-based Neural Network Accelerators
No full textDue to the increase in the use of large-sized Deep Neural Networks (DNNs) over the years, specialized hardware accelerators such as Tensor Processing Unit and Eyeriss have been developed to accelerate the forward pass of the network. The essential component of these devices is an array processor which is composed of multiple individual compute units for efficiently executing Multiplication and Accumulation (MAC) operation. As the size of this array limits the amount of DNN processing of a single layer, the computation is performed in several batches serially leading to extra compute cycles along both the axes. In practice, due to the mismatch between matrix and array sizes, the computation does not map on the array exactly. In this work, we address the issue of minimizing processing cycles on the array by adjusting the DNN model parameters by using a structured hardware array dependent optimization. We introduce two techniques in this paper: Array Aware Training (AAT) for efficient training and Array Aware Pruning (AAP) for efficient inference. Weight pruning is an approach to remove redundant parameters in the network to decrease the size of the network. The key idea behind pruning in this paper is to adjust the model parameters (the weight matrix) so that the array is fully utilized in each computation batch. Our goal is to compress the model based on the size of the array so as to reduce the number of computation cycles. We observe that both the proposed techniques results into similar accuracy as the original network while saving a significant number of processing cycles (75%).This is a manuscript of a proceeding published as Chitty-Venkata, Krishna Teja, and Arun K. Somani. "Array Aware Training/Pruning: Methods for Efficient Forward Propagation on Array-based Neural Network Accelerators." In 2020 IEEE 31st International Conference on Application-specific Systems, Architectures and Processors (ASAP) (2020): 37-44. DOI: 10.1109/ASAP49362.2020.00016. Posted with permission.</p
ConVision Benchmark: A Contemporary Framework to Benchmark CNN and ViT Models
No full textConvolutional Neural Networks (CNNs) and Vision Transformers (ViTs) have shown remarkable performance in computer vision tasks, including object detection and image recognition. These models have evolved significantly in architecture, efficiency, and versatility. Concurrently, deep-learning frameworks have diversified, with versions that often complicate reproducibility and unified benchmarking. We propose ConVision Benchmark, a comprehensive framework in PyTorch, to standardize the implementation and evaluation of state-of-the-art CNN and ViT models. This framework addresses common challenges such as version mismatches and inconsistent validation metrics. As a proof of concept, we performed an extensive benchmark analysis on a COVID-19 dataset, encompassing nearly 200 CNN and ViT models in which DenseNet-161 and MaxViT-Tiny achieved exceptional accuracy with a peak performance of around 95%. Although we primarily used the COVID-19 dataset for image classification, the framework is adaptable to a variety of datasets, enhancing its applicability across different domains. Our methodology includes rigorous performance evaluations, highlighting metrics such as accuracy, precision, recall, F1 score, and computational efficiency (FLOPs, MACs, CPU, and GPU latency). The ConVision Benchmark facilitates a comprehensive understanding of model efficacy, aiding researchers in deploying high-performance models for diverse applications.This article is published as Bangalore Vijayakumar, Shreyas, Krishna Teja Chitty-Venkata, Kanishk Arya, and Arun K. Somani. "ConVision Benchmark: A Contemporary Framework to Benchmark CNN and ViT Models." AI 5, no. 3 (2024): 1132-1171. doi: https://doi.org/10.3390/ai5030056. Copyright: © 2024 by the authors.This article is an open access article
distributed under the terms and conditions of the Creative CommonsAttribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/)
