CORE
COnnecting
REpositories

    6 research outputs found

    OAC: Output-adaptive Calibration for Accurate Post-training Quantization

    Author
    Publication venue
    Publication date
    Field of study
    No full text
    Deployment of Large Language Models (LLMs) has major computational costs, due to their rapidly expanding size. Compression of LLMs reduces the memory footprint, latency, and energy required for their inference. Post-training Quantization (PTQ) techniques have been developed to compress LLMs while avoiding expensive re-training. Most PTQ approaches formulate the quantization error based on a layer-wise Euclidean loss, ignoring the model output. Then, each layer is calibrated using its layer-wise Hessian to update the weights towards minimizing the quantization error. The Hessian is also used for detecting the most salient weights to quantization. Such PTQ approaches are prone to accuracy drop in low-precision quantization. We propose Output-adaptive Calibration (OAC) to incorporate the model output in the calibration process. We formulate the quantization error based on the distortion of the output cross-entropy loss. OAC approximates the output-adaptive Hessian for each layer under reasonable assumptions to reduce the computational complexity. The output-adaptive Hessians are used to update the weight matrices and detect the salient weights towards maintaining the model output. Our proposed method outperforms the state-of-the-art baselines such as SpQR and BiLLM, especially, at extreme low-precision (2-bit and binary) quantization
    Association for the Advancement of Artificial Intelligence: AAAI Publications

    Mitigating Outlier Activations in Low-Precision Fine-Tuning of Language Models

    Author
    Publication venue
    Publication date
    Field of study
    No full text
    Low-precision fine-tuning of language models has gained prominence as a cost-effective and energy-efficient approach to deploying large-scale models in various applications. However, this approach is susceptible to the existence of outlier values in activation. The outlier values in the activation can negatively affect the performance of fine-tuning language models in the low-precision regime since they affect the scaling factor and thus make representing smaller values harder. This paper investigates techniques for mitigating outlier activation in low-precision integer fine-tuning of the language models. Our proposed novel approach enables us to represent the outlier activation values in 8-bit integers instead of floating-point (FP16) values. The benefit of using integers for outlier values is that it enables us to use operator tiling to avoid performing 16-bit integer matrix multiplication to address this problem effectively. We provide theoretical analysis and supporting experiments to demonstrate the effectiveness of our approach in improving the robustness and performance of low-precision fine-tuned language models
    arXiv.org e-Print Archive

    Tiny Noise-Robust Voice Activity Detector for Voice Assistants

    Author
    Publication venue
    Publication date
    Field of study
    No full text
    PolyPublie

    Zeroth order optimization for pretraining language models

    Author
    Publication venue
    Publication date
    Field of study
    No full text
    ABSTRACT: The physical memory for training Large Language Models (LLMs) grow with the model size, and are limited to the GPU memory. In particular, back-propagation that requires the computation of the first-order derivatives adds to this memory overhead. Training extremely large language models with memory-efficient algorithms is still a challenge with theoretical and practical implications. Back-propagation-free training algorithms, also known as zeroth-order methods, are recently examined to address this challenge. Their usefulness has been proven in fine-tuning of language models. However, so far, there has been no study for language model pretraining using zeroth-order optimization, where the memory constraint is manifested more severely. We build the connection between the second order, the first order, and the zeroth order theoretically. Then, we apply the zeroth order optimization to pre-training light-weight language models, and discuss why they cannot be readily applied. We show in p articular that the curse of dimensionality is the main obstacle, and pave the way towards modifications of zeroth order methods for pre-training such models
    PolyPublie

    Author Correction: Mapping local patterns of childhood overweight and wasting in low- and middle-income countries between 2000 and 2017

    Author
    Publication venue
    Publication date
    Field of study
    No full text
    Crossref

    Author Correction: Mapping local patterns of childhood overweight and wasting in low- and middle-income countries between 2000 and 2017 (Nature Medicine, (2020), 26, 5, (750-759), 10.1038/s41591-020-0807-6)

    Author
    Publication venue
    Publication date
    Field of study
    No full text
    An amendment to this paper has been published and can be accessed via a link at the top of the paper
    DRO Deakin Research Online
    corecore