1,721,087 research outputs found
Runtime Algorithm and Hardware Management for Efficient DNN Inference on Mobile/Embedded Platforms
No full textDeep neural network (DNN) inference is increasingly being executed on mobile and embedded platforms due to enhanced privacy, reduced latency, and improved energy efficiency. Efficient DNN deployment on these platforms is challenging due to limited computing resources. Although many static DNN model compression approaches have been proposed, they rely on prior knowledge of application performance requirements and hardware resource availability to determine the compression ratio. However, because both of these factors vary at runtime, statically compressed models cannot maintain consistent performance.Prior work has addressed this issue through algorithmic approaches (e.g., DNN model switching or dynamic DNNs) or runtime hardware resource management. However, there is limited literature on integrating the advantages of both algorithms and hardware at runtime. In this thesis, we investigate runtime DNN algorithm and hardware management, and develop a runtime system to optimise DNN performance as well as power and energy efficiency by leveraging the trade off opportunities from both algorithms and hardware platforms.First, our study finds that earlier dynamic DNN models suffer from significant memory overhead, limited runtime model compression ratio, and a narrow range of dynamic performance trade-offs. To address these issues, we propose a dynamic DNN approach that uses incremental training and group convolution pruning. In this approach, the channels of each DNN convolutional layer are divided into groups, which are then trained incrementally. At runtime, these pre-trained groups can be pruned to reduce latency and energy consumption or added back for accuracy recovery, all in real time and without the need for any retraining. At the same compression ratio, our proposed dynamic DNN model achieves a 2.4× reduction in memory footprint compared to prior work. In addition, we combine dynamic voltage and frequency scaling (DVFS) and task mapping with the model, enabling fine-grained and wide-ranging dynamic performance trade-offs. Next, we identify three common issues with all existing dynamic DNN approaches: (1) significant training time, (2) incompatibility with state-of-the-art Neural Architecture Search (NAS) deployment pipeline, and (3) suboptimal inference on heterogeneous hardware platforms. To address these problems, we propose the Dynamic Super-Network, a novel dynamic DNN approach designed specifically for NAS models. Unlike traditional resource-intensive approaches to train dynamic DNN models, this approach pre-samples diverse sub-networks from NAS super-network, eliminating the need for training. By sampling separate sub-network libraries for each type of heterogeneous hardware resource (e.g., CPU and GPU) on modern SoCs, one backbone super-network can efficiently scale across all hardware resources. On an Nvidia Jetson Xavier NX platform using the ImageNet dataset, our approach outperforms state-of-the-art work by achieving up to 3.5× (CPU) and 2.4× (GPU) faster inference at similar Top-1 accuracy, or delivering 3.8% (CPU) and 5.1% (GPU) higher accuracy at similar latency.To explore opportunities in both algorithms and hardware platforms, we propose a hierarchical runtime resource management approach that adjusts dynamic DNN models and DVFS to meet application- and user-level performance requirements (e.g., accuracy and latency) while respecting hardware constraints (e.g., power consumption). Compared with the Linux schedutil governor, our approach achieves a 13.7% reduction in energy consumption and a 6.5% reduction in latency when deploying a single DNN model, and up to a 47.2% reduction in energy consumption and a 19% reduction in latency when deploying two DNN models concurrently
Dynamic DNNs and runtime management for efficient inference on mobile/embedded devices
No full textDeep neural network (DNN) inference is increasingly being executed on mobile and embedded platforms due to several key advantages in latency, privacy and always-on availability. However, due to limited computing resources, efficient DNN deployment on mobile and embedded platforms is challenging. Although many hardware accelerators and static model compression methods were proposed by previous works, at system runtime, multiple applications are typically executed concurrently and compete for hardware resources. This raises two main challenges: Runtime Hardware Availability and Runtime Application Variability. Previous works have addressed these challenges through either dynamic neural networks that contain sub-networks with different performance trade-offs or runtime hardware resource management. In this thesis, we proposed a combined method, a system was developed for DNN performance trade-off management, combining the runtime trade-off opportunities in both algorithms and hardware to meet dynamically changing application performance targets and hardware constraints in real time. We co-designed novel Dynamic Super-Networks to maximise runtime system-level performance and energy efficiency on heterogeneous hardware platforms. Compared with SOTA, our experimental results using ImageNet on the GPU of Jetson Xavier NX show our model is 2.4x faster for similar ImageNet Top-1 accuracy, or 5.1% higher accuracy at similar latency. We also designed a hierarchical runtime resource manager that tunes both dynamic neural networks and DVFS at runtime. Compared with the Linux DVFS governor schedutil, our runtime approach achieves up to a 19% energy reduction and a 9% latency reduction in single model deployment scenario, and an 89% energy reduction and a 23% latency reduction in a two concurrent model deployment scenario
Dynamic DNNs meet runtime resource management for efficient heterogeneous computing
No full textDeep Neural Network (DNN) inference is increasingly being deployed on edge devices, driven by the advantages of lower latency and enhanced privacy. However, the deployment of these models on such platforms poses considerable challenges due to the intensive computation and memory access requirements. While various static model compression techniques have been proposed, they often struggle when adapting to the dynamic computing environments of modern heterogeneous platforms. The two main challenges we focus on in our research are: (1) Dynamic Hardware and Runtime Conditions: Modern edge devices are equipped with heterogeneous computing resources, including CPUs, GPUs, NPUs, and FPGAs. Their availability and performance can change dynamically during runtime, influenced by factors such as device state, power constraints, and thermal conditions. Moreover, DNN models may need to share resources with other applications or models, introducing an additional layer of complexity to the quest for consistent performance and efficiency. (2) Dynamic Application Requirements: The same DNN model can be used in a variety of applications, each with unique and potentially fluctuating performance requirements. In this poster, we will explore the world of dynamic neural networks, with a particular focus on their role in efficient model deployment in dynamic computing environments. Our system leverages runtime trade-offs in both algorithms and hardware to optimize DNN performance and energy efficiency. A cornerstone of our system is the Dynamic-OFA, a dynamic version of the 'once-for-all network', designed to efficiently scale the ConvNet architecture to fit the dynamic application requirements and hardware resources. It exhibits strong generalization across different model architectures, such as Transformer. We will also discuss the benefits of integrating algorithmic techniques with hardware opportunities, including Dynamic Voltage and Frequency Scaling (DVFS) and task mapping. Our experimental results, using ImageNet on a Jetson Xavier NX, reveal that the Dynamic-OFA outperforms state of-the-art Dynamic DNNs, offering up to 3.5x (CPU) and 2.4x (GPU) speed improvements for similar ImageNet Top-1 accuracy, or a 3.8% (CPU) and 5.1% (GPU) increase in accuracy at similar latency
Dataset supporting the conference paper "Power- and Deadline-Aware Dynamic Inference on Intermittent Computing Systems"
No full textThis dataset supports the publication:
"Power- and Deadline-Aware Dynamic Inference on Intermittent Computing Systems" by Hengrui Zhao, Lei Xun, Jagmohan Chauhan and Geoff Merrett
CONFERENCE: Design, Automation and Test in Europe Conference 2025
This dataset includes the experimental results for Figure 1, 6, 7, 8, 9 of the paper.
This dataset contains:
'Fig1.csv': Data supporting Fig. 1. Comparison of accuracy and inference latency of ePerceptron with the baseline.
'Fig6.csv': Data supporting Fig. 6. The operating cycles of the system when running the largest sub-network of DualAdaptNet at different current inputs.
'Fig7.csv': Data supporting Fig. 7. The accuracy and inference latency of DualAdaptNet, ePerceptive, and Zygarde under continuous power supply.
'Fig8.csv': Data supporting Fig. 8. The number of completed and correct inferences performed by the three models (DualAdaptNet, ePerceptive, and Zygarde) under constant current supply.
'Fig9.csv': Data supporting Fig. 9. The number of completed and correct inferences performed by the three models (DualAdaptNet, ePerceptive, and Zygarde) under varying current supply.
</span
Power- and deadline-aware dynamic inference on intermittent computing systems
No full textIn energy-harvesting intermittent computing systems, balancing power constraints with the need for timely and accurate inference remains a critical challenge. Existing methods often sacrifice significant accuracy or fail to adapt effectively to fluctuating power conditions. This paper presents DualAdaptNet, a power- and deadline-aware neural network architecture that dynamically adapts both its width and depth to ensure reliable inference under variable power conditions. Additionally, a runtime scheduling method is introduced to select an appropriate subnetwork configuration based on real-time energy-harvesting conditions and system deadlines. Experimental results on the MNIST dataset demonstrate that our approach completes up to 7.0% more inference tasks within a specified deadline, while also improving average accuracy by 15.4% compared to the state-of-the-art
Dynamic DNNs meet runtime resource management on mobile and embedded platforms
No full textDeep neural network (DNN) inference is increasingly being executed on mobile and embedded platforms due to low latency and better privacy. However, efficient deployment on these platforms is challenging due to the intensive computation and memory access. We propose a holistic system design for DNN performance and energy optimisation, combining the trade-off opportunities in both algorithms and hardware. The system can be viewed as three abstract layers: the device layer contains heterogeneous computing resources; the application layer has multiple concurrent workloads; and the runtime resource management layer monitors the dynamically changing algorithms' performance targets as well as hardware resources and constraints, and tries to meet them by tuning the algorithm and hardware at the same time. Moreover, We illustrate the runtime approach through a dynamic version of 'once-for-all network' (namely Dynamic-OFA), which can scale the ConvNet architecture to fit heterogeneous computing resources efficiently and has good generalisation for different model architectures such as Transformer. Compared to the state-of-the-art Dynamic DNNs, our experimental results using ImageNet on a Jetson Xavier NX show that the Dynamic-OFA is up to 3.5x (CPU), 2.4x (GPU) faster for similar ImageNet Top-1 accuracy, or 3.8% (CPU), 5.1% (GPU) higher accuracy at similar latency. Furthermore, compared with Linux governor (e.g. performance, schedutil), our runtime approach reduces the energy consumption by 16.5% at similar latency
Dataset supporting publication "Realisation of early-exit dynamic neural networks on reconfigurable hardware"
No full textThis dataset supports the publication " Realisation of Early-Exit Dynamic Neural Networks on Reconfigurable Hardware " to be published in the IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems.
This dataset contains:
- 'Fig5a.csv': Data supporting Fig. 5 (a): Experimental results comparing average Execution Time per sample. Average values are calculated based on each exit point trigger rate (ai).
- 'Fig5b.csv': Data supporting Fig. 5 (b): Experimental results comparing average Energy Consumption per sample. Average values are calculated based on each exit point trigger rate (ai).
- 'Fig6a.csv': Data supporting Fig. 6 (a): Comparison of the pipeline and parallel designs over average execution time across early-exit LeNet-5, AlexNet, VGG19 and ResNet32.
- 'Fig6b.csv': Data supporting Fig. 6 (b): Comparison of the pipeline and parallel designs over average energy consumption across early-exit LeNet-5, AlexNet, VGG19 and ResNet32.
- 'Fig6c.csv': Data supporting Fig. 6 (c): Comparison of the pipeline and parallel designs over average data movement across early-exit LeNet-5, AlexNet, VGG19 and ResNet32.
- 'Table2.csv': Data supporting TABLE II: Execution Time (ms).
- 'Table3.csv': Data supporting TABLE III: Energy Consumption (mJ), with (w/ EE) and without (w/oEE) early exits.
- 'Table4.csv': Data supporting TABLE IV: Performance Comparison With Exisiting Implementations.
- 'Fig7a.csv': Data supporting Fig. 7 (a): On an 3 point early-exit Resnet-32 shows each exit’s trigger rate for different Confidence Thresholds.
- 'Fig7b.csv': Data supporting Fig. 7 (b): On an 3 point early-exit Resnet-32 the percentage difference of parallel over pipeline approaches over Energy and Time for different Confidence Thresholds.
Related projects: Engineering and Physical Sciences Research Council (EPSRC) under EP/S030069/1
Licence: CC BY 4.0</span
Realisation of early-exit dynamic neural networks on reconfigurable hardware
No full textEarly-Exiting is a strategy that’s becoming popular in Deep Neural Networks (DNNs), as it can lead to faster execution and a reduction in the computational intensity of inference. To achieve this intermediate classifiers abstract information from the input samples to strategically stop forward propagation and generate an output at an earlier stage. Confidence criteria are used to identify easier to recognise samples over the ones that need further filtering. However, such dynamic DNNs have only been realised in conventional computing systems (CPU+GPU) using libraries designed for static networks. In this paper, we do a first exploration to efficiently realise early-exit dynamic DNNs on FPGAs, a platform already proven to be highly effective for neural network applications. We consider two approaches for implementing and executing the intermediate classifiers: pipeline, which uses existing hardware, and parallel, which uses additional dedicated modules. We model their energy needs and execution time and explore their performance using the BranchyNet early exit approach on LeNet-5, AlexNet, VGG19 and ResNet32, and a Xilinx ZCU106 Evaluation Board. We found that the dynamic approaches are at least 24% faster than a static network executed on a FPGA, consuming a minimum of 1.32x lower energy. We further observe that FPGAs can enhance the performance of early-exit dynamic DNNs, by minimising the complexities introduced by the decision intermediate classifiers, through parallel execution. Finally we compare the two approaches, and identify which is best for different network types and confidence levels
Dataset for "Optimising Resource Management for Embedded Machine Learning"
No full textDataset supports: Xun, L., Tran-Thanh, L., Al-Hashimi, B., & Merrett, G. (2019). Optimising resource management for embedded machine learning. In Design, Automation and Test in Europe Conference 2020 (DATE'20).</span
Runtime DNN performance scaling through resource management on heterogeneous embedded platforms
No full textDNN inference is increasingly being executed locally on embedded platforms, due to the clear advantages in latency, privacy and connectivity. Modern SoCs typically execute a combination of different and dynamic workloads concurrently, it is challenging to consistently meet latency/energy budgets because the local computing resources available to the DNN vary considerably. In this poster, we show how resource management can be applied to optimise the performance of DNN workloads by monitoring and tuning both software and hardware constantly at runtime. This work shows how dynamic DNNs trade-off accuracy with latency/energy/power on heterogeneous embedded CPU-GPU platform
- …
