4,247 research outputs found
Teacher-apprentices RL (TARL): leveraging complex policy distribution through generative adversarial hypernetwork in reinforcement learning
No full textTypically, a Reinforcement Learning (RL) algorithm focuses in learning a single deployable policy as the end product. Depending on the initialization methods and seed randomization, learning a single policy could possibly leads to convergence to different local optima across different runs, especially when the algorithm is sensitive to hyper-parameter tuning. Motivated by the capability of Generative Adversarial Networks (GANs) in learning complex data manifold, the adversarial training procedure could be utilized to learn a population of good-performing policies instead. We extend the teacher-student methodology observed in the Knowledge Distillation field in typical deep neural network prediction tasks to RL paradigm. Instead of learning a single compressed student network, an adversarially-trained generative model (hypernetwork) is learned to output network weights of a population of good-performing policy networks, representing a school of apprentices. Our proposed framework, named Teacher-Apprentices RL (TARL), is modular and could be used in conjunction with many existing RL algorithms. We illustrate the performance gain and improved robustness by combining TARL with various types of RL algorithms, including direct policy search Cross-Entropy Method, Q-learning, Actor-Critic, and policy gradient-based methods.Green Open Access added to TU Delft Institutional Repository ‘You share, we take care!’ – Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.Interactive Intelligenc
Reinforcement Learning for Fair and Efficient Charging Coordination for Smart Grid
No full textThe integration of renewable energy sources, such as rooftop solar panels, into smart grids poses significant challenges for managing customer-side battery storage. In response, this paper introduces a novel reinforcement learning (RL) approach aimed at optimizing the coordination of these batteries. Our approach utilizes a single-agent, multi-environment RL system designed to balance power saving, customer satisfaction, and fairness in power distribution. The RL agent dynamically allocates charging power while accounting for individual battery levels and grid constraints, employing an actor–critic algorithm. The actor determines the optimal charging power based on real-time conditions, while the critic iteratively refines the policy to enhance overall performance. The key advantages of our approach include: (1) Adaptive Power Allocation: The RL agent effectively reduces overall power consumption by optimizing grid power allocation, leading to more efficient energy use. (2) Enhanced Customer Satisfaction: By increasing the total available power from the grid, our approach significantly reduces instances of battery levels falling below the critical state of charge (SoC), thereby improving customer satisfaction. (3) Fair Power Distribution: Fairness improvements are notable, with the highest fair reward rising by 173.7% across different scenarios, demonstrating the effectiveness of our method in minimizing discrepancies in power distribution. (4) Improved Total Reward: The total reward also shows a significant increase, up by 94.1%, highlighting the efficiency of our RL-based approach. Experimental results using a real-world dataset confirm that our RL approach markedly improves fairness, power efficiency, and customer satisfaction, underscoring its potential for optimizing smart grid operations and energy management systems
BADDr: Bayes-Adaptive Deep Dropout RL for POMDPs
No full textWhile reinforcement learning (RL) has made great advances in scalability, exploration and partial observability are still active research topics. In contrast, Bayesian RL (BRL) provides a principled answer to both state estimation and the exploration-exploitation trade-off, but struggles to scale. To tackle this challenge, BRL frameworks with various prior assumptions have been proposed, with varied success. This work presents a representation-agnostic formulation of BRL under partially observability, unifying the previous models under one theoretical umbrella. To demonstrate its practical significance we also propose a novel derivation, Bayes-Adaptive Deep Dropout rl (BADDr), based on dropout networks. Under this parameterization, in contrast to previous work, the belief over the state and dynamics is a more scalable inference problem. We choose actions through Monte-Carlo tree search and empirically show that our method is competitive with state-of-the-art BRL methods on small domains while being able to solve much larger ones.Green Open Access added to TU Delft Institutional Repository ‘You share, we take care!’ – Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.Interactive Intelligenc
Predictive control for HEMS: from MPC to hybrid continual Meta-RL
No full textLAUREA MAGISTRALELa crescente complessità dei sistemi energetici residenziali, guidata dall'integrazione di fonti rinnovabili, veicoli elettrici, sistemi di accumulo termico e tariffazione dinamica, ha reso i Home Energy Management Systems (HEMS) elementi centrali per le operazioni delle smart grid. Questa tesi esplora l'evoluzione delle tecniche di controllo per i HEMS, partendo dai metodi classici come i Rule-Based Controllers (RBC), i controllori Proporzionali-Integrali-Derivativi (PID) e i Model Predictive Controllers (MPC), fino ad arrivare alle moderne tecniche basate su Reinforcement Learning (RL).
Le limitazioni intrinseche di questi approcci motivano lo sviluppo di architetture ibride che combinino l’interpretabilità e la gestione dei vincoli proprie del MPC con l’adattabilità e l’efficienza del RL. A tal fine, vengono esplorati tre schemi ibridi: MPC come all'interno della deployed policy, come attore esperto e come critico. Per affrontare la non stazionarietà dei sistemi residenziali, la tesi analizza anche le architetture di controllo adattive e il Meta-Reinforcement Learning (Meta-RL), concentrandosi in particolare sull’innovativa architettura di Continual Meta-RL.
Il contributo principale di questo lavoro consiste in una nuova architettura ibrida MPC e Continual Meta-RL, in cui un controller MPC completamente parametrizzato viene aggiornato in tempo reale tramite un algoritmo di Continual Meta Policy Search. Questa soluzione risponde efficacemente all’esigenza di adattabilità dei HEMS, sfruttando i punti di forza già dimostrati dagli approcci di controllo ibrido.
Collegando la teoria classica del controllo con i moderni metodi data-driven, questa tesi pone le basi per architetture HEMS del futuro: intelligenti, robuste e scalabili. Come applicazione prospettica, l’approccio ibrido proposto potrebbe essere esteso a scenari di teleriscaldamento (DH), riflettendo la crescente rilevanza dell’integrazione dei sistemi HEMS nelle infrastrutture energetiche comunitarie.The increasing complexity of residential energy systems, driven by renewables, electric vehicles, thermal storage, and dynamic pricing, has made Home Energy Management Systems (HEMS) central to smart grid operations. This thesis explores the evolution of HEMS control, from classical methods like Rule-Based Control (RBC), PID, and Model Predictive Control (MPC), to modern Reinforcement Learning (RL) techniques.
Limitations of standalone methods motivate the development of hybrid architectures that combine MPC’s interpretability and constraint handling with RL’s adaptability and efficiency. Three hybrid schemes are explored: MPC as a deployed policy, as an expert actor, and as a critic. To address system non-stationarities, the thesis also investigates adaptive control architectures and Meta-RL, focusing on Continual Meta Policy Search for lifelong learning.
The primary contribution of this thesis is a novel hybrid continual Meta-RL architecture that embeds a parameterized MPC within a Continual Meta-Learning framework. This solution addresses the need for adaptability in residential energy systems, while leveraging the strengths of a hybrid control approach.
By bridging classical control theory with modern learning-based methods, this work lays the groundwork for future-ready HEMS architectures that are intelligent, robust, and scalable. As a forward-looking application, the hybrid approach could extended to a Distric Heating (DH) scenario, reflecting the increasing importance of predictive energy optimization at the community level
Reinforcement Learning for Intelligent Healthcare Systems: A Review of Challenges, Applications, and Open Research Issues
No full textThe rise of chronic disease patients and the pandemic pose immediate threats to healthcare expenditure and mortality rates. This calls for transforming healthcare systems away from one-on-one patient treatment into intelligent health systems, leveraging the recent advances of Internet of Things and smart sensors. Meanwhile, reinforcement learning (RL) has witnessed an intrinsic breakthrough in solving a variety of complex problems for distinct applications and services. Thus, this article presents a comprehensive survey of the recent models and techniques of RL that have been developed/used for supporting Intelligent-healthcare (I-health) systems. It can guide the readers to deeply understand the state-of-the-art regarding the use of RL in the context of I-health. Specifically, we first present an overview of the I-health systems' challenges, architecture, and how RL can benefit these systems. We then review the background and mathematical modeling of different RL, deep RL (DRL), and multiagent RL models. We highlight important guidelines on how to select the appropriate RL model for a given problem, and provide quantitative comparisons, showing the results of deploying key RL models in two scenarios that can be followed in monitoring applications. After that, we conduct an in-depth literature review on RL's applications in I-health systems, covering edge intelligence, smart core network, and dynamic treatment regimes. Finally, we highlight emerging challenges and future research directions to enhance RL's success in I-health systems, which opens the door for exploring some interesting and unsolved problems.Green Open Access added to TU Delft Institutional Repository ‘You share, we take care!’ – Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.Networked System
Context-Aware Machine Learning for Smart Manufacturing
No full textRapid evolution of Industry 4.0 and 5.0 demands intelligent, autonomous systems capable of making adaptive decisions in complex dynamic environments using advanced artificial intelligence and machine learning with respect to various contexts. Traditional approaches in both reinforcement learning (RL) and neural networks (NNs) often struggle with several critical challenges. Specifically, RL methods face difficulties in balancing conflicting objectives and quickly adapting to changing contexts, particularly in industrial applications where operational efficiency must be balanced with system maintenance. Similarly, traditional NNs lack the ability to incorporate contextual awareness, limiting their robustness and adaptability in real-world scenarios. This paper addresses these gaps by proposing several solutions. First, we introduce a context-aware balanced RL framework that integrates supervised learning principles into RL, enhancing the ability of intelligent agents to make balanced decisions that consider both external operational goals and internal “health” metrics. Second, we propose a Context-Aware NN architecture that incorporates contextual attributes, improving the network’s performance and relevance in dynamic environments. Third, we present a Self-Context-Aware NN architecture, which enhances model robustness by incorporating sample attribution analysis as a contextual attribute. Finally, we developed a context-aware digital vaccination framework for enhancing models’ robustness. Collectively, these approaches contribute to context-aware, adaptable systems for smart manufacturing.
Find presentation slides here: https://ai.it.jyu.fi/ISM-2024-CONTEXT.pptxpeerReviewe
Refined Risk Management in Safe Reinforcement Learning with a Distributional Safety Critic
No full textSafety is critical to broadening the real-world use of reinforcement learning (RL). Modeling the safety aspects using a safety-cost signal separate from the reward is becoming standard practice, since it avoids the problem of finding a good balance between safety and performance. However, the total safety-cost distribution of different trajectories is still largely unexplored. In this paper, we propose an actor critic method for safe RL that uses an implicit quantile network to approximate the distribution of accumulated safety-costs. Using an accurate estimate of the distribution of accumulated safetycosts, in particular of the upper tail of the distribution, greatly improves the performance of riskaverse RL agents. The empirical analysis shows that our method achieves good risk control in complex safety-constrained environments.AlgorithmicsIntelligent Electrical Power Grid
Optimizing Smart City Water Distribution Systems Using Deep Reinforcement Learning
No full textInefficient scheduling in water distribution systems can lead to energy waste, costly overflows, and a system that cannot keep up with demand. Simultaneous real-time management of system components such as pumps and valves to optimize operation in response to demand variations is a challenging task. Recent advances in deep reinforcement learning provides an opportunity to overcome the state explosion problem using function approximation to generalize from a limited interaction with the environment. In this work, we train a Long Short-Term Memory (LSTM) based Reinforcement Learning (RL) agent to optimize the energy usage of a smart water distribution system while maintaining a safe operating envelope. We compare the performance of the RL agent to two agents based on human experience in the domain; a baseline controller that is based on simple operational logic, and a fuzzy logic controller that captures imprecise human requirements. We show that the RL agent outperforms the other agents in terms of energy usage and operational safety, indicating its potential benefits for large-scale smart city systems. Future research work will focus on prioritized large-scale system scheduling to cope with smart city emergency situations
Simulation-Based Benchmarking of RL Algorithms for Adaptive Thermal Control in IoT-Enabled Smart Umbrella Systems
No full textThis paper presents a simulation-based benchmarking analysis of three reinforcement learning (RL) algorithms—Soft Actor-Critic (SAC), Deep Q-Network (DQN), and Proximal Policy Optimization (PPO)—for real-time thermal regulation in IoT-enabled smart umbrella systems. Motivated by the environmental challenges faced during the Hajj pilgrimage, where pilgrims are exposed to extreme heat and humidity, a custom simulation environment was developed to emulate realistic Hajj conditions. Each RL agent was trained and evaluated across multiple training horizons. Experimental results show that SAC consistently outperforms DQN and PPO in achieving stable thermal comfort while minimizing energy consumption. Compared to traditional on-policy and value-based methods, SAC demonstrated faster convergence, better adaptability to dynamic environmental shifts, and suitability for deployment on resource-constrained microcontroller platforms. These findings highlight the potential of entropy-regularized reinforcement learning (RL) in climate-adaptive, embedded AI systems for human-centric applications
From prediction to action: A hybrid ML-RL model for smart grid stability control
No full textThe increasing complexity of modern electrical grids, driven by the integration of renewable energy sources and distributed generation, presents significant challenges in maintaining grid stability. Traditional methods for stability prediction often fall short due to high computational demands and limited scalability for realtime applications. This paper proposes a novel hybrid approach that combines Machine Learning (ML) and Reinforcement Learning (RL) to address these challenges. The ML component leverages stacking classifiers to predict grid stability, while RL algorithms, including Proximal Policy Optimization (PPO), Advantage Actor-Critic (A2C), and Deep Q-Networks (DQN), optimize power control actions to stabilize the grid. Experimental results demonstrate that the hybrid model achieves a 100% success rate in stabilizing the grid, converging 40% faster than standalone RL methods. The hybrid approach not only reduces computational overhead but also enhances real-time adaptability, making it a promising solution for the dynamic and complex nature of modern smart gridsEngineerin
- …
