2,232 research outputs found
Receding horizon control for water resources management
No full textIntegrated water resources management (IWRM) is recognized worldwide as the reference paradigm to meet society's long-term needs for water resources while maintaining essential ecological services and economic benefits. In previous publications [A. Castelletti, R. Soncini-Sessa, A procedural approach to strengthening integration and participation in water resource planning, Environmental Modelling & Software 21(10) (2006) 1455 1470; A. Castelletti, F. Pianosi, R. Soncini-Sessa, Integration, participation and optimal control in water resources planning and management, Applied Mathematics and Computation, (2007), doi: 10.1016/j.amc.2007.09.069], the authors have already insisted on the need for a procedural approach to make the IWRM paradigm truly operational; they have emphasized the role played by dynamic optimization in rationalizing and facilitating the selection by the decision maker of a best compromise planning alternative. When planning alternatives also include management policies, as in the case of the water reservoir networks considered in this paper, the best compromise off-line policy resulting from the planning exercise has to be actually implemented in the daily management of the system. Here, again, dynamic optimization may play a central role, as it can be adopted on-line to improve the performance of the off-line policy by exploiting any new useful information available in real-time (e. g., inflow predictions, a power station being temporarily out of service, etc.). In this paper, this approach is explored through a real-world case study of a simple reservoir system. The off-line management policy computed in a previous planning process is refined on-line with a receding horizon control scheme combined with an inflow predictor. The results yield indications that the approach can provide significant advantages to cope with extreme events, particularly those occurring in unusual periods of the year. (C) 2008 Elsevier Inc. All rights reserved.</p
A multiobjective reinforcement learning approach to water resources systems operation: Pareto frontier approximation in a single run
No full textThe operation of large-scale water resources systems often involves several conflicting and noncommensurable objectives. The full characterization of tradeoffs among them is a necessary step to inform and support decisions in the absence of a unique optimal solution. In this context, the common approach is to consider many single objective problems, resulting from different combinations of the original problem objectives, each one solved using standard optimization methods based on mathematical programming. This scalarization process is computationally very demanding as it requires one optimization run for each trade-off and often results in very sparse and poorly informative representations of the Pareto frontier. More recently, bio-inspired methods have been applied to compute an approximation of the Pareto frontier in one single run. These methods allow to acceptably cover the full extent of the Pareto frontier with a reasonable computational effort. Yet, the quality of the policy obtained might be strongly dependent on the algorithm tuning and preconditioning. In this paper we propose a novel multiobjective Reinforcement Learning algorithm that combines the advantages of the above two approaches and alleviates some of their drawbacks. The proposed algorithm is an extension of fitted Q-iteration (FQI) that enables to learn the operating policies for all the linear combinations of preferences (weights) assigned to the objectives in a single training process. The key idea of multiobjective FQI (MOFQI) is to enlarge the continuous approximation of the value function, that is performed by single objective FQI over the state-decision space, also to the weight space. The approach is demonstrated on a real-world case study concerning the optimal operation of the HoaBinh reservoir on the Da river, Vietnam. MOFQI is compared with the reiterated use of FQI and a multiobjective parameterization-simulation-optimization (MOPSO) approach. Results show that MOFQI provides a continuous approximation of the Pareto front with comparable accuracy as the reiterated use of FQI. MOFQI outperforms MOPSO when no a priori knowledge on the operating policy shape is available, while produces slightly less accurate solutions when MOPSO can exploit such knowledge
Tree-based Fitted Q-iteration for Multi-Objective Markov Decision problems
No full textThis paper is about solving multi-objective control problems using a model-free batch-mode reinforcement-learning approach. Although many real-world applications have several conflicting objectives, reinforcement-learning (RL) literature has mainly focused on single-objective control problems. As a consequence, in the presence of multiple objectives, the usual approach is to consider many single-objective control problems (resulting from different combinations of the original problem objectives), each one solved using standard RL techniques. The algorithm proposed in this paper is an extension of Fitted Q-iteration (FQI) that enables to learn the control policies for all the linear combinations of preferences (weights) assigned to the objectives in a single training process. The key idea of multi-objective FQI (MOFQI) is to enlarge the continuous approximation of the action-value function, which is performed by single-objective FQI over the state-action space, also to the weight space. The approach is demonstrated on an interesting real-world application for multi-objective RL algorithms: the optimal operation of a multi-purpose water reservoir
Directly assessing uncertainty in designing the optimal operation of water resources systems by batch mode reinforcement learning
No full textWater reservoir control under economic, social and environmental constraints
No full textAlthough great progress has been made in the last 40 years, efficient operation of water reservoir systems still remains a very active research area. The combination of multiple water uses, non-linearities in the model and in the objectives, strong uncertainties in inputs and high dimensional state make the problem challenging and intriguing. The purpose of this paper is to review, in a strict Control Theory perspective, recent and significant advances in designing management policies for water reservoir networks, under economic, social and environmental constraints. A general and thorough problem formulation is provided, along with a description of traditional solution techniques, their limitations and possible alternative approaches
- …
