14 research outputs found
Safe Multi-Agent Reinforcement Learning with Bilevel Optimization in Autonomous Driving
Get PDFEnsuring safety in MARL, particularly when deploying it in real-world
applications such as autonomous driving, emerges as a critical challenge. To
address this challenge, traditional safe MARL methods extend MARL approaches to
incorporate safety considerations, aiming to minimize safety risk values.
However, these safe MARL algorithms often fail to model other agents and lack
convergence guarantees, particularly in dynamically complex environments. In
this study, we propose a safe MARL method grounded in a Stackelberg model with
bi-level optimization, for which convergence analysis is provided. Derived from
our theoretical analysis, we develop two practical algorithms, namely
Constrained Stackelberg Q-learning (CSQ) and Constrained Stackelberg
Multi-Agent Deep Deterministic Policy Gradient (CS-MADDPG), designed to
facilitate MARL decision-making in autonomous driving applications. To evaluate
the effectiveness of our algorithms, we developed a safe MARL autonomous
driving benchmark and conducted experiments on challenging autonomous driving
scenarios, such as merges, roundabouts, intersections, and racetracks. The
experimental results indicate that our algorithms, CSQ and CS-MADDPG,
outperform several strong MARL baselines, such as Bi-AC, MACPO, and MAPPO-L,
regarding reward and safety performance. The demos and source code are
available at
{https://github.com/SafeRL-Lab/Safe-MARL-in-Autonomous-Driving.git}
TeaMs-RL: Teaching LLMs to Generate Better Instruction Datasets via Reinforcement Learning
Get PDFThe development of Large Language Models (LLMs) often confronts challenges
stemming from the heavy reliance on human annotators in the reinforcement
learning with human feedback (RLHF) framework, or the frequent and costly
external queries tied to the self-instruct paradigm. In this work, we pivot to
Reinforcement Learning (RL) -- but with a twist. Diverging from the typical
RLHF, which refines LLMs following instruction data training, we use RL to
directly generate the foundational instruction dataset that alone suffices for
fine-tuning. Our method, TeaMs-RL, uses a suite of textual operations and
rules, prioritizing the diversification of training datasets. It facilitates
the generation of high-quality data without excessive reliance on external
advanced models, paving the way for a single fine-tuning step and negating the
need for subsequent RLHF stages. Our findings highlight key advantages of our
approach: reduced need for human involvement and fewer model queries (only
of the strong baseline's total), along with enhanced capabilities of
LLMs in crafting and comprehending complex instructions compared to strong
baselines, and substantially improved model privacy protection. Code is
available at the link: https://github.com/SafeRL-Lab/TeaMs-R
Motion Planning for an Unmanned Surface Vehicle with Wind and Current Effects
No full textAiming at the problem that unmanned surface vehicle (USV) motion planning is disturbed by effects of wind and current, a USV motion planning method based on regularization-trajectory cells is proposed. First, a USV motion mathematical model is established while considering the influence of wind and current, and the motion trajectory is analyzed. Second, a regularization-trajectory cell library under the influence of wind and current is constructed, and the influence of wind and current on the weight of the search cost is analyzed. Finally, derived from the regularization-trajectory cell and the search algorithm, a motion planning method for a USV that considers wind and current effects is provided. The experimental results indicate that the motion planning is closer to the actual trajectory of a USV in complex environments and that our method is highly practicable
A human-centered safe robot reinforcement learning framework with interactive behaviors
No full textDeployment of Reinforcement Learning (RL) algorithms for robotics applications in the real world requires ensuring the safety of the robot and its environment. Safe Robot RL (SRRL) is a crucial step toward achieving human-robot coexistence. In this paper, we envision a human-centered SRRL framework consisting of three stages: safe exploration, safety value alignment, and safe collaboration. We examine the research gaps in these areas and propose to leverage interactive behaviors for SRRL. Interactive behaviors enable bi-directional information transfer between humans and robots, such as conversational robot ChatGPT. We argue that interactive behaviors need further attention from the SRRL community. We discuss four open challenges related to the robustness, efficiency, transparency, and adaptability of SRRL with interactive behaviors
Constrained Reinforcement Learning for Vehicle Motion Planning with Topological Reachability Analysis
No full textRule-based traditional motion planning methods usually perform well with prior knowledge of the macro-scale environments but encounter challenges in unknown and uncertain environments. Deep reinforcement learning (DRL) is a solution that can effectively deal with micro-scale unknown and uncertain environments. Nevertheless, DRL is unstable and lacks interpretability. Therefore, it raises a new challenge: how to combine the effectiveness and overcome the drawbacks of the two methods while guaranteeing stability in uncertain environments. In this study, a multi-constraint and multi-scale motion planning method is proposed for automated driving with the use of constrained reinforcement learning (RL), named RLTT, and comprising RL, a topological reachability analysis used for vehicle path space (TPS), and a trajectory lane model (TLM). First, a dynamic model of vehicles is formulated; then, TLM is developed on the basis of the dynamic model, thus constraining RL action and state space. Second, macro-scale path planning is achieved through TPS, and in the micro-scale range, discrete routing points are achieved via RLTT. Third, the proposed motion planning method is designed by combining sophisticated rules, and a theoretical analysis is provided to guarantee the efficiency of our method. Finally, related experiments are conducted to evaluate the effectiveness of the proposed method; our method can reduce 19.9% of the distance cost in the experiments as compared to the traditional method. Experimental results indicate that the proposed method can help mitigate the gap between data-driven and traditional methods, provide better performance for automated driving, and facilitate the use of RL methods in more fields
Balance Reward and Safety Optimization for Safe Reinforcement Learning: A Perspective of Gradient Manipulation
No full textEnsuring the safety of Reinforcement Learning (RL) is crucial for its deployment in real-world applications. Nevertheless, managing the trade-off between reward and safety during exploration presents a significant challenge. Improving reward performance through policy adjustments may adversely affect safety performance. In this study, we aim to address this conflicting relation by leveraging the theory of gradient manipulation. Initially, we analyze the conflict between reward and safety gradients. Subsequently, we tackle the balance between reward and safety optimization by proposing a soft switching policy optimization method, for which we provide convergence analysis. Based on our theoretical examination, we provide a safe RL framework to overcome the aforementioned challenge, and we develop a Safety-MuJoCo Benchmark to assess the performance of safe RL algorithms. Finally, we evaluate the effectiveness of our method on the Safety-MuJoCo Benchmark and a popular safe benchmark, Omnisafe. Experimental results demonstrate that our algorithms outperform several state-of-the-art baselines in terms of balancing reward and safety optimization
Safe and Balanced: A Framework for Constrained Multi-Objective Reinforcement Learning
Get PDFIn numerous reinforcement learning (RL) problems involving safety-critical
systems, a key challenge lies in balancing multiple objectives while
simultaneously meeting all stringent safety constraints. To tackle this issue,
we propose a primal-based framework that orchestrates policy optimization
between multi-objective learning and constraint adherence. Our method employs a
novel natural policy gradient manipulation method to optimize multiple RL
objectives and overcome conflicting gradients between different tasks, since
the simple weighted average gradient direction may not be beneficial for
specific tasks' performance due to misaligned gradients of different task
objectives. When there is a violation of a hard constraint, our algorithm steps
in to rectify the policy to minimize this violation. We establish theoretical
convergence and constraint violation guarantees in a tabular setting.
Empirically, our proposed method also outperforms prior state-of-the-art
methods on challenging safe multi-objective reinforcement learning tasks
A human-centered safe robot reinforcement learning framework with interactive behaviors
Get PDFDeployment of Reinforcement Learning (RL) algorithms for robotics applications in the real world requires ensuring the safety of the robot and its environment. Safe Robot RL (SRRL) is a crucial step toward achieving human-robot coexistence. In this paper, we envision a human-centered SRRL framework consisting of three stages: safe exploration, safety value alignment, and safe collaboration. We examine the research gaps in these areas and propose to leverage interactive behaviors for SRRL. Interactive behaviors enable bi-directional information transfer between humans and robots, such as conversational robot ChatGPT. We argue that interactive behaviors need further attention from the SRRL community. We discuss four open challenges related to the robustness, efficiency, transparency, and adaptability of SRRL with interactive behaviors
A Review of Safe Reinforcement Learning: Methods, Theories and Applications
Get PDFReinforcement Learning (RL) has achieved tremendous success in many complex decision-making tasks. However, safety concerns are raised during deploying RL in real-world applications, leading to a growing demand for safe RL algorithms, such as in autonomous driving and robotics scenarios. While safe control has a long history, the study of safe RL algorithms is still in the early stages. To establish a good foundation for future safe RL research, in this paper, we provide a review of safe RL from the perspectives of methods, theories, and applications. First, we review the progress of safe RL from five dimensions and come up with five crucial problems for safe RL being deployed in real-world applications, coined as '2H3W'. Second, we analyze the algorithm and theory progress from the perspectives of answering the '2H3W' problems. Particularly, the sample complexity of safe RL algorithms is reviewed and discussed, followed by an introduction to the applications and benchmarks of safe RL algorithms. Finally, we open the discussion of the challenging problems in safe RL, hoping to inspire future research on this thread. To advance the study of safe RL algorithms, we release an open-sourced repository containing major safe RL algorithms at the link.</p
Balance Reward and Safety Optimization for Safe Reinforcement Learning: A Perspective of Gradient Manipulation
Get PDFEnsuring the safety of Reinforcement Learning (RL) is crucial for its
deployment in real-world applications. Nevertheless, managing the trade-off
between reward and safety during exploration presents a significant challenge.
Improving reward performance through policy adjustments may adversely affect
safety performance. In this study, we aim to address this conflicting relation
by leveraging the theory of gradient manipulation. Initially, we analyze the
conflict between reward and safety gradients. Subsequently, we tackle the
balance between reward and safety optimization by proposing a soft switching
policy optimization method, for which we provide convergence analysis. Based on
our theoretical examination, we provide a safe RL framework to overcome the
aforementioned challenge, and we develop a Safety-MuJoCo Benchmark to assess
the performance of safe RL algorithms. Finally, we evaluate the effectiveness
of our method on the Safety-MuJoCo Benchmark and a popular safe RL benchmark,
Omnisafe. Experimental results demonstrate that our algorithms outperform
several state-of-the-art baselines in terms of balancing reward and safety
optimization
