1,721,077 research outputs found
Smooth Exploration for Robotic Reinforcement Learning
Get PDFReinforcement learning (RL) enables robots to learn skills from interactions
with the real world. In practice, the unstructured step-based exploration used
in Deep RL -- often very successful in simulation -- leads to jerky motion
patterns on real robots. Consequences of the resulting shaky behavior are poor
exploration, or even damage to the robot. We address these issues by adapting
state-dependent exploration (SDE) to current Deep RL algorithms. To enable this
adaptation, we propose two extensions to the original SDE, using more general
features and re-sampling the noise periodically, which leads to a new
exploration method generalized state-dependent exploration (gSDE). We evaluate
gSDE both in simulation, on PyBullet continuous control tasks, and directly on
three different real robots: a tendon-driven elastic robot, a quadruped and an
RC car. The noise sampling interval of gSDE permits to have a compromise
between performance and smoothness, which allows training directly on the real
robots without loss of performance. The code is available at
https://github.com/DLR-RM/stable-baselines3.Comment: Code: https://github.com/DLR-RM/stable-baselines3/ Training scripts:
https://github.com/DLR-RM/rl-baselines3-zoo
Zuschneiden von Roboteraktionen auf Aufgabenkontexte mittels Aktionsmodellen
No full textIn motor control, mapping high-level goals to low-level motor commands often leads to ambiguities, as there are usually many ways a task can be achieved. The key to resolving such ambiguities, for both humans and robots, lies in prediction. In this dissertation, a novel approach for acquiring and applying predictive robot action models is presented. Action models are first learned from observed experience, and enable robots to predict the performance and outcome of their actions. These models are then used to tailor actions to the context they are being executed in, for instance to optimize action sequences, tailor actions to novel goals, or coordinate actions of several robots. The main principle behind this approach is that in robot controller design, knowledge that robots learn themselves from observed experience complements well the abstract knowledge that humans specify.In der Bewegungssteuerung führt die Abbildung von abstrakten Ziele auf konkreten Bewegungsbefehlen oft zu Ambiguitäten, da Aufgaben oft auf viele Weisen gelöst werden können. In der Natur wie in der Robotik können diese Ambiguitäten mit Vorhersagen aufgelöst werden. In dieser Dissertation wird ein neuer Ansatz für den Erwerb und die Anwendung von Vorhersagemodellen für Roboter vorgestellt. Aktionsmodelle werden erst aus beobachteter Erfahrung gelernt, und ermöglichen Robotern die Performanz und das Resultat ihrer Aktionen vorherzusagen. Mit Hilfe diese Modelle werden Aktionen bezüglich ihres Ausführungskontextes maßgeschneidert, zum Beispiel um Aktionsketten zu optimieren, Aktionen an neue Ziele anzupassen, oder Aktionen mehrere Roboter zu koordinieren. Das Hauptprinzip dieses Ansatzes besteht darin, das Wissen, das sich Roboter selbst durch Beobachtung aneignen, jenes durch den Menschen bestimmte abstrakte Wissen gut komplettiert
Going Beyond Counting First Authors in Author Co-citation Analysis
Get PDFThe present study examines one of the fundamental aspects of author co-citation analysis (ACA) - the way co-citation
counts are defined. Co-citation counting provides the data on which all subsequent statistical analyses and mappings
are based, and we compare ACA results based on two different types of co-citation counting - the traditional type that
only counts the first one among a cited work's authors on the one hand and a non-traditional type that takes into
account the first 5 authors of a cited work on the other hand. Results indicate that the picture produced through this non-traditional author co-citation counting contains more coherent author groups and is therefore considerably clearer. However, this picture represents fewer specialties in the research field being studied than that produced through the traditional first-author co-citation counting when the same number of top-ranked authors is selected and analyzed. Reasons for these effects are discussed
Variations on the Author
Get PDF“Variations on the Author” discusses two of Eduardo Coutinho’s recent films (Um Dia na Vida, from 2010, and Últimas Conversas, posthumously released in 2015) and their contribution to the general question of documentary authorship. The director’s filmography is characterized by a consistent yet self-effacing form of authorial self-inscription: Coutinho often features as an interviewer that rather than express opinions propels discourses; an interviewer that is good at listening. This mode of self-inscription characterizes him as an author who is not expressive but who is nonetheless markedly present on the screen. In Um Dia na Vida, however, Coutinho is completely absent form the image, while Últimas Conversas, on the contrary, includes a confessional prologue that moves the director from the margins to the center of his films. This article examines the ways in which these works stand out in the filmography of a director who offers new insights into the notion of cinematic authorship
DmpBbo: A versatile Python/C++ library for Function Approximation, Dynamical Movement Primitives, and Black-Box Optimization
No full text<p>C++ library for Function Approximation, Dynamical Movement Primitives, and Black-Box Optimization</p>
Appropriate Similarity Measures for Author Cocitation Analysis
Get PDFWe provide a number of new insights into the methodological discussion about author cocitation analysis. We first argue that the use of the Pearson correlation for measuring the similarity between authors’ cocitation profiles is not very satisfactory. We then discuss what kind of similarity measures may be used as an alternative to the Pearson correlation. We consider three similarity measures in particular. One is the well-known cosine. The other two similarity measures have not been used before in the bibliometric literature. Finally, we show by means of an example that our findings have a high practical relevance.information science;Pearson correlation;cosine;similarity measure;author cocitation analysis
Policy Improvement: Between Black-Box Optimization and Episodic Reinforcement Learning
Get PDFNational audiencePolicy improvement methods seek to optimize the parameters of a policy with respect to a utility function. There are two main approaches to performing this optimization: reinforcement learning (RL) and black-box optimization (BBO). In recent years, benchmark comparisons between RL and BBO have been made, and there have been several attempts to specify which approach works best for which types of problem classes. In this article, we make several contributions to this line of research by: 1) Classifying several RL algorithms in terms of their algorithmic properties. 2) Showing how the derivation of ever more powerful RL algorithms displays a trend towards BBO. 3) Continuing this trend by applying two modifications to the state-of-the-art PI2 algorithm, which yields an algorithm we denote PIBB. We show that PIBB is a BBO algorithm. 4) Demonstrating that PIBB achieves similar or better performance than PI2 on several evaluation tasks. 5) Analyzing why BBO outperforms RL on these tasks. Rather than making the case for BBO or RL - in general we expect their relative performance to depend on the task considered - we rather provide two algorithms in which such cases can be made, as the algorithms are identical in all respects except in being RL or BBO approaches to policy improvement
Adaptation de la matrice de covariance pour l'apprentissage par renforcement direct
Get PDFNational audienceLa résolution de problèmes à états et actions continus par l'optimisation de politiques paramétriques est un sujet d'intérêt récent en apprentissage par renforcement. L'algorithme PI2 est un exemple de cette approche, qui bénéficie de fondements mathématiques solides tirés de la commande stochastique optimale et des outils de la théorie de l'estimation statistique. Dans cet article, nous considérons PI2 en tant que membre de la famille plus vaste des méthodes qui partagent le concept de moyenne pondérée par les probabilités pour mettre à jour itérativement des paramètres afin d'optimiser une fonction de coût. Nous comparons PI2 à d'autres membres de la même famille - la " méthode d'entropie croisée " et CMA-ES 1 - au niveau conceptuel et en termes de performance. La comparaison débouche sur la dérivation d'un nouvel algorithme que nous appelons PI2-CMA pour " Path Integral Policy Improvement with Covariance Matrix Adaptation ". Le principal avantage de PI2-CMA est qu'il détermine l'amplitude du bruit d'exploration automatiquement
Robot Skill Learning: From Reinforcement Learning to Evolution Strategies
No full textPolicy improvement methods seek to optimize the parameters of a policy with respect to a utility function. Owing to current trends involving searching in parameter space (rather than action space) and using reward-weighted averaging (rather than gradient estimation), reinforcement learning algorithms for policy improvement, e.g. PoWER and PI2, are now able to learn sophisticated high-dimensional robot skills. A side-effect of these trends has been that, over the last 15 years, reinforcement learning (RL) algorithms have become more and more similar to evolution strategies such as (μW , λ)-ES and CMA-ES. Evolution strategies treat policy improvement as a black-box optimization problem, and thus do not leverage the problem structure, whereas RL algorithms do. In this paper, we demonstrate how two straightforward simplifications to the state-of-the-art RL algorithm PI2 suffice to convert it into the black-box optimization algorithm (μW, λ)-ES. Furthermore, we show that (μW , λ)-ES empirically outperforms PI2 on the tasks in [36]. It is striking that PI2 and (μW , λ)-ES share a common core, and that the simpler algorithm converges faster and leads to similar or lower final costs. We argue that this difference is due to a third trend in robot skill learning: the predominant use of dynamic movement primitives (DMPs). We show how DMPs dramatically simplify the learning problem, and discuss the implications of this for past and future work on policy improvement for robot skill learnin
- …
