4,267 research outputs found
Teacher-apprentices RL (TARL): leveraging complex policy distribution through generative adversarial hypernetwork in reinforcement learning
No full textTypically, a Reinforcement Learning (RL) algorithm focuses in learning a single deployable policy as the end product. Depending on the initialization methods and seed randomization, learning a single policy could possibly leads to convergence to different local optima across different runs, especially when the algorithm is sensitive to hyper-parameter tuning. Motivated by the capability of Generative Adversarial Networks (GANs) in learning complex data manifold, the adversarial training procedure could be utilized to learn a population of good-performing policies instead. We extend the teacher-student methodology observed in the Knowledge Distillation field in typical deep neural network prediction tasks to RL paradigm. Instead of learning a single compressed student network, an adversarially-trained generative model (hypernetwork) is learned to output network weights of a population of good-performing policy networks, representing a school of apprentices. Our proposed framework, named Teacher-Apprentices RL (TARL), is modular and could be used in conjunction with many existing RL algorithms. We illustrate the performance gain and improved robustness by combining TARL with various types of RL algorithms, including direct policy search Cross-Entropy Method, Q-learning, Actor-Critic, and policy gradient-based methods.Green Open Access added to TU Delft Institutional Repository ‘You share, we take care!’ – Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.Interactive Intelligenc
BADDr: Bayes-Adaptive Deep Dropout RL for POMDPs
No full textWhile reinforcement learning (RL) has made great advances in scalability, exploration and partial observability are still active research topics. In contrast, Bayesian RL (BRL) provides a principled answer to both state estimation and the exploration-exploitation trade-off, but struggles to scale. To tackle this challenge, BRL frameworks with various prior assumptions have been proposed, with varied success. This work presents a representation-agnostic formulation of BRL under partially observability, unifying the previous models under one theoretical umbrella. To demonstrate its practical significance we also propose a novel derivation, Bayes-Adaptive Deep Dropout rl (BADDr), based on dropout networks. Under this parameterization, in contrast to previous work, the belief over the state and dynamics is a more scalable inference problem. We choose actions through Monte-Carlo tree search and empirically show that our method is competitive with state-of-the-art BRL methods on small domains while being able to solve much larger ones.Green Open Access added to TU Delft Institutional Repository ‘You share, we take care!’ – Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.Interactive Intelligenc
Refined Risk Management in Safe Reinforcement Learning with a Distributional Safety Critic
No full textSafety is critical to broadening the real-world use of reinforcement learning (RL). Modeling the safety aspects using a safety-cost signal separate from the reward is becoming standard practice, since it avoids the problem of finding a good balance between safety and performance. However, the total safety-cost distribution of different trajectories is still largely unexplored. In this paper, we propose an actor critic method for safe RL that uses an implicit quantile network to approximate the distribution of accumulated safety-costs. Using an accurate estimate of the distribution of accumulated safetycosts, in particular of the upper tail of the distribution, greatly improves the performance of riskaverse RL agents. The empirical analysis shows that our method achieves good risk control in complex safety-constrained environments.AlgorithmicsIntelligent Electrical Power Grid
qgym: A Gym for Training and Benchmarking RL-Based Quantum Compilation
No full textCompiling a quantum circuit for specific quantum hardware is a challenging task. Moreover, current quantum computers have severe hardware limitations. To make the most use of the limited resources, the compilation process should be optimized. To improve currents methods, Reinforcement Learning (RL), a technique in which an agent interacts with an environment to learn complex policies to attain a specific goal, can be used. In this work, we present qgym, a software framework derived from the OpenAI gym, together with environments that are specifically tailored towards quantum compilation. The goal of qgym is to connect the research fields of Artificial Intelligence (AI) with quantum compilation by abstracting parts of the process that are irrelevant to either domain. It can be used to train and benchmark RL agents and algorithms in highly customizable environments.Green Open Access added to TU Delft Institutional Repository ‘You share, we take care!’ – Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.Quantum Circuit Architectures and Technolog
Influence-Augmented Local Simulators: a Scalable Solution for Fast Deep RL in Large Networked Systems
No full textLearning effective policies for real-world problems is still an open challenge for the field of reinforcement learning (RL). The main limitation being the amount of data needed and the pace at which that data can be obtained. In this paper, we study how to build lightweight simulators of complicated systems that can run sufficiently fast for deep RL to be applicable. We focus on domains where agents interact with a reduced portion of a larger environment while still being affected by the global dynamics. Our method combines the use of local simulators with learned models that mimic the influence of the global system. The experiments reveal that incorporating this idea into the deep RL workflow can considerably accelerate the training process and presents several opportunities for the future.Interactive IntelligenceAlgorithmic
Subtask-masked curriculum learning for reinforcement learning with application to UAV maneuver decision-making
No full textUnmanned Aerial Vehicle (UAV) maneuver strategy learning remains a challenge when using Reinforcement Learning (RL) in this sparse reward task. In this paper, we propose Subtask-Masked curriculum learning for RL (SUBMAS-RL), an efficient RL paradigm that implements curriculum learning and knowledge transfer for UAV maneuver scenarios involving multiple missiles. First, this study introduces a novel concept known as subtask mask to create source tasks from a target task by masking partial subtasks. Then, a subtask-masked curriculum generation method is proposed to generate a sequenced curriculum by alternately conducting task generation and task sequencing. To establish efficient knowledge transfer and avoid negative transfer, this paper employs two transfer techniques, policy distillation and policy reuse, along with an explicit transfer condition that masks irrelevant knowledge. Experimental results demonstrate that our method achieves a 94.8% success rate in the UAV maneuver scenario, where the direct use of reinforcement learning always fails. The proposed RL framework SUBMAS-RL is expected to learn an effective policy in complex tasks with sparse rewards.Green Open Access added to TU Delft Institutional Repository ‘You share, we take care!’ – Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.Algorithmic
Reinforcement Learning for Intelligent Healthcare Systems: A Review of Challenges, Applications, and Open Research Issues
No full textThe rise of chronic disease patients and the pandemic pose immediate threats to healthcare expenditure and mortality rates. This calls for transforming healthcare systems away from one-on-one patient treatment into intelligent health systems, leveraging the recent advances of Internet of Things and smart sensors. Meanwhile, reinforcement learning (RL) has witnessed an intrinsic breakthrough in solving a variety of complex problems for distinct applications and services. Thus, this article presents a comprehensive survey of the recent models and techniques of RL that have been developed/used for supporting Intelligent-healthcare (I-health) systems. It can guide the readers to deeply understand the state-of-the-art regarding the use of RL in the context of I-health. Specifically, we first present an overview of the I-health systems' challenges, architecture, and how RL can benefit these systems. We then review the background and mathematical modeling of different RL, deep RL (DRL), and multiagent RL models. We highlight important guidelines on how to select the appropriate RL model for a given problem, and provide quantitative comparisons, showing the results of deploying key RL models in two scenarios that can be followed in monitoring applications. After that, we conduct an in-depth literature review on RL's applications in I-health systems, covering edge intelligence, smart core network, and dynamic treatment regimes. Finally, we highlight emerging challenges and future research directions to enhance RL's success in I-health systems, which opens the door for exploring some interesting and unsolved problems.Green Open Access added to TU Delft Institutional Repository ‘You share, we take care!’ – Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.Networked System
Treatment of varicocele in pediatric patients with controindication for open surgery with transfemoral retrograde sclero-embolization under local anesthesia
No full textThe author describe an alternative approach for the treatment of varicocele by scleroembolization of spermatic vein
Some Closed-Form Results for Adhesive Rough Contacts Near Complete Contact on Loading and Unloading in the Johnson, Kendall, and Roberts Regime
No full textRecently, generalizing the solution of the adhesiveless random rough contact proposed by Xu, Jackson, and Marghitu (XJM model), the first author has obtained a model for adhesive contact near full contact, under the Johnson, Kendall, and Roberts (JKR) assumptions, which leads to quite strong effect of the fractal dimension. We extend here the results with closed-form equations, including both loading and unloading which were not previously discussed, showing that the conclusions are confirmed. A large effect of hysteresis is found, as was expected. The solution is therefore competitive with Persson's JKR solution, at least in the range of nearly full contact, with an enormous advantage in terms of simplicity. Two examples of real surfaces are discussed
On the ecology and evolution of microorganisms associated with fungus-growing termites
No full text Organisms living in symbiosis fascinate us with their adaptations to live in extreme proximity to, or even inside, a partner that may be from a completely different Class, Phylum or Kingdom. Combinations of organisms that live in mutualistic symbiosis seem very exceptional, but when studying any organism more closely one may find involvement in mutualistic symbiosis to be the rule rather than an exception. For example, most of the animals have microorganisms in their guts that help digestion, and many plants have fungi around their roots that aid in uptake of nutrients from the soil. Having complementary traits and reciprocally benefitting each other, cooperating organisms may evolve into extremely successful species. CHAPTER 1 introduces the topic of this thesis: fungus-growing termites. Fungus-growing termites play a dominant role as ecosystem engineers in sub-Saharan Africa and South Asia. They change soil properties by their building and foraging activities, and are major players in decomposition of wood and dead vegetation. Though they are often regarded as a pest, termites can be very useful for people. Besides eating the termites and mushrooms that emerge from the termite mound, people use termite soil-engineering to improve the fertility of agricultural fields. The termite and fungus live in obligate mutualistic symbiosis. Termites (Blattodea: Termitidae, subfamily Macrotermitinae) provide the fungus Termitomyces (Basidiomycota: Agaricales: Lyophyllaceae) with fragmented dead plant material and create a controlled environment perfect for the fungus, whereas Termitomyces decomposes the low-quality matter into a nutritious food source and produces mushroom primordia both of which are eaten by the termites. The symbiosis exists in a world where other organisms are awaiting their chance to exploit the richness of the termite nests. Hence, one could expect to find other organisms in the nest, next to termites and Termitomyces. There is at least one fungus associated with fungus-growing termites that emerges very prominently after termites are no longer active: species of Xylaria (Ascomycota: Xylariales: Xylariaceae, subgenus Pseudoxylaria) are frequently overgrowing the fungus gardens of dead termite nests. What is the status of Pseudoxylaria in the fungus-growing termite symbiosis, does it play a role? How are the fungus-growing termite gardens kept free of weeds, parasites and pathogens? These questions form the foundation of this thesis on the ecology and evolution of microorganisms associated with fungus-growing termites, with particular focus on the role and interactions with associated Pseudoxylaria. CHAPTER 2 investigates the specificity of Pseudoxylaria for fungus-growing termites. I hypothesize that specificity or selectivity for fungus-growing termites would mean that Pseudoxylaria is not present coincidentally as opportunist, but truly associated with fungus-growing termite symbiosis. Hundred and eight South-African fungus-growing termite nests were sampled for Pseudoxylaria, and it was found in most of the nests. Partial rDNA sequences of the obtained isolates were compared with those of Xylaria from the environment and isolates from other parts of the world. I found 16 different molecular types (‘species’) of Pseudoxylaria. They formed a separate group, showing that Pseudoxylaria specifically occurs in fungus-growing termite nests indeed. No specificity for the termite genus or species was found, implying that Pseudoxylaria may have specialised on the fungus garden substrate, rather than on the termite host or the mutualistic fungus Termitomyces. CHAPTER 3 focuses on the role of Pseudoxylaria in the fungus-growing termite nest. Pseudoxylaria is inconspicuous in healthy termite nests and usually only occurs when termites are no longer present in the nest, or when pieces of fungus garden are incubated without termites in the lab. Therefore, it seems to be suppressed and an unwelcome nest inhabitant. I postulate that Pseudoxylaria is a benign stowaway that practices a sit-and-wait strategy to survive in the termite nest. First, Pseudoxylaria and Termitomyces were grown independently on different carbon sources; to test if they have a complementary diet preference, degrading complementary substrate components as had been suggested previously. The carbon source use of both fungi overlapped, implying that Pseudoxylaria is not a beneficial or benign symbiont. Second, the role of Pseudoxylaria in termite nests was inferred from interactions between mycelia of Pseudoxylaria, Termitomyces, and their free-living relatives. Both fungi were grown on the same plate, and also combinations with each other’s free-living relatives were tested. This revealed that Pseudoxylaria is not parasitizing Termitomyces. Furthermore, Pseudoxylaria grew relatively less than its free-living relatives when combined with Termitomyces. This result suggests that the symbiotic lifestyle adopted by Pseudoxylaria went together with adaptations that changed the interaction between both fungi, consistent with Pseudoxylaria being a stowaway. CHAPTER 4 tests the hypothesis that termite workers play a crucial role in maintaining the fungus garden hygiene. The occurrence of microorganisms other than Termitomyces was monitored for pieces of fungus garden that were incubated with, without, or temporarily without termite workers. The effect that workers had on the fungus-comb hygiene, as well as observations on worker cleaning behaviour and their response to mycelium tissue of Pseudoxylaria and Termitomyces, show that termites play an important role in maintaining the fungus-garden hygiene indeed. CHAPTER 5 explores the potential of Actinobacteria for a mutualistic role as defensive symbiont against Pseudoxylaria in the fungus-growing termite nest. Actinobacteria play a mutualistic role as defensive symbionts in many biological systems. It was unclear by which mechanism the termites suppress Pseudoxylaria. Thirty fungus-growing termite colonies from two geographically distant sites were sampled for Actinobacteria. Resulting isolates were characterized based on morphology and 16S rRNA sequences. Next, the obtained Actinobacteria were tested for their antibiotic effect on both Pseudoxylaria and Termitomyces. This chapter describes the first discovery of an assembly of Actinobacteria occurring in fungus-growing termite nests. Actinobacteria were found throughout all sampled nests and materials, and in the phylogenetic tree their 16S rRNA sequences were interspersed with those of Actinobacteria from origins other than fungus-growing termites. The bioassays showed that many Actinobacteria inhibited both the substrate competitor Pseudoxylaria and the termite cultivar Termitomyces. The lack of specificity of the Actinobacteria for fungus-growing termites, and lack of specific defence against Pseudoxylaria, make it unlikely that Actinobacteria play a role as defensive symbionts in fungus-growing termites. Final CHAPTER 6 reflects on the previous chapters, focussing on underlying mechanisms. What caused fungus-growing termites to survive for thirty million years already, and what makes them so successful that they dominate semi-arid ecosystems in sub-Saharan Africa and South Asia? How are conflicts of interest between symbiotic partners resolved? How does cooperation between termites and Termitomyces remain stable over evolutionary time scales? The roles of termites, Termitomyces, Pseudoxylaria, and other organisms in the fungus-growing termite nest are discussed more elaborately. In addition, the question to what extent certain aspects determine whether an organism behaves parasitically or mutualistically, and the question whether symbiont role affects the level of specificity between symbiotic partners, are examined. An analogy is drawn with human agriculture and directions for future research are given. The chapter ends with main conclusions of this thesis. Fungus-growing termites are so successful in maintaining a Termitomyces monoculture that the means by which they accomplish this may be further studied for human agricultural interests. Pseudoxylaria species occur specifically in fungus-growing termite nests, where they are suppressed by termites while awaiting an opportunity to overgrow the fungus garden. </p
- …
