2,409 research outputs found
Teacher-apprentices RL (TARL): leveraging complex policy distribution through generative adversarial hypernetwork in reinforcement learning
No full textTypically, a Reinforcement Learning (RL) algorithm focuses in learning a single deployable policy as the end product. Depending on the initialization methods and seed randomization, learning a single policy could possibly leads to convergence to different local optima across different runs, especially when the algorithm is sensitive to hyper-parameter tuning. Motivated by the capability of Generative Adversarial Networks (GANs) in learning complex data manifold, the adversarial training procedure could be utilized to learn a population of good-performing policies instead. We extend the teacher-student methodology observed in the Knowledge Distillation field in typical deep neural network prediction tasks to RL paradigm. Instead of learning a single compressed student network, an adversarially-trained generative model (hypernetwork) is learned to output network weights of a population of good-performing policy networks, representing a school of apprentices. Our proposed framework, named Teacher-Apprentices RL (TARL), is modular and could be used in conjunction with many existing RL algorithms. We illustrate the performance gain and improved robustness by combining TARL with various types of RL algorithms, including direct policy search Cross-Entropy Method, Q-learning, Actor-Critic, and policy gradient-based methods.Green Open Access added to TU Delft Institutional Repository ‘You share, we take care!’ – Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.Interactive Intelligenc
BADDr: Bayes-Adaptive Deep Dropout RL for POMDPs
No full textWhile reinforcement learning (RL) has made great advances in scalability, exploration and partial observability are still active research topics. In contrast, Bayesian RL (BRL) provides a principled answer to both state estimation and the exploration-exploitation trade-off, but struggles to scale. To tackle this challenge, BRL frameworks with various prior assumptions have been proposed, with varied success. This work presents a representation-agnostic formulation of BRL under partially observability, unifying the previous models under one theoretical umbrella. To demonstrate its practical significance we also propose a novel derivation, Bayes-Adaptive Deep Dropout rl (BADDr), based on dropout networks. Under this parameterization, in contrast to previous work, the belief over the state and dynamics is a more scalable inference problem. We choose actions through Monte-Carlo tree search and empirically show that our method is competitive with state-of-the-art BRL methods on small domains while being able to solve much larger ones.Green Open Access added to TU Delft Institutional Repository ‘You share, we take care!’ – Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.Interactive Intelligenc
Exploring the utility of indicators of uncomplicated malaria burden from routine health facility surveillance data in identifying and mapping high-risk areas for malaria in Uganda
No full textBackground and aim: Routine surveillance is increasingly recognised as central to multi-dimensional malaria control efforts, especially for programme planning and impact assessment. Whilst it is global strategy to transform surveillance into a core programmatic component, essential in-depth interpretation of routine surveillance data remains limited, especially in higher transmission settings. I therefore aimed to explore utility of indicators of uncomplicated malaria burden from routine health facility surveillance data in identifying and mapping high-risk areas for malaria in Uganda. Methods and data sources: To examine routine surveillance indicators of malaria burden, I first evaluated internal consistency between measures from three national reference health facilities, comparing incidence and test positivity rates over time and space. In addition, I examined impacts of control interventions on the age associated burden of malaria, stratified by endemicity and intervention. I then extended this to compare routine reporting data with concurrent community cohort incidence estimates across three sub-counties to evaluate potential sources of bias. Finally, using four years of national health management information system (HMIS)-reported confirmed malaria data in a Bayesian autoregressive analytical framework, I explored the space-time distribution of malaria, and estimated adjusted national and local HMIS-based incidence rates. Primary findings: At the health facility level, HMIS-based incidence and test positivity rates showed similar trends and predicable relationships, with reduced transmission associated with increasing age of test confirmed malaria cases. Comparison of HMIS and cohort data suggested that HMIS data could provide a relatively unbiased proxy for true incidence - especially in lower-transmission, better performing surveillance systems settings. Lastly, space-time modelling of national HMIS data revealed high-burden and high-risk areas within health facility catchments, districts, and regions, highlighting the utility of routine surveillance data in identifying programmatically relevant heterogeneities in malaria burden in Uganda. Conclusion: This thesis highlights the potential viability of routine data in evaluating endemic malaria risk with improved routine HMIS. This is shown by: similar trends of HMIS-based incidence with other measures; its unbiased relationship with community cohort incidence; and, its capacity to identify high case rate locations. To realize the potential of these data, coordinated efforts are needed towards high testing rates, complete and timely recording and reporting, and multilevel feedback within national malaria control programme systems. Further research opportunities include treatment or non-care seeking and non-reporting care alternatives impacts on surveillance-based indicators of malaria burden
Refined Risk Management in Safe Reinforcement Learning with a Distributional Safety Critic
No full textSafety is critical to broadening the real-world use of reinforcement learning (RL). Modeling the safety aspects using a safety-cost signal separate from the reward is becoming standard practice, since it avoids the problem of finding a good balance between safety and performance. However, the total safety-cost distribution of different trajectories is still largely unexplored. In this paper, we propose an actor critic method for safe RL that uses an implicit quantile network to approximate the distribution of accumulated safety-costs. Using an accurate estimate of the distribution of accumulated safetycosts, in particular of the upper tail of the distribution, greatly improves the performance of riskaverse RL agents. The empirical analysis shows that our method achieves good risk control in complex safety-constrained environments.AlgorithmicsIntelligent Electrical Power Grid
Exploring the spatial heterogeneity of trachomatous trichiasis
No full textProlonged conjunctival infection with Chlamydia trachomatis leads to an inflammatory response, trachomatous inflammation follicular (TF). Over time, repeat infection can progress to scarring of the conjunctiva causing the eyelid to turn inward, resulting in lashes rubbing against the cornea. This painful stage of the disease is called trachomatous trichiasis (TT). TT can damage the cornea, leading to vision impairment or blindness. Trachoma is targeted for elimination as a public health problem by the year 2020, which for TT is defined as less than 1 TT-positive person, who is not already known to the health system, per 1,000 population. For trachoma to meet the elimination targets, massive resources are required for both mapping and intervention. A particularly large knowledge gap exists around identifying areas where TT is likely to be found. To better align resources and plan for elimination, the trachoma community needs to understand how much TT currently exists and requires management, how to accurately measure TT prevalence, and where TT cases are mostly likely to be located. Understanding these elements will help position trachoma control programs to meet the TT elimination targets. In my thesis I first calculate an updated global estimate of TT cases and describe the methods involved. Second, I provide a survey design for measuring TT with adequate precision for control activities, along with validation exercise results and a brief time-cost analysis. I then examine the spatial structure of TF and TT and identify areas of spatial autocorrelation. Finally, I identify environmental factors associated with higher than expected TT prevalence to identify TT hot spots. The outcomes of these activities provide an updated global estimate of existing TT cases, a validated tool for measuring TT prevalence at the implementation unit (district) level, and insight on where to begin case finding activities in the context of the “end game”. These outputs are critical to the continued effort of trachoma elimination as a public health problem, specifically providing targeted direction for TT resources
qgym: A Gym for Training and Benchmarking RL-Based Quantum Compilation
No full textCompiling a quantum circuit for specific quantum hardware is a challenging task. Moreover, current quantum computers have severe hardware limitations. To make the most use of the limited resources, the compilation process should be optimized. To improve currents methods, Reinforcement Learning (RL), a technique in which an agent interacts with an environment to learn complex policies to attain a specific goal, can be used. In this work, we present qgym, a software framework derived from the OpenAI gym, together with environments that are specifically tailored towards quantum compilation. The goal of qgym is to connect the research fields of Artificial Intelligence (AI) with quantum compilation by abstracting parts of the process that are irrelevant to either domain. It can be used to train and benchmark RL agents and algorithms in highly customizable environments.Green Open Access added to TU Delft Institutional Repository ‘You share, we take care!’ – Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.Quantum Circuit Architectures and Technolog
Plasmodium infection and its risk factors in eastern Uganda.
No full textBACKGROUND: Malaria is a leading cause of disease burden in Uganda, although surprisingly few contemporary, age-stratified data exist on malaria epidemiology in the country. This report presents results from a total population survey of malaria infection and intervention coverage in a rural area of eastern Uganda, with a specific focus on how risk factors differ between demographic groups in this population. METHODS: In 2008, a cross-sectional survey was conducted in four contiguous villages in Mulanda, sub-county in Tororo district, eastern Uganda, to investigate the epidemiology and risk factors of Plasmodium species infection. All permanent residents were invited to participate, with blood smears collected from 1,844 individuals aged between six months and 88 years (representing 78% of the population). Demographic, household and socio-economic characteristics were combined with environmental data using a Geographical Information System. Hierarchical models were used to explore patterns of malaria infection and identify individual, household and environmental risk factors. RESULTS: Overall, 709 individuals were infected with Plasmodium, with prevalence highest among 5-9 year olds (63.5%). Thin films from a random sample of 20% of parasite positive participants showed that 94.0% of infections were Plasmodium falciparum and 6.0% were P. malariae; no other species or mixed infections were seen. In total, 68% of households owned at least one mosquito although only 27% of school-aged children reported sleeping under a net the previous night. In multivariate analysis, infection risk was highest amongst children aged 5-9 years and remained high in older children. Risk of infection was lower for those that reported sleeping under a bed net the previous night and living more than 750 m from a rice-growing area. After accounting for clustering within compounds, there was no evidence for an association between infection prevalence and socio-economic status, and no evidence for spatial clustering. CONCLUSION: These findings demonstrate that mosquito net usage remains inadequate and is strongly associated with risk of malaria among school-aged children. Infection risk amongst adults is influenced by proximity to potential mosquito breeding grounds. Taken together, these findings emphasize the importance of increasing net coverage, especially among school-aged children
Influence-Augmented Local Simulators: a Scalable Solution for Fast Deep RL in Large Networked Systems
No full textLearning effective policies for real-world problems is still an open challenge for the field of reinforcement learning (RL). The main limitation being the amount of data needed and the pace at which that data can be obtained. In this paper, we study how to build lightweight simulators of complicated systems that can run sufficiently fast for deep RL to be applicable. We focus on domains where agents interact with a reduced portion of a larger environment while still being affected by the global dynamics. Our method combines the use of local simulators with learned models that mimic the influence of the global system. The experiments reveal that incorporating this idea into the deep RL workflow can considerably accelerate the training process and presents several opportunities for the future.Interactive IntelligenceAlgorithmic
Plasmodium-helminth coinfection and its sources of heterogeneity across East Africa.
No full textBACKGROUND: Plasmodium-helminth coinfection can have a number of consequences for infected hosts, yet our knowledge of the epidemiology of coinfection across multiple settings is limited. This study investigates the distribution and heterogeneity of coinfection with Plasmodium falciparum and 3 major helminth species across East Africa. METHODS: Cross-sectional parasite surveys were conducted among 28 050 children in 299 schools across a range of environmental settings in Kenya, Uganda, and Ethiopia. Data on individual, household, and environmental risk factors were collected and a spatially explicit Bayesian modeling framework was used to investigate heterogeneities of species infection and coinfection and their risk factors as well as school- and individual-level associations between species. RESULTS: Broad-scale geographical patterns of Plasmodium-helminth coinfection are strongly influenced by the least common infection and by species-specific environmental factors. At the individual level, there is an enduring positive association between P. falciparum and hookworm but no association between P. falciparum and Schistosoma species. However, the relative importance of such within-individual associations is less than the role of spatial factors in influencing coinfection risks. CONCLUSIONS: Patterns of coinfection seem to be influenced more by the distribution of the least common species and its environmental risk factors, rather than any enduring within-individual associations
Spatial and genetic epidemiology of hookworm in a rural community in Uganda.
No full textThere are remarkably few contemporary, population-based studies of intestinal nematode infection for sub-Saharan Africa. This paper presents a comprehensive epidemiological analysis of hookworm infection intensity in a rural Ugandan community. Demographic, kinship, socioeconomic and environmental data were collected for 1,803 individuals aged six months to 85 years in 341 households in a cross-sectional community survey. Hookworm infection was assessed by faecal egg count. Spatial variation in the intensity of infection was assessed using a Bayesian negative binomial spatial regression model and the proportion of variation explained by host additive genetics (heritability) and common domestic environment was estimated using genetic variance component analysis. Overall, the prevalence of hookworm was 39.3%, with the majority of infections (87.7%) of light intensity (<or=1000 eggs per gram faeces). Intensity was higher among older individuals and was associated with treatment history with anthelmintics, walking barefoot outside the home, living in a household with a mud floor and education level of the household head. Infection intensity also exhibited significant household and spatial clustering: the range of spatial correlation was estimated to be 82 m and was reduced by a half over a distance of 19 m. Heritability of hookworm egg count was 11.2%, whilst the percentage of variance explained by unidentified domestic effects was 17.8%. In conclusion, we suggest that host genetic relatedness is not a major determinant of infection intensity in this community, with exposure-related factors playing a greater role
- …
