2,403 research outputs found
Modeling fall propensity in Parkinson's disease: Deficits in the attentional control of complex movements in rats with cortical-cholinergic and striatal-dopaminergic deafferentation
No full textCognitive symptoms, complex movement deficits, and increased propensity for falls are interrelated and levodopa-unresponsive symptoms in patients with Parkinson's disease (PD). We developed a test system for the assessment of fall propensity in rats and tested the hypothesis that interactions between loss of cortical cholinergic and striatal dopaminergic afferents increase fall propensity. Rats were trained to traverse stationary and rotating rods, placed horizontally or at inclines, and while exposed to distractors. Rats also performed an operant Sustained Attention Task (SAT). Partial cortical cholinergic and/or caudate dopaminergic deafferentation were produced by bilateral infusions of 192 IgG-saporin (SAP) into the basal forebrain and/or 6-hydroxydopamine (6-OHDA) into the caudate nucleus, respectively, modeling the lesions seen in early PD. Rats with dual cholinergic-dopaminergic lesions (DL) fell more frequently than SAP or 6-OHDA rats. Falls in DL rats were associated with incomplete rebalancing after slips and low traversal speed. Ladder rung walking and pasta handling performance did not indicate sensorimotor deficits. SAT performance was impaired in DL and SAP rats; however, SAT performance and falls were correlated only in DL rats. Furthermore, in DL rats, but not in rats with only dopaminergic lesions, the placement and size of dopaminergic lesion correlated significantly with fall rates. The results support the hypothesis that after dual cholinergic-dopaminergic lesions, attentional resources can no longer be recruited to compensate for diminished striatal control of complex movement, thereby "unmasking" impaired striatal control of complex movements and yielding falls
Modeling fall propensity in Parkinson's disease: Deficits in the attentional control of complex movements in rats with cortical-cholinergic and striatal-dopaminergic deafferentation
No full textCognitive symptoms, complex movement deficits, and increased propensity for falls are interrelated and levodopa-unresponsive symptoms in patients with Parkinson's disease (PD). We developed a test system for the assessment of fall propensity in rats and tested the hypothesis that interactions between loss of cortical cholinergic and striatal dopaminergic afferents increase fall propensity. Rats were trained to traverse stationary and rotating rods, placed horizontally or at inclines, and while exposed to distractors. Rats also performed an operant Sustained Attention Task (SAT). Partial cortical cholinergic and/or caudate dopaminergic deafferentation were produced by bilateral infusions of 192 IgG-saporin (SAP) into the basal forebrain and/or 6-hydroxydopamine (6-OHDA) into the caudate nucleus, respectively, modeling the lesions seen in early PD. Rats with dual cholinergic-dopaminergic lesions (DL) fell more frequently than SAP or 6-OHDA rats. Falls in DL rats were associated with incomplete rebalancing after slips and low traversal speed. Ladder rung walking and pasta handling performance did not indicate sensorimotor deficits. SAT performance was impaired in DL and SAP rats; however, SAT performance and falls were correlated only in DL rats. Furthermore, in DL rats, but not in rats with only dopaminergic lesions, the placement and size of dopaminergic lesion correlated significantly with fall rates. The results support the hypothesis that after dual cholinergic-dopaminergic lesions, attentional resources can no longer be recruited to compensate for diminished striatal control of complex movement, thereby "unmasking" impaired striatal control of complex movements and yielding falls
Striatal Dopamine: The Cement of the Brain?
No full texthttp://deepblue.lib.umich.edu/bitstream/2027.42/177388/2/Movement Disorders - 2023 - Albin - Striatal Dopamine The Cement of the Brain.pdfPublished versionDescription of Movement Disorders - 2023 - Albin - Striatal Dopamine The Cement of the Brain.pdf : Published versio
Teacher-apprentices RL (TARL): leveraging complex policy distribution through generative adversarial hypernetwork in reinforcement learning
No full textTypically, a Reinforcement Learning (RL) algorithm focuses in learning a single deployable policy as the end product. Depending on the initialization methods and seed randomization, learning a single policy could possibly leads to convergence to different local optima across different runs, especially when the algorithm is sensitive to hyper-parameter tuning. Motivated by the capability of Generative Adversarial Networks (GANs) in learning complex data manifold, the adversarial training procedure could be utilized to learn a population of good-performing policies instead. We extend the teacher-student methodology observed in the Knowledge Distillation field in typical deep neural network prediction tasks to RL paradigm. Instead of learning a single compressed student network, an adversarially-trained generative model (hypernetwork) is learned to output network weights of a population of good-performing policy networks, representing a school of apprentices. Our proposed framework, named Teacher-Apprentices RL (TARL), is modular and could be used in conjunction with many existing RL algorithms. We illustrate the performance gain and improved robustness by combining TARL with various types of RL algorithms, including direct policy search Cross-Entropy Method, Q-learning, Actor-Critic, and policy gradient-based methods.Green Open Access added to TU Delft Institutional Repository ‘You share, we take care!’ – Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.Interactive Intelligenc
Deficits in the attentional control of posture and complex movements in a rat model of early state, multisystem Parkinson’s disease
No full textParkinson’s disease (PD) is increasingly recognized as a multisystem neurodegenerative disorder. Loss of basal forebrain cholinergic neurons occurs as early as the loss of midbrain dopaminergic neurons and is hypothesized to contribute to the cognitive deficits in PD. PD patients also suffer from a propensity for falls and associated impairments in posture control and movement efficacy; these symptoms do not benefit from L-DOPA treatment. Our research hypothesizes that attentional deficits caused by loss of cholinergic neurons interact with limitations in posture control and movement efficacy caused by dopaminergic cell loss to cause falls. We therefore generated rats with a 50-70% loss of the cortical cholinergic input system and a mild loss of dopaminergic afferents of the dorsal striatum (‘dual’ lesions) to model the individual and combined loss of neurons in these two key neuronal systems affected early in PD. To assess posture control and complex movements demanding attentional supervision, we developed the Michigan Complex Motor Control Test (MCMCT). In addition to several control procedures, the core element of this test battery consists of 2-m long rotating (10 rpm) square or round rods, placed at zero, 22.5° or 45° angles to the vertical plane. Animals are trained to traverse these rods to enter the home cage. Performance was assessed using the Mobility Error Index (MEI) that assigns point-scale values for missteps, limb slips and falls. Traversing these rods requires continuous attention to rotation speed, posture and limb coordination. Furthermore, and importantly, the ability of an olfactory distractor (almond extract, placed 10 cm below the rod at about the 1-m mark) to evoke freezing and falls was assessed. To obtain an independent measure of their attentional capacity rats also practiced daily the Sustained Attention Task (SAT). Dual lesions resulted in near chance SAT performance. The lesions did not produce impairments in motor functions as tested by standard rodent neurological tests, a ladder task, and the vermicelli test. MCMCT testing revealed that dual lesions resulted in a greater number of falls and higher MEI scores relative to shams. Presentation of the olfactory distracter caused missteps and transient lower body imbalances that duals were unable to overcome leading to increased falls. Collectively, this evidence supports the usefulness of the MCMCT for research concerning the neuronal basis of falls and associated impairments in movement control, and indicates that limited dual loss in cholinergic and striatal dopamine systems disrupt posture control and movement efficacy in conditions requiring attentional supervision
Deficits in attentional control of balance, mobility, and complex movements in a rat model of early state, multisystem Parkinson disease.
No full textIn Parkinson disease (PD), basal forebrain cholinergic loss coincides with midbrain dopaminergic neuron loss and contributes to attentional deficits in PD. We hypothesize that these attentional deficits contribute to L-DOPA-insensitive impairments of mobility and postural control in PD. To assess complex movement control, we developed a novel Complex Motor Control Test (CMCT) for rats. The CMCT consists of several 2 m long beams (plank, 13.34 cm width; round rod, 3.81 cm diameter; square rod, 2.54 cm side length), which can be placed at zero, 22.5° or 45° angles in the vertical plane. Rods can rotate at 1 rpm or 10 rpm. A separate ladder apparatus (100 cm long, 7 cm wide, 2 cm between rungs, 5 mm rung diameter) can be placed at zero, 22.5° or 45° angles in the vertical plane and tilted laterally at 15° or 30° angles. Four high-resolution cameras and mirror system record animals’ performances. Rats are habituated by learning that plank traversal allows entry of home compartments containing individual bedding and palatable food. To separately assess attentional performance, we employed our Sustained Attention Task (SAT), including a distractor condition (dSAT). Our initial experiments determined CMCT and SAT performance in three groups: (1) animals with limited (40-60%) loss of cortical cholinergic afferents following immunotoxin 192-IgG saporin basal forebrain lesions (SAP); (2) animals with dopaminergic deafferentation following 6-OHDA dorsal striatal lesions (6-OHDA); (3) animals with both types of deafferentation (DUAL). SAT performance was dramatically impaired in SAP and DUAL animals. Control animals rapidly traversed angled and rotating rods and angled and tilted ladders. Deafferented animals were able to traverse the plank at all angles as effectively as control animals. Cholinergic lesions robustly impaired animals’ ability to maintain balance on the rods, to re-adjust posture on and traverse rotating rods, and had falls (into a net) or dismounts more frequently than control animals. These data reveal unexpectedly striking impairments in complex gait and movement control resulting from loss of corticopetal cholinergic neurons. These results support the hypothesis that basal forebrain cholinergic cell loss in PD contributes to complex posture and movement control deficits
BADDr: Bayes-Adaptive Deep Dropout RL for POMDPs
No full textWhile reinforcement learning (RL) has made great advances in scalability, exploration and partial observability are still active research topics. In contrast, Bayesian RL (BRL) provides a principled answer to both state estimation and the exploration-exploitation trade-off, but struggles to scale. To tackle this challenge, BRL frameworks with various prior assumptions have been proposed, with varied success. This work presents a representation-agnostic formulation of BRL under partially observability, unifying the previous models under one theoretical umbrella. To demonstrate its practical significance we also propose a novel derivation, Bayes-Adaptive Deep Dropout rl (BADDr), based on dropout networks. Under this parameterization, in contrast to previous work, the belief over the state and dynamics is a more scalable inference problem. We choose actions through Monte-Carlo tree search and empirically show that our method is competitive with state-of-the-art BRL methods on small domains while being able to solve much larger ones.Green Open Access added to TU Delft Institutional Repository ‘You share, we take care!’ – Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.Interactive Intelligenc
Refined Risk Management in Safe Reinforcement Learning with a Distributional Safety Critic
No full textSafety is critical to broadening the real-world use of reinforcement learning (RL). Modeling the safety aspects using a safety-cost signal separate from the reward is becoming standard practice, since it avoids the problem of finding a good balance between safety and performance. However, the total safety-cost distribution of different trajectories is still largely unexplored. In this paper, we propose an actor critic method for safe RL that uses an implicit quantile network to approximate the distribution of accumulated safety-costs. Using an accurate estimate of the distribution of accumulated safetycosts, in particular of the upper tail of the distribution, greatly improves the performance of riskaverse RL agents. The empirical analysis shows that our method achieves good risk control in complex safety-constrained environments.AlgorithmicsIntelligent Electrical Power Grid
qgym: A Gym for Training and Benchmarking RL-Based Quantum Compilation
No full textCompiling a quantum circuit for specific quantum hardware is a challenging task. Moreover, current quantum computers have severe hardware limitations. To make the most use of the limited resources, the compilation process should be optimized. To improve currents methods, Reinforcement Learning (RL), a technique in which an agent interacts with an environment to learn complex policies to attain a specific goal, can be used. In this work, we present qgym, a software framework derived from the OpenAI gym, together with environments that are specifically tailored towards quantum compilation. The goal of qgym is to connect the research fields of Artificial Intelligence (AI) with quantum compilation by abstracting parts of the process that are irrelevant to either domain. It can be used to train and benchmark RL agents and algorithms in highly customizable environments.Green Open Access added to TU Delft Institutional Repository ‘You share, we take care!’ – Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.Quantum Circuit Architectures and Technolog
Influence-Augmented Local Simulators: a Scalable Solution for Fast Deep RL in Large Networked Systems
No full textLearning effective policies for real-world problems is still an open challenge for the field of reinforcement learning (RL). The main limitation being the amount of data needed and the pace at which that data can be obtained. In this paper, we study how to build lightweight simulators of complicated systems that can run sufficiently fast for deep RL to be applicable. We focus on domains where agents interact with a reduced portion of a larger environment while still being affected by the global dynamics. Our method combines the use of local simulators with learned models that mimic the influence of the global system. The experiments reveal that incorporating this idea into the deep RL workflow can considerably accelerate the training process and presents several opportunities for the future.Interactive IntelligenceAlgorithmic
- …
