625 research outputs found
RSEs in HPC Centers: Funding, Coordinating, Doing
No full text<p>Research Software Engineering (RSEng) as a professional designation has grown over the last 10+ years in industry, academia, and government sectors. Within HPC centers, Research Software Engineers (RSE) fill the role of combining software engineering expertise with the in-depth process of participating in and applying research. In this panel, we invite practicing RSEs, funders, university, and HPC center leaders who are experienced and dedicated to Research Software Engineering to present their varying perspectives on funding, managing, and doing RSEng within worldwide HPC centers. The moderator is Daniel S. Katz (Chief Scientist, NCSA; co-founder, US-RSE), and panelists are Gabrielle Allen (Director, School of Computing, University of Wyoming), Neil Chue Hong (EPCC, University of Edinburgh; Director, Software Sustainability Institute), Alison Kennedy (Strategic Advisor, UK Research and Innovation), Fabio Kon (Special Advisor, São Paulo Research Foundation), and Miranda Mundt (RSE, Sandia National Laboratories; Steering Committee Member, US-RSE).</p>
Exploring HPC and Big Data Convergence: A Graph Processing Study on Intel Knights Landing
No full textThe question 'Can big data and HPC infrastructure converge?' has important implications for many operators and clients of modern computing. However, answering it is challenging. The hardware is currently different, and fast evolving: big data uses machines with modest numbers of fat cores per socket, large caches, and much memory, whereas HPC uses machines with larger numbers of (thinner) cores, non-trivial NUMA architectures, and fast interconnects. In this work, we investigate the convergence of big data and HPC infrastructure for one of the most challenging application domains, the highly irregular graph processing. We contrast through a systematic, experimental study of over 300,000 core-hours the performance of a modern multicore, Intel Knights Landing (KNL) and of traditional big data hardware, in processing representative graph workloads using state-of-the-art graph analytics platforms. The experimental results indicate KNL is convergence-ready, performance-wise, but only after extensive and expert-level tuning of software and hardware parameters.Accepted author manuscriptData-Intensive System
Towards a Dutch Hybrid Quantum/HPC Infrastructure
No full textQuantum Inspire has taken important steps to enable quantum applications by developing a setting that allows the execution of hybrid algorithms. Currently, the setting uses a classical server (HPC node) co-located with the quantum computer for the high frequency coupling needed by hybrid algorithms. A fast task manager (dispatcher) has been developed to orchestrate the interaction between the server and the quantum computer. Although successful, the setting imposes a specific hybrid job-structure. This is most likely always going to be the case and we are currently discussing how to make sure this does not hamper the uptake of the setting. Furthermore, first steps have been taken towards the integration with the Dutch National High-Performance Computing (HPC) Center, hosted by SURF. As a first approach we have setup a setting consisting of two SLURM clusters, one in the HPC (C1) and the second (C2) co-located with Quantum Inspire API. Jobs are submitted from C1 to C2. Quantum Inspire can then schedule with C2 the jobs to the quantum computer. With this setting, we enable control from both SURF and Quantum Inspire on the jobs being executed. By using C1 for the jobs submission we remove the accounting burden from Quantum Inspire. By having C2 co-located with Quantum Inspire API, we make the setting more resilient towards network failures. This setting can be extended for other HPC centers to submit jobs to Quantum Inspire backends.ALG/GeneralBUS/TNO STAF
Desk, HPC-0395
No full textDesk for Bernard Rodey, author of the enabling 1889 legislation that created UNM. Burled wood inlay, but wood type is unknown. Located in stairwell. 87-1/2 x 39-1/4 x 21-3/8 inches. HPC-0395
Singularity containers on HPC
No full textContainerisation is a form of virtualisation that has become very popular in the world of IT
services and cloud computing, offering straightforward portability and deployment of
software and services without having to install a complex set of dependencies. It has
recently become available on High Performance Computing (HPC) systems through the
popular Singularity software package. Singularity is a containerisation tool that is
particularly suitable for HPC and scientific applications, featuring immutable software to
support reproducibility of scientific results, as well as integration with HPC file systems, MPI,
and more.This talk will outline the basic concepts of containerisation and discuss a recent NeSI
consultancy project where a web server and database were containerised to process data
on the Mahuika HPC. The project is now easily portable and can scale out to many cores,
enabling very significant speed-ups.ABOUT THE AUTHOR(S) Wolfgang Hayek is a research software engineer at NeSI and NIWA, and group
manager of NIWA’s scientific programming group, with many years of experience in
scientific computing and HPC. Blair Bethwaite is solutions manager at NeSI; he has strong expertise in HPC, cloud
computing, cloud architectures, and scientific computing Ben Roberts is an application support specialist at NeSI and has many years of
experience in scientific computing and HPC</div
Breaking HPC Barriers with the 56GbE Cloud
No full textAbstractWith the widespread adoption of cloud computing, high-performance computing (HPC) is no longer limited to organisations with the funds and manpower necessary to house and run a supercomputer. However, the performance of large-scale scientific applications in the cloud has in the past been constrained by latency and bandwidth. The main reasons for these constraints are the design decisions of cloud providers, primarily focusing on high-density applications such as web services and data hosting.In this paper, we provide an overview of a high performance OpenStack cloud implementation at the National Computational Infrastructure (NCI). This cloud is targeted at high-performance scientific applications, and enables scientists to build their own clusters when their demands and software stacks conflict with traditional bare-metal HPC environments. In this paper, we present the architecture of our 56 GbE cloud and a preliminary set of HPC benchmark results against the more traditional cloud and native InfiniBand HPC environments.Three different network interconnects and configurations were tested as part of the Cloud deployment. These were 10G Ethernet, 56G Fat-tree Ethernet and native FDR Full Fat-tree InfiniBand (IB). In this paper, these three solutions are discussed from the viewpoint of on-demand HPC clusters focusing on bandwidth, latency and security. A detailed analysis of these metrics in the context of micro-benchmarks and scientific applications is presented, including the affects of using TCP and RDMA on scientific applications
Understanding User Behavior: From HPC to HTC
No full textAbstractIn this paper, we investigate the differences and similarities in user job submission behavior in High Performance Computing (HPC) and High Throughput Computing (HTC). We consider job submission behavior in terms of parallel batch-wise submissions, as well as delays and pauses in job submission. Our findings show that modeling user-based HTC job submission behavior requires knowledge of the underlying bags of tasks, which is often unavailable. Furthermore, we find evidence that subsequent job submission behavior is not influenced by the different complexities and requirements of HPC and HTC jobs
Performance study of HPC applications on an Arm-based cluster using a generic efficiency model
No full textHPC systems and parallel applications are increasing their complexity. Therefore the possibility of easily study and project at large scale the performance of scientific applications is of paramount importance. In this paper we describe a performance analysis method and we apply it to four complex HPC applications. We perform our study on a pre-production HPC system powered by the latest Arm-based CPUs for HPC, the Marvell ThunderX2. For each application we spot inefficiencies and factors that limit their scalability. The results show that in several cases the bottlenecks do not come from the hardware but from the way applications are programmed or the way the system software is configured.This work is partially supported by the Spanish Government (SEV-2015-0493), by the Spanish Ministry of Science and Technology (TIN2015-65316-P), by the Generalitat de Catalunya (2017-SGR-1414), and by the European MontBlanc 3 project (GA n. 671697) and POP CoE (GA n. 824080).Peer ReviewedPostprint (author's final draft
Data set for anomaly detection on a HPC system
No full text<p>This data set contains the data collected on the DAVIDE HPC system (CINECA & E4 & University of Bologna, Bologna, Italy) in the period March-May 2018.</p>
<p>The data set has been used to train a autoencoder-based model to automatically detect anomalies in a semi-supervised fashion, on a real HPC system.</p>
<p>This work is described in:</p>
<p>1) "Anomaly Detection using Autoencoders in High Performance Computing Systems", <a href="https://arxiv.org/search/cs?searchtype=author&query=Borghesi%2C+A">Andrea Borghesi</a>, <a href="https://arxiv.org/search/cs?searchtype=author&query=Bartolini%2C+A">Andrea Bartolini</a>, <a href="https://arxiv.org/search/cs?searchtype=author&query=Lombardi%2C+M">Michele Lombardi</a>, <a href="https://arxiv.org/search/cs?searchtype=author&query=Milano%2C+M">Michela Milano</a>, <a href="https://arxiv.org/search/cs?searchtype=author&query=Benini%2C+L">Luca Benini,</a> IAAI19 (proceedings in process) -- https://arxiv.org/abs/1902.08447</p>
<p>2) "Online Anomaly Detection in HPC Systems", <a href="https://arxiv.org/search/cs?searchtype=author&query=Borghesi%2C+A">Andrea Borghesi</a>, <a href="https://arxiv.org/search/cs?searchtype=author&query=Libri%2C+A">Antonio Libri</a>, <a href="https://arxiv.org/search/cs?searchtype=author&query=Benini%2C+L">Luca Benini</a>, <a href="https://arxiv.org/search/cs?searchtype=author&query=Bartolini%2C+A">Andrea Bartolini, </a>AICAS19 (proceedings in process) -- https://arxiv.org/abs/1811.05269</p>
<p>See the git repository for usage examples & details --> https://github.com/AndreaBorghesi/anomaly_detection_HPC</p>
Adaptive memory power management techniques for HPC workloads
No full textThe memory subsystem is responsible for a large fraction of the energy consumed by compute nodes in High Performance Computing (HPC) systems. The rapid increase in the number of cores has been accompanied by a proportional increase in the DRAM capacity and bandwidth. Thus, the memory system consumes a significant amount of the power budget available to a compute node. There is a broad research effort focused on power management techniques using DRAM low-power modes. However, memory power management still presents many challenges towards Exascale. In this thesis, the potential of Dynamic Voltage and Frequency memory Scaling (DVFS) is studied considering the ability to select different frequencies for different memory channels. The approach adopted is based on tuning voltage and frequency dynamically to maximize the energy savings while maintaining performance degradation within tolerable limits. It was observed that HPC workloads rarely require maximum bandwidth, and the bandwidth demand placed by applications is spread over different channels. Also, HPC applications do not use all the bandwidth in a sustained manner, and they have phases where this bandwidth demand is not at its peak. Hence applications can tolerate reduction in bandwidth to save energy. Channel access patterns of applications are studied to determine the potential additional energy savings by controlling channels independently. Evaluation of proposed DVFS algorithm is conducted through a novel hybrid evaluation methodology that includes simulation and executions on real hardware. Results show the large potential of adaptive memory power management techniques based on DVFS for HPC workloads.M.S.Includes bibliographical referencesby Karthik Elangova
- …
