405905 research outputs found
Sort by
Energy efficiency trends in HPC: what high-energy and astrophysicists need to know
No full textThe growing energy demands of High Performance Computing (HPC) systems have made energy efficiency a critical concern for system developers and operators. However, HPC users are generally less aware of how these energy concerns influence the design, deployment, and operation of supercomputers even though they experience the consequences. This paper examines the implications of HPC’s energy consumption, providing an overview of current trends aimed at improving energy efficiency. We describe how hardware innovations such as energy-efficient processors, novel system architectures, power management techniques, and advanced scheduling policies do have a direct impact on how applications need to be programmed and executed on HPC systems. For application developers, understanding how these new systems work and how to analyse and report the performances of their own software is critical in the dialog with HPC system designers and administrators. The paper aims to raise awareness about energy efficiency among users, particularly in the high energy physics and astrophysics domains, offering practical advice on how to analyse and optimise applications to reduce their energy consumption without compromising on performance
Elucidating Parasitic Currents in Proton‐Exchange‐Membrane Electrolytic Cells via Physics‐Based and Data‐Driven Modeling
No full textA Convolutional Neural Network for the Detection of Gravity Waves in Satellite Observations and Numerical Simulations
No full textComparisons between observed and model-resolved gravity waves (GWs) are crucial for evaluating general circulation model (GCM) simulation accuracy and understanding wave characteristics. However, observational noise often obscures waves, complicating such comparisons. To address this, we have developed a GW detection method using a convolutional neural network (CNN). The CNN is trained on Atmospheric Infrared Sounder (AIRS) temperatures with labels indicating wave presence based on Berthelemy et al. (2025, https://doi.org/10.5194/egusphere-2025-455). Their method detects noise-induced pixel-to-pixel variations in horizontal wavelengths; in contrast, the CNN robustly identify waves even when applied to smoothly varying model data. Using this method, we compare stratospheric GWs in boreal winters between AIRS observations and a high-top GW-permitting GCM, Japanese Atmospheric GCM for Upper Atmosphere Research (JAGUAR). The results agree well and exhibit similar interannual variability, with discrepancies also identified, including a more zonally elongated distribution of tropical GWs in JAGUAR. This method is broadly applicable to the future use of satellites for guiding wave-resolving atmospheric model development
Advanced Framework for State of Health Estimation Using Equivalent Circuit Models and Machine Learning
No full textNeutron scattering and thermodynamic evidence for emergent photons and fractionalization in a pyrochlore spin ice
No full textImpact of Drying on Membrane Electrode Assembly Morphology and Performance in PEM Electrolysis: Insights from X-RayTomography and Radiography
No full textInvestigating the metastability of amorphous calcium carbonate by droplet microfluidics experiments using machine learning
No full textPedotransfer functions and their impact on soil water dynamics simulation and yield prediction: A HERMES model case study
No full textLiberTEM/LiberTEM: 0.15.1
No full textHomepage: https://libertem.github.io/LiberTEM/GitHub repository: https://github.com/LiberTEM/LiberTEM/PyPI: https://pypi.org/project/libertem/ LiberTEM is an open source platform for high-throughput distributed processing of large-scale binary data sets and live data streams using a modified MapReduce programming model. The current focus is pixelated scanning transmission electron microscopy (STEM) [MacLaren et al. (2016), Ophus (2019)] and scanning electron beam diffraction data. MapReduce-like processing allows to specify an algorithm through two functions: One function that is mapped on portions of the input data, and another function that merges (reduces) a partial result from this mapping step into the complete result. A wide range of TEM and 4D STEM processing tasks can be expressed in this fashion, see Applications. The UDF interface of LiberTEM offers a standardized, versatile API to decouple the mathematical core of an algorithm from details of data source, parallelism, and use of results. Mapping and merging can be performed in any order and with different subdivisions of the input data, including running parts of the calculation concurrently. That means the same implementation can be used in a wide range of modalities, including massive scaling on clusters. Since each merge step produces an intermediate result, this style of processing is suitable for displaying live results from a running calculation in a GUI application and for processing live data streams. A closed-loop feedback between processing and instrument control can be realized as well. See User-defined functions for more details on the LiberTEM UDF interface. The LiberTEM back-end offers high throughput and scalability on PCs, single server nodes, clusters and cloud services. On clusters it can use fast distributed local storage on high-performance SSDs. That way it achieves very high aggregate IO performance on a compact and cost-efficient system built from stock components. All CPU cores and CUDA devices in a system can be used in parallel. LiberTEM is supported on Linux, Mac OS X and Windows. Other platforms that allow installation of Python 3.7+ and the required packages will likely work as well. The GUI is running in a web browser. Installation The short version: source ~/libertem-venv/bin/activate (libertem-venv) python -m pip install 'libertem[torch]' # optional for GPU support # See also https://docs.cupy.dev/en/stable/install.html (libertem-venv) python -m pip install cupy Please see our documentation for details! Alternatively, to run the LiberTEM Docker image: apptainer exec docker://ghcr.io/libertem/libertem /venv/bin/libertem-server Deployment for offline data processing on a single-node system for a local user is thoroughly tested and can be considered stable. Deployment on a cluster is experimental and still requires some additional work, see Issue #105. Back-end support for live data processing is still experimental as well, see https://github.com/LiberTEM/LiberTEM-live. Applications Since LiberTEM is programmable through user-defined functions (UDFs), it can be used for a wide range of processing tasks on array-like data and data streams. The following applications have been implemented already: Virtual detectors (virtual bright field, virtual HAADF, center of mass [Krajnak et al. (2016)], custom shapes via masks) Analysis of amorphous materials Strain mapping Off-axis electron holography reconstruction Single Side Band ptychography Some of these applications are available through an interactive web GUI. Please see the applications section of our documentation for details! The Python API and user-defined functions (UDFs) can be used for complex operations such as arbitrary linear operations and other features like data export. Example Jupyter notebooks are available in the examples directory. If you are having trouble running the examples, please let us know by filing an issue or by joining our Gitter chat. LiberTEM is suitable as a high-performance processing backend for other applications, including live data streams. Contact us if you are interested! LiberTEM is evolving rapidly and prioritizes features following user demand and contributions. Currently we are working on live data processing, improving application support for sparse data and event-based detectors, performance improvements for GPU processing, and implementing analysis methods for various applications of pixelated STEM and other large-scale detector data. If you like to influence the direction this project is taking, or if you'd like to contribute, please join our gitter chat and our general mailing list. File formats LiberTEM currently opens most file formats used for pixelated STEM. See our general information on loading data and format-specific documentation for more information! Raw binary files NumPy .npy binary files Thermo Fisher EMPAD detector [Tate et al. (2016)] files Quantum Detectors MIB format Nanomegas .blo block files Direct Electron DE5 files (HDF5-based) and Norpix SEQ files for DE-Series detectors Gatan K2 IS raw format Stacks of Gatan DM3 and DM4 files (via openNCEM) Single-file Gatan DM4 scans when saved using C-ordering FRMS6 from PNDetector pnCCD cameras [Simson et al. (2015)] (currently alpha, gain correction still needs UI changes) FEI SER files (via openNCEM) MRC (via openNCEM) HDF5-based formats such as HyperSpy files, NeXus and EMD TVIPS binary files Sparse data in Raw CSR (compressed sparse row) format, as is possible to generate from event-based detectors Please contact us if you are interested in support for an additional format! Live processing and detectors (experimental) See LiberTEM-live! License LiberTEM is licensed under the MIT license. Acknowledgements We are very grateful for your continuing support for LiberTEM! See the acknowledgement page for a list of authors and contributors to LiberTEM and its subprojects. See also our info on funding and industry partners