45 research outputs found
Exploiting Big Data solutions for CMS computing operations analytics
No full textComputing operations at the Large Hadron Collider (LHC) at CERN rely on the Worldwide LHC Computing Grid (WLCG) infrastructure, designed to efficiently allow storage, access, and processing of data at the pre-exascale level. A close and detailed study of the exploited computing systems for the LHC physics mission represents an increasingly crucial aspect in the roadmap of High Energy Physics (HEP) towards the exascale regime. In this context, the Compact Muon Solenoid (CMS) experiment has been collecting and storing over the last few years a large set of heterogeneous non-collision data (e.g. meta-data about replicas placement, transfer operations, and actual user access to physics datasets). All this data richness is currently residing on a distributed Hadoop cluster, and it is organized so that running fast and arbitrary queries using the Spark analytics framework is a viable approach for Big Data mining efforts. Using a data-driven approach oriented to the analysis of this meta-data deriving from several CMS computing services, such as DBS (Data Bookkeeping Service) and MCM (Monte Carlo Management system), we started to focus on data storage and data access over the WLCG infrastructure, and we drafted an embryonal software toolkit to investigate recurrent patterns and provide indicators about physics datasets popularity. As a long-term goal, this aims at contributing to the overall design of a predictive/adaptive system that would eventually reduce costs and complexity of the CMS computing operations, while taking into account the stringent requests by the physics analysts communit
A Clustering Method for Multiple-Answer Questions on Pre-service Primary Teachers’ Views of Mathematics
No full textIn the last years, research has paid strong attention to pre-service primary teachers’ views of mathematics. Interviews and questionnaires to pre-service teachers during their academic studies are the mainly used tools for collecting data. Qualitative and quantitative approaches may give different insights. In this paper, after a review of the different methods used in the literature to face the topic of pre-service primary teachers’ views of mathematics, we propose a new method. A clustering technique is applied to data collected with multiple-answer questions about pre-service primary teachers’ views of mathematical ability. Obtained clusters are interpreted and compared
Big Data Analysis for Predictive Maintenance at the INFN-CNAF Data Center using Machine Learning Approaches
No full textCollection and harmonization of system logs and prototypal Analytics services with the Elastic (ELK) suite at the INFN-CNAF computing centre
No full textThe distributed Grid infrastructure for High Energy Physics experiments at the Large Hadron Collider (LHC) in Geneva comprises a set of computing centres, spread all over the world, as part of the Worldwide LHC Computing Grid (WLCG). In Italy, the Tier-1 functionalities are served by the INFN-CNAF data center, which provides also computing and storage resources to more than
twenty non-LHC experiments. For this reason, a high amount of logs are collected each day from various sources, which are highly heterogeneous and difficult to harmonize. In this contribution, a working implementation of a system that collects, parses and displays the log information from
CNAF data sources and the investigation of a Machine Learning based predictive maintenance system, is presented
Towards Predictive Maintenance with Machine Learning at the INFN-CNAF computing centre
No full textThe INFN-CNAF computing center, one of the Worldwide LHC Computing Grid Tier-1 sites, is serving a large set of scientific communities, in High Energy Physics and beyond. In order to increase efficiency and to remain competitive in the long run, CNAF is launching various activities aiming at implementing a global predictive maintenance solution for the site.
This requires a site-wide effort in collecting, cleaning and structuring all possibly useful data coming from log files of the various Tier-1 services and systems, as a necessary step prior to designing machine learning based approaches for predictive maintenance.
Among the Tier-1 services, efficient storage systems are one of the key ingredients of Tier-1 operations. CNAF uses the StoRM service as a Grid Storage Resource Manager solution: its operations are logged in a very complex manner, as the log content is deeply unstructured and hard to be exploited for analytics purposes. Despite such difficulty, the StoRM logs are a precious source of information for operators (e. g. real-time monitoring and anomaly detection), for developers (e. g. debugging, service stability, code improvements) and for site managers (service optimization, storage usage efficiency, time and money saving ways to spot and prevent unwanted behaviors).
Based on previous experiences on Big Data Analytics and Machine/Deep learning in the CMS experiment, this work describes how the StoRM logs can be handled and parsed to extract the relevant information, how such log handling can be designed to work automatically, how to define and implement metrics to tag critical states of the service, how to correlate StoRM events with external services events, and ultimately how to contribute to the future CNAF-wide predictive maintenance system.
Initial results in this activity are presented and discussed. Furthermore, a mention to ongoing complementary work at the CNAF center is also mentioned
Inverted CERN School of Computing 2024
No full textThis 2-hour course will teach you how to use Git beyond the basic add, commit, push routine. We'll consolidate core concepts and introduce powerful commands like switch, restore, rebase, and reset. We will also discuss the differences between rebasing vs. merging, and explore advanced admin tool such as filter-repo and hooks.
The hands-on exercises will reinforce your learning, focusing on mastering rebasing techniques in a dedicated practice repository. Optional advanced exercises will teach how to write your own git hooks, and use filter-repo to alter the history of the repository
Big data analytics towards predictive maintenance at the INFN-CNAF computing centre
No full textLa Fisica delle Alte Energie (HEP) è da lungo tra i precursori nel gestire e processare enormi dataset scientifici e nell'operare alcuni tra i più grandi data centre per applicazioni scientifiche.
HEP ha sviluppato una griglia computazionale (Grid) per il calcolo al Large Hadron Collider (LHC) del CERN di Ginevra, che attualmente coordina giornalmente le operazioni di calcolo su oltre 800k processori in 170 centri di calcolo e gestendo mezzo Exabyte di dati su disco distribuito in 5 continenti.
Nelle prossime fasi di LHC, soprattutto in vista di Run-4, il quantitativo di dati gestiti dai centri di calcolo aumenterà notevolmente.
In questo contesto, la HEP Software Foundation ha redatto un Community White Paper (CWP) che indica il percorso da seguire nell'evoluzione del software moderno e dei modelli di calcolo in preparazione alla fase cosiddetta di High Luminosity di LHC. Questo lavoro ha individuato in tecniche di Big Data Analytics un enorme potenziale per affrontare le sfide future di HEP.
Uno degli sviluppi riguarda la cosiddetta Operation Intelligence, ovvero la ricerca di un aumento nel livello di automazione all'interno dei workflow. Questo genere di approcci potrebbe portare al passaggio da un sistema di manutenzione reattiva ad uno, più evoluto, di manutenzione predittiva o addirittura prescrittiva.
La tesi presenta il lavoro fatto in collaborazione con il centro di calcolo dell'INFN-CNAF per introdurre un sistema di ingestione, organizzazione e processing dei log del centro su una piattaforma di Big Data Analytics unificata, al fine di prototipizzare un modello di manutenzione predittiva per il centro. Questa tesi contribuisce a tale progetto con lo sviluppo di un algoritmo di clustering dei messaggi di log basato su misure di similarità tra campi testuali, per superare il limite connesso alla verbosità ed eterogeneità dei log raccolti dai vari servizi operativi 24/7 al centro
A Cloud-Edge Orchestration Platform for the Innovative Industrial Scenarios of the IoTwins Project
No full textThe concept of digital twins has growing more and more interest not only in the academic field but also among industrial environments thanks to the fact that the Internet of Things has enabled its cost-effective implementation. Digital twins (or digital models) refer to a virtual representation of a physical product or process that integrate data from various sources such as data APIs, historical data, embedded sensors and open data, giving to the manufacturers an unprecedented view into how their products are performing. The EU-funded IoTwins project plans to build testbeds for digital twins in order to run real-time computation as close to the data origin as possible (e.g., IoT Gateway or Edge nodes), and whilst batch-wise tasks such as Big Data analytics and Machine Learning model training are advised to run on the Cloud, where computing resources are abundant. In this paper, the basic concepts of the IoTwins project, its reference architecture, functionalities and components have been presented and discussed
Development and optimization of the control software for a mobile computed tomography system for cultural heritage.
No full textIn quest’elaborato sono descritti l’ottimizzazione e lo sviluppo del software di controllo di un apparato tomografico con sorgente di raggi X per analisi nel campo di Beni Artistici e Culturali. In particolare, il lavoro è stato effettuato sul software preesistente di un sistema mobile in uso presso il Dipartimento di Fisica e Astronomia per indagini tomografiche. Il sistema, sviluppato nell’arco di più anni, consiste di un tubo a raggi X, un detector flat-panel e una tavola rotativa per la tomografia. Tre assi traslazionali consentono il movimento di detector e sorgente, ottenendo un'area scansionabile di 1,5×1,5 m². Il software di controllo si occupa dell’intero processo di acquisizione: gestisce il movimento degli assi, effettua la rotazione della tavola che sostiene l’oggetto durante la tomografia e controlla la scheda di acquisizione in comunicazione con il detector per la cattura delle immagini. Con l’upgrade sviluppato in questo lavoro vengono introdotte diverse routine automatizzate e una più comoda gestione delle regioni di interesse per la scansione radio-tomografica, con lo scopo di alleggerire il carico dell’operatore e ridurre i tempi di acquisizione. Il lavoro di tesi si conclude con un’indagine presso Palazzo Vecchio a Firenze in cui sono state effettuate analisi radiografiche e tomografiche di una serie di dipinti su tavola attribuiti in buona parte al Pontormo. In quest’occasione il software aggiornato è stato testato sul campo per verificarne la praticità e l’efficienza delle nuove funzioni. L’esperienza ha messo in evidenza alcuni problemi e carenze del software e del sistema stesso che suggeriscono l’opportunità di un certo numero di aggiornamenti e di una eventuale riscrittura del codice. Nonostante ciò, l’automatizzazione delle operazioni di acquisizione radiografica e tomografica si è rivelata efficace, riducendo il numero di interventi manuali richiesti e con essi il tempo necessario per l’analisi stessa
Inverted CERN School of Computing 2024
No full textThis 2-hour course will teach you how to use Git beyond the basic add, commit, push routine. We'll consolidate core concepts and introduce powerful commands like switch, restore, rebase, and reset. We will also discuss the differences between rebasing vs. merging, and explore advanced admin tool such as filter-repo and hooks.
The hands-on exercises will reinforce your learning, focusing on mastering rebasing techniques in a dedicated practice repository. Optional advanced exercises will teach how to write your own git hooks, and use filter-repo to alter the history of the repository
