16 research outputs found
DESY Report
No full textStatus and plans at the DESY computer center, the T2 and the national analysis facilit
ILC Computing Model
No full textThe computing model and computing challenges of the international linear collider
Containerized Batch System Monitoring
No full textRunning a batch system for grid jobs and for local users, we investigate a generic solution to monitor the resource usages of jobs. We extend a standard toolkit for monitoring container performance metrics also for non-containerized applications, so that we can make easy use of a popular industry solution. By concentrating on the basic kernel features also used by containers frameworks, we envisage to use the same tools on non-containerized batch systems as well as container orchestrators.
For deployment, we encapsulate the tools as Singularity containers and distribute them via CVMFS
Containerized Batch System Monitoring
Get PDFRunning a batch system for grid jobs and for local users, we investigate a generic solution to monitor the resource usages of jobs. We extend a standard toolkit for monitoring container performance metrics also for non-containerized applications, so that we can make easy use of a popular industry solution. By concentrating on the basic kernel features also used by containers frameworks, we envisage to use the same tools on non-containerized batch systems as well as container orchestrators.
For deployment, we encapsulate the tools as Singularity containers and distribute them via CVMFS
dCache: Visualization of dCache accounting information with state-of-the-art Data Analysis Tools.
No full textOver the previous years, storage providers in scientific infrastructures were facing a significant change in the usage profile of their resources. While in the past, a small number of experiment frameworks were accessing those resources in a coherent manner, now, a large amount of small groups or even individuals request access in a completely chaotic way. Moreover, scientific laboratories have been recently forced to provide detailed accounting information for their communities and individuals. Another consequence of the chaotic access profiles is the difficulty, for often rather small operating teams, to detect malfunctions in extremely complex storage systems, composed of a large variety of different hardware components. Although information about usage and possible malfunction is available in the corresponding log and billing files, the sheer amount of collected meta data makes it extremely difficult to be handled or interpreted. Simply the dCache production instances at DESY are producing Gigabytes of meta data per day. To cope with those pressing issues, DESY has been evaluating and put into production a Big Data processing tool, enabling our operation team to analyze log and billing information by providing a configurable and easy to interpret visualization of that data. This presentation will demonstrate how DESY built a real-time monitoring system, visualizing dCache billing files and providing an intuitive and simple to operate Web interface, using ElasticSearch, Logstash and Kibana
Analyzing storage access data with Apache-Spark and Jupiter notebooks
No full textRunning a data center is never a trivial job. In addition to daily routine tasks, service operation teams have to provide a meaningful information for monitoring, reporting and access pattern analytic. The dCache production instances at DESY, produce gigabytes of billing files per day. However, with a help of modern BigData analysis tools like Apache-Spark and Jupiter notebooks such task can be easily achieved. Moreover, the tool set for storage access analysts can be shared with scientific community making it re-usable computational resource as well as shared knowledg
Integration of Grid and Local Batch Resources at DESY
No full textAs one of the largest resource centres DESY has to support differing work flows of users from various scientific backgrounds. Users can be for one HEP experiments in WLCG or Belle II as well as local HEP users but also physicists from other fields as photon science or accelerator development. By abandoning specific worker node setups in favour of generic flat nodes with middleware resources provided via CVMFS, we gain flexibility to subsume different use cases in a homogeneous environment.Grid jobs and the local batch system are managed in a HTCondor based setup, accepting pilot, user and containerized jobs. The unified setup allows dynamic re-assignment of resources between the different use cases. Monitoring is implemented on global batch system metrics as well as on a per job level utilizing corresponding cgroup information
Data Challenges in Photon Science
No full textThis presentation will give an overview about the data life cycle in photon science and the resulting challenges on data handling. The discussed topics include data taking, analysis, storage and archival. The technical realization is illustrated on an example of currently developed infrastructures for synchrotrons and free electron lasers
ASAP3 - New Data Taking and Analysis Infrastructure for PETRA III
No full textData taking and analysis infrastructures in HEP (High Energy Physics) have evolved during many years to a well known problem domain. In contrast to HEP, third generation synchrotron light sources, existing and upcoming free electron lasers are confronted with an explosion in data rates driven primarily by recent developments in 2D pixel array detectors. The next generation of detectors will produce data in the region upwards of 50 Gbytes per second. At synchrotrons, data was traditionally taken away by users following data taking using portable media. This will clearly not scale at all.We present first experiences of our new architecture and underlying services based on results taken from the resumption of data taking in April 2015. Technology choices were undertaking over a period of twelve month. The work involved a close collaboration between central IT, beamline controls, and beamline support staff. In addition a cooperation was established between DESY IT and IBM to include industrial research and development experience and skills.Our approach integrates HPC technologies for storage systems and protocols. In particular, our solution uses a single file-system instance with a multiple protocol access, while operating within a single namespace
