53 research outputs found
Grid applications for the BaBar experiment
No full textThis paper discusses the use of e-Science Grid in providing
computational resources for modern international High Energy
Physics (HEP) experiments. We investigate the suitability of
the current generation of Grid software to provide the necessary
resources to perform large-scale simulation of the experiment and
analysis of data in the context of multinational collaboration
Exploring the virtues of XRootD5: Declarative API
No full textAcross the years, being the backbone of numerous data management solutions used within the WLCG collaboration, the XRootD framework and protocol became one of the most important building blocks for storage solutions in the High Energy Physics (HEP) community. The latest big milestone for the project, release 5, introduced multitude of architectural improvements and functional enhancements, including the new client side declarative API, which is the main focus of this study. In this contribution, we give an overview of the new client API and we discuss its motivation and its positive impact on overall software quality (coupling, cohesion), readability and composability
XRootD S3 Gateway for WLCG Storage
No full textThe S3 Gateway is a server based application that provides a bridge between protocols and security used in the HEP community to S3 protocols and its associated security model. This allows the use of common copy tools based on the HEP security models to store or download data from S3 based storage. This storage can reside in a public or private cloud. This paper details the motivation for implementing such a service and how it can address certain problem when dealing with storage only accessible via S3 protoco
Scalla: Structured Cluster Architecture for Low Latency Access
No full textScalla is a distributed low-latency file access system that incorporates novel techniques that minimize latency and maximize scalability over a large distributed system with a distributed namespace. Scalla's techniques have shown to be effective in nearly a decade of service for the high-energy physics community using commodity hardware and interconnects. We describe the two components used in Scalla that are instrumental in its ability to provide low-latency, fault-tolerant name resolution and load distribution, and enable its use as a high-throughput, low-latency communication layer in the Qserv system, the Large Synoptic Survey Telescope's (LSST's) prototype astronomical query system. Scalla arguably exceeded its three main design objectives: low latency, scaling, and recoverability. In retrospect, these objectives were met using a simple but effective design. Low latency was met by uniformly using linear or constant time algorithms in all high-use paths, avoiding locks whenever possible, and using compact data structures to maximize the memory caching efficiency. Scaling was achieved by architecting the system as a 64-ary tree. Nodes can be added easily and as the number of nodes increases, search performance increases at an exponential rate. Recoverability is inherent in that no permanent state information is maintained and whatever state information is needed it can be quickly constructed or reconstructed in real time. This allows dynamic changes in a cluster of servers with little impact on over-all performance or usability. Today, Scalla is being deployed in environments and for uses that were never conceived in 2001. This speaks well for the systems adaptability but the underlying reason is that the system can meet its three fundamental objectives at the same time
XRootD S3 Gateway for WLCG Storage
No full textThe S3 Gateway is a server based application that provides a bridge between protocols and security used in the HEP community to S3 protocols and its associated security model. This allows the use of common copy tools based on the HEP security models to store or download data from S3 based storage. This storage can reside in a public or private cloud. This paper details the motivation for implementing such a service and how it can address certain problem when dealing with storage only accessible via S3 protocol
Computing in High Energy and Nuclear Physics (CHEP) 2012
No full textFor more than a year, the ATLAS Western Tier 2 (WT2) at SLAC National Accelerator has been successfully operating a two tiered storage system based on Xrootd's flexible cross-cluster data placement framework, the File Residency Manager. The architecture allows WT2 to provide both, high performance storage at the higher tier to ATLAS analysis jobs, as well as large, low cost disk capacity at the lower tier. Data automatically moves between the two storage tiers based on the needs of analysis jobs and is completely transparent to the jobs
Recommended from our members
Data Access Performance Through Parallelization and Vectored Access: Some Results
No full textHigh Energy Physics data processing and analysis applications typically deal with the problem of accessing and processing data at high speed. Recent studies, development and test work have shown that the latencies due to data access can often be hidden by parallelizing them with the data processing, thus giving the ability to have applications which process remote data with a high level of efficiency. Techniques and algorithms able to reach this result have been implemented in the client side of the Scalla/xrootd system, and in this contribution we describe the results of some tests done in order to compare their performance and characteristics. These techniques, if used together with multiple streams data access, can also be effective in allowing to efficiently and transparently deal with data repositories accessible via a Wide Area Network
- …
