95 research outputs found
Ten quick tips for bioinformatics analyses using an Apache Spark distributed computing environment
No full textSome scientific studies involve huge amounts of bioinformatics : data that cannot be analyzed on personal computers usually employed by researchers for day-to-day activities but rather necessitate effective computational infrastructures that can work in a distributed way. For this purpose, distributed computing systems have become useful tools to analyze large amounts of bioinformatics data and to generate relevant results on virtual environments, where software can be executed for hours or even days without affecting the personal computer or laptop of a researcher. Even if distributed computing resources have become pivotal in multiple bioinformatics laboratories, often researchers and students use them in the wrong ways, making mistakes that can cause the distributed computers to underperform or that can even generate wrong outcomes. In this context, we present here ten quick tips for the usage of Apache Spark distributed computing systems for bioinformatics analyses: ten simple guidelines that, if taken into account, can help users avoid common mistakes and can help them run their bioinformatics analyses smoothly. Even if we designed our recommendations for beginners and students, they should be followed by experts too. We think our quick tips can help anyone make use of Apache Spark distributed computing systems more efficiently and ultimately help generate better, more reliable scientific results
Overcoming the Obfuscation of Java Programs by Identifier Renaming
No full textDecompilation is the process of translating object code to source code and is usually the first step towards the reverse-engineering
of an application. Many obfuscation techniques and tools have been developed, with the aim of modifying a program, such that its
functionalities are preserved, while its understandability is compromised for a human reader or the decompilation is made unsuccessful.
Some approaches rely on malicious identifiers renaming, i.e., on the modification of the program identifiers in order to introduce
confusion and possibly prevent the decompilation of the code.
In this work we introduce a new technique to overcome the obfuscation of Java programs by identifier renaming. Such a technique
relies on the intelligent modification of identifiers in Java bytecode.
We present a new software tool which implements our technique and allows the processing of an obfuscated program in order to
rename the identifiers as required by our technique. Moreover, we show how to use the existing tools to provide a partial implementation
of the technique we propose.
Finally, we discuss the feasibility of our approach by showing how to contrast the obfuscation techniques based on malicious
identifier renaming recently presented in literature
A Distributed Alignment-free Pipeline for Human SNPs Genotyping
No full textIdentification of known genetic traits and disease-related variants within an individual requires a fundamental task: genotyping a set of variants from a database. However, the efficiency of this process is challenged by the growing volume of sequencing data and variant databases. At such scale, even the fastest genotyping tool available can deliver a result in a time that is unacceptable.
To address this issue, we present SparkGeno, the first known distributed alignment-free pipeline for genotyping the particular case of Single Nucleotide Polymorphisms (SNPs). Building upon a distributed reformulation of traditional alignment-free genotyping pipelines, and using the Apache Spark framework, we introduce several optimizations to further enhance the performance of our code in a distributed environment. Our pipeline comes in two versions that employ different data structures, making them suitable for processing datasets featuring different numbers of SNPs.
Moreover, we present the results of an experimental analysis on widely studied datasets to assess how relying on distributed computing allows for a fast, accurate and scalable solution for large-scale genotyping. Finally, we also report the results of an additional experiment for validating the effectiveness of the signature-based approach we used to perform genotyping.
Our results show that SparkGeno, when run on a distributed system, is able to genotype variants from whole-genome sequencing data orders of growth faster than existing tools, in a scalable manner in terms of the number of the available computational units. This makes SparkGeno a promising solution for large-scale genotyping applications, such as precision medicine and population-scale studies
DISCERN: A collaborative visualization system for learning cryptographic protocols
No full textIn this paper we propose a novel approach to the learning of cryptographic protocols, based on a collaborative role-based visualization system, DISCERN, that helps students to understand a protocol by actively engaging them in a simulation of its execution. In DISCERN, each student shares a visual exemplification of a real-world scenario with other students and impersonates one of the parties involved in the execution of a protocol. Students may take the role of legal or malicious parties and are provided with primitives that are useful for the implementation of several protocols. To achieve a certain security goal correctly, legal parties have to collaborate and carefully execute the steps required by the implemented protocol in the correct order. If any error is made, the security of the protocol is exposed to the threats coming from other students impersonating malicious parties. The entire process is run under the supervision of the teacher
Alignment-free Genomic Analysis via a Big Data Spark Platform
No full textMotivation: Alignment-free distance and similarity functions (AF functions, for short) are a well-established alternative to pairwise and multiple sequence alignments for many genomic, metagenomic and epigenomic tasks. Due to data-intensive applications, the computation of AF functions is a Big Data problem, with the recent literature indicating that the development of fast and scalable algorithms computing AF functions is a high-priority task. Somewhat surprisingly, despite the increasing popularity of Big Data technologies in computational biology, the development of a Big Data platform for those tasks has not been pursued, possibly due to its complexity.Results: We fill this important gap by introducing FADE, the first extensible, efficient and scalable Spark platform for alignment-free genomic analysis. It supports natively eighteen of the best performing AF functions coming out of a recent hallmark benchmarking study. FADE development and potential impact comprises novel aspects of interest. Namely, (i) a considerable effort of distributed algorithms, the most tangible result being a much faster execution time of reference methods like MASH and FSWM; (ii) a software design that makes FADE user-friendly and easily extendable by Spark non-specialists; (iii) its ability to support data- and compute-intensive tasks. About this, we provide a novel and much needed analysis of how informative and robust AF functions are, in terms of the statistical significance of their output. Our findings naturally extend the ones of the highly regarded benchmarking study, since the functions that can really be used are reduced to a handful of the eighteen included in FADE
FASTA/Q data compressors for MapReduce-Hadoop genomics: space and time savings made easy
No full textBackground: Storage of genomic data is a major cost for the Life Sciences, effectively addressed via specialized data compression methods. For the same reasons of abundance in data production, the use of Big Data technologies is seen as the future for genomic data storage and processing, with MapReduce-Hadoop as leaders. Somewhat surprisingly, none of the specialized FASTA/Q compressors is available within Hadoop. Indeed, their deployment there is not exactly immediate. Such a State of the Art is problematic. Results: We provide major advances in two different directions. Methodologically, we propose two general methods, with the corresponding software, that make very easy to deploy a specialized FASTA/Q compressor within MapReduce-Hadoop for processing files stored on the distributed Hadoop File System, with very little knowledge of Hadoop. Practically, we provide evidence that the deployment of those specialized compressors within Hadoop, not available so far, results in better space savings, and even in better execution times over compressed data, with respect to the use of generic compressors available in Hadoop, in particular for FASTQ files. Finally, we observe that these results hold also for the Apache Spark framework, when used to process FASTA/Q files stored on the Hadoop File System. Conclusions: Our Methods and the corresponding software substantially contribute to achieve space and time savings for the storage and processing of FASTA/Q files in Hadoop and Spark. Being our approach general, it is very likely that it can be applied also to FASTA/Q compression methods that will appear in the future. Availability: The software and the datasets are available at https://github.com/fpalini/fastdoop
Data structures resilient to memory faults: An experimental study of dictionaries
No full textWe address the problem of implementing data structures resilient to memory faults, which may arbitrarily corrupt memory locations. In this framework, we focus on the implementation of dictionaries and perform a thorough experimental study using a testbed that we designed for this purpose. Our main discovery is that the best-known (asymptotically optimal) resilient data structures have very large space overheads. More precisely, most of the space used by these data structures is not due to key storage. This might not be acceptable in practice, since resilient data structures are meant for applications where a huge amount of data (often of the order of terabytes) has to be stored. Exploiting techniques developed in the context of resilient (static) sorting and searching, in combination with some new ideas, we designed and engineered an alternative implementation, which, while still guaranteeing optimal asymptotic time and space bounds, performs much better in terms of memory without compromising the time efficiency. © 2013 ACM
The price of resiliency: a case study on sorting with memory faults
No full textWe address the problem of sorting in the presence of faults that may arbitrarily corrupt memory locations, and investigate the impact of memory faults both on the correctness and on the running times of mergesort-based algorithms. To achieve this goal, we develop a software testbed that simulates different fault injection strategies, and perform a thorough experimental study using a combination of several fault parameters. Our experiments give evidence that simple-minded approaches to this problem are largely impractical, while the design of more sophisticated resilient algorithms seems really worth the effort. Another contribution of our computational study is a carefully engineered implementation of a resilient sorting algorithm, which appears robust to different memory fault patterns
CATAI: Concurrent Algorithms and Data Types Animation over the Internet
No full textIn this paper we present Catai (for Concurrent Algorithms and data Types Animation over the Internet), an algorithm animation distributed system. Catai adopts an object-oriented animation approach that allows a programmer with an algorithmic background to transparently and efficiently animate a target algorithm. Multiple users can simultaneously watch and interact with a same animation made with Catai using a light-weighted Java animation client. We believe this to be a good compromise between two different viewpoints: the programmer's perspective, which typically includes the goal of animating efficiently and unobtrusively a given algorithmic code, and the user's perspective, which can benefit from interactive, easy-to-use, distributed and cooperative interfaces
- …
