1,720,976 research outputs found
A Distributed Alignment-free Pipeline for Human SNPs Genotyping
Get PDFIdentification of known genetic traits and disease-related variants within an individual requires a fundamental task: genotyping a set of variants from a database. However, the efficiency of this process is challenged by the growing volume of sequencing data and variant databases. At such scale, even the fastest genotyping tool available can deliver a result in a time that is unacceptable.
To address this issue, we present SparkGeno, the first known distributed alignment-free pipeline for genotyping the particular case of Single Nucleotide Polymorphisms (SNPs). Building upon a distributed reformulation of traditional alignment-free genotyping pipelines, and using the Apache Spark framework, we introduce several optimizations to further enhance the performance of our code in a distributed environment. Our pipeline comes in two versions that employ different data structures, making them suitable for processing datasets featuring different numbers of SNPs.
Moreover, we present the results of an experimental analysis on widely studied datasets to assess how relying on distributed computing allows for a fast, accurate and scalable solution for large-scale genotyping. Finally, we also report the results of an additional experiment for validating the effectiveness of the signature-based approach we used to perform genotyping.
Our results show that SparkGeno, when run on a distributed system, is able to genotype variants from whole-genome sequencing data orders of growth faster than existing tools, in a scalable manner in terms of the number of the available computational units. This makes SparkGeno a promising solution for large-scale genotyping applications, such as precision medicine and population-scale studies
Alignment-free Genomic Analysis via a Big Data Spark Platform
Get PDFMotivation: Alignment-free distance and similarity functions (AF functions, for short) are a well-established alternative to pairwise and multiple sequence alignments for many genomic, metagenomic and epigenomic tasks. Due to data-intensive applications, the computation of AF functions is a Big Data problem, with the recent literature indicating that the development of fast and scalable algorithms computing AF functions is a high-priority task. Somewhat surprisingly, despite the increasing popularity of Big Data technologies in computational biology, the development of a Big Data platform for those tasks has not been pursued, possibly due to its complexity.Results: We fill this important gap by introducing FADE, the first extensible, efficient and scalable Spark platform for alignment-free genomic analysis. It supports natively eighteen of the best performing AF functions coming out of a recent hallmark benchmarking study. FADE development and potential impact comprises novel aspects of interest. Namely, (i) a considerable effort of distributed algorithms, the most tangible result being a much faster execution time of reference methods like MASH and FSWM; (ii) a software design that makes FADE user-friendly and easily extendable by Spark non-specialists; (iii) its ability to support data- and compute-intensive tasks. About this, we provide a novel and much needed analysis of how informative and robust AF functions are, in terms of the statistical significance of their output. Our findings naturally extend the ones of the highly regarded benchmarking study, since the functions that can really be used are reduced to a handful of the eighteen included in FADE
FASTA/Q data compressors for MapReduce-Hadoop genomics: space and time savings made easy
Get PDFBackground: Storage of genomic data is a major cost for the Life Sciences, effectively addressed via specialized data compression methods. For the same reasons of abundance in data production, the use of Big Data technologies is seen as the future for genomic data storage and processing, with MapReduce-Hadoop as leaders. Somewhat surprisingly, none of the specialized FASTA/Q compressors is available within Hadoop. Indeed, their deployment there is not exactly immediate. Such a State of the Art is problematic. Results: We provide major advances in two different directions. Methodologically, we propose two general methods, with the corresponding software, that make very easy to deploy a specialized FASTA/Q compressor within MapReduce-Hadoop for processing files stored on the distributed Hadoop File System, with very little knowledge of Hadoop. Practically, we provide evidence that the deployment of those specialized compressors within Hadoop, not available so far, results in better space savings, and even in better execution times over compressed data, with respect to the use of generic compressors available in Hadoop, in particular for FASTQ files. Finally, we observe that these results hold also for the Apache Spark framework, when used to process FASTA/Q files stored on the Hadoop File System. Conclusions: Our Methods and the corresponding software substantially contribute to achieve space and time savings for the storage and processing of FASTA/Q files in Hadoop and Spark. Being our approach general, it is very likely that it can be applied also to FASTA/Q compression methods that will appear in the future. Availability: The software and the datasets are available at https://github.com/fpalini/fastdoop
PNU Spoofing: a menace for biometrics authentication systems?
No full textThe Pixel Non-Uniformity noise (PNU noise, for short) is a characteristic noise of digital camera sensors that has been originally used a mean to perform Source Camera Identification (SCI), that is, to identify the digital camera that has been used to take an image under scrutiny. Actually, its usage has been extended to other relevant application domains, such as carrying out sensor identification in iris biometrics, to resolve the integrity of a biometric authentication system, in health monitoring systems based on bio-signal processing, and to prevent health fraud scams. As a consequence of this popularity, several counter-forensics techniques have been proposed in the scientific literature to deceive this identification approach and, thus, putting at risk its effectiveness. The goal of this paper is to experiment with some of these counter-forensics techniques and to assess whether they are able or not to deceive the PNU-based identification process. We focus our attention on the particular case where a target image is modified so as to appear as been taken by a different device than the original one (i.e., spoofing). The results of our experiments show that this kind of spoofing is apparently successful in deceiving the PNU-based identification process. However, we also show that the spoofed images retain traces of their originating camera. By leveraging on this information, it is still possible to trace back the device used to take that image. This finding may have a deep impact on the usage of Pixel Non Uniformity (PNU) noise for identification tasks, such as the ones carried out by biometric recognition and monitoring systems, and on the possibility to really cheat them by means of counter-forensics techniques
Using software visualization to support the teaching of distributed programming
No full textIn this paper, we introduce MARVEL, a system designed to simplify the teaching
of MapReduce, a popular distributed programming paradigm, through software
visualization. At its core, it allows a teacher to describe and recreate a MapReduce
application by interactively requesting, through a graphical interface, the execution
of a sequence of MapReduce transformations that target an input dataset. Then, the
execution of each operation is illustrated on the screen by playing an appropriate
graphical animation stage, highlighting aspects related to its distributed nature. The
sequence of all animation stages, played back one after the other in a sequential
order, results in a visualization of the whole algorithm. The content of the resulting visualization is not simulated or fctitious, but refects the real behavior of the
requested operations, thanks to the adoption of an architecture based on a real
instance of a distributed system running on Apache Spark. On the teacher’s side, it
is expected that by using MARVEL he/she will spend less time preparing materials
and will be able to design a more interactive lesson than with electronic slides or a
whiteboard. To test the efectiveness of the proposed approach on the learner side,
we also conducted a small scientifc experiment with a class of volunteer students
who formed a control group. The results are encouraging, showing that the use of
software visualization guarantees students a learning experience at least equivalent
to that of conventional approaches
Correction to: FASTA/Q data compressors for MapReduce-Hadoop genomics: space and time savings made easy
Get PDFFollowing publication of the original article [1], the authors identified that the affiliations of Giuseppe Cattaneo and Raffaele Giancarlo were interchanged. The correct affiliations are given below.
The correct affiliation of Giuseppe Cattaneo is:
2Dipartimento di Informatica, Università di Salerno, Fisciano, Italy.
The correct affiliation of Raffaele Giancarlo is:
3Dipartimento di Matematica ed Informatica, Università di Palermo, Palermo, Italy.
The original article [1] has been corrected
Ten quick tips for bioinformatics analyses using an Apache Spark distributed computing environment
Get PDFSome scientific studies involve huge amounts of bioinformatics : data that cannot be analyzed on personal computers usually employed by researchers for day-to-day activities but rather necessitate effective computational infrastructures that can work in a distributed way. For this purpose, distributed computing systems have become useful tools to analyze large amounts of bioinformatics data and to generate relevant results on virtual environments, where software can be executed for hours or even days without affecting the personal computer or laptop of a researcher. Even if distributed computing resources have become pivotal in multiple bioinformatics laboratories, often researchers and students use them in the wrong ways, making mistakes that can cause the distributed computers to underperform or that can even generate wrong outcomes. In this context, we present here ten quick tips for the usage of Apache Spark distributed computing systems for bioinformatics analyses: ten simple guidelines that, if taken into account, can help users avoid common mistakes and can help them run their bioinformatics analyses smoothly. Even if we designed our recommendations for beginners and students, they should be followed by experts too. We think our quick tips can help anyone make use of Apache Spark distributed computing systems more efficiently and ultimately help generate better, more reliable scientific results
Scheduling K-mers Counting in a Distributed Environment
No full textAlignment-free algorithms are used in bioinformatics to efficiently evaluate the similarity between pairs of genomic sequences. They work by extracting and aggregating features from the sequences under study and, then, by comparing them using alignment-free functions. When working on large collections of huge sequences, it is possible to improve the performance of these algorithms by executing the extraction and the aggregation steps, in parallel, across the computing nodes of a distributed system. In this work, we address the problem of finding the optimal schedule to use for assigning to computing nodes the features to be aggregated, in order to minimize the maximum aggregation time. For this purpose, we consider one exact mathematical programming approach and two approximated ones, based on the Longest Processing Time heuristic. These have been implemented using the Gurobi solver, and compared to the algorithm used by the Spark distributed computing framework for assigning tasks to computing nodes. The experiments have been performed on some large collections of genomic sequences well-known in literature. The results show that the proposed approaches perform favourably with respect to the scheduling strategy used by Spark, in terms of quality of the solution. In particular the exact approach, run up to the time limit, allows to reduce the makespan up to 77.11%, when considering the largest instance tested. However, the required computational time of the exact approach is not compatible with its online application. The approximated approaches appear more promising in terms of computational time, while providing good quality solutions
- …
