1,721,073 research outputs found
Evaluating CSP as a programming model to build distributed systems
No full textIn this paper we investigate the use of the CSP programming model for implementing distributed systems in an educational setting. For a practical example we choose a well-studied classic distributed system: a distributed hash table. We describe our implementation and compare it to a number of existing open-source implementations. We discuss a number of parameters for the example implementation, such as amount of code, potential for errors, and similar parameters.</p
MUMMERING Platform Idea's & ubiquitous data analysis
No full textThe growth from terabytes of 3D imaging data and soon approaching petabytes from material analysis has left the scientists involved with a set of challenges. In particular, the ability to efficiently analyze an ever growing collection of material tomography scans. The MUMMERING research project aims to solve this by providing ability to submit workflows to automate the process of analyzing the collected data. We explore and present our initial design thoughts in this endeavor. This includes a proposal to utilize the IDMC system developed at UCPH to provide an efficient method in terms of scheduling and execution of workflows. Beyond this mere exploration of thoughts about a potential solution for the MUMMERING project, this paper will also we introduce our initial work in providing ubiquitous access to the produced datasets. The aim here is to provide a simple API for loading/storing datasets during an image analysis, by providing this universal data access library (i,e. mig utils). With this we hope to help the non computer scientists involved in defining imaging analysis programs that easily either can executed locally during the experimentation phase or subsequently scheduled by a workflow scheduler.</p
Big data analysis with skeletons on SOFA
No full textThis paper explores how a skeleton based approach can be used to perform big data analysis. We introduce a restricted storage system based on blocks with a fixed maximum size. The storage design removes the residual data problem commonly found in storage systems, and enables processing on individual blocks. We then introduce a stream-oriented query system that can be used on top of the distributed storage system. The query system is built on a limited number of core operations. Each of the perform a specified function, such as filtering elements, but are skeleton operations where the programmer needs to fill in how to perform the operation. The operations are designed to allow splitting across the blocks in the storage system, giving concurrent execution while maintaining a completely sequential program description. To assist in understanding the data flow, we also introduce a graphical representation for each of the methods, enabling a visual expression of an algorithm. To evaluate the query system we implement a number of classic Big-Data queries and show how to implement them with code, and how the queries can be visualized with the graphical representation.</p
Concurrent composition of I/O redundancy behaviors in Go
No full textThe Go programming language defines simple I/O interfaces that any data type may implement. In this paper we introduce a Go package that allows arbitrary implementations of these interfaces to be composed into RAID-like redundant (and/or) high-performance striped arrays. The package also allows spares to be added for fail-over functionality. The package is focused on providing a highly available write setting that tolerates multiple failures but can always receive data as long as a single redundant path exists. This is achieved by allowing reads to become unavailable in the presence of failures that cannot be solved while the array is operating. The package is highly concurrent and parallelized and exploits the Go programming language's built-in light-weight concurrency features.</p
Emit - Communicating sequential processes in Ruby
No full textCSP is an algebra for reasoning about concurrent systems of processes. Being able to do so has become a necessity for computer scientists. Having to think about abstractions like mutexes and threads in practice can be cumbersome, complex, and erroneous. Ruby as a programming language has been described as fun to program in. It is however missing a CSP framework that it can call its own. Emit, which is presented in this paper, tries to mitigate this by providing such a CSP framework. As a CSP framework, Emit makes it easy to think about processes, channels, and communication. It is not yet feature complete, however comparing it to its nearest peer, PyCSP, shows good performance for the COMMSTIME benchmark, where Emit is 100 times faster.</p
Teaching concurrency:10 years of programming projects at UCPH
No full textWhile CSP is traditionally taught as an algebra, with a focus on definitions and proofs, it may also be presented as a style of programming, that is process-oriented programming. For the last decade University of Copenhagen (UCPH) has been teaching CSP as a mix of the two, including both the formal aspects and process-oriented programming. This paper summarized the work that has been made to make process-oriented programming relevant to students, through programming assignments where process orientation is clearly simpler than an equivalent solution in imperative programming style.</p
Bohrium.rb - The Ruby front end
No full textThe acceptance of Ruby in the scientific community lags a bit behind, partly because it is missing a good library for linear algebra and vector programming. It has a matrix class in its standard library, but its execution tends to be rather slow. Only a couple of actual scientific computing libraries like NumPy for Python exist for Ruby. In this paper we introduce a new library called Bohrium.rb. Bohrium.rb acts as a front end for the Bohrium framework, which generates and runs JIT-compiled OpenMP/OpenCL kernels. It currently supports Python/NumPy and C++, however as it is built of processes communicating hierarchically to each other, we can replace the front ends with new ones. This new Ruby front end is described with examples and is then compared to the standard library and an already established Ruby library Numo/Narray, where Bohrium.rb seems to be faster for still larger matrix calculations. This is also the trend we have seen in similar areas with Bohrium, being faster once its overhead has been amortized.</p
aPyCSP - Asynchronous PyCSP using python coroutines and asyncio
No full textPyCSP provides a CSP (Communicating Sequential Processes) based library for Python, where CSP processes can be executed as threads, operating system processes or greenlets. The main drawback of using threads and processes is that they limit the number of CSP processes that a program can use due to operating system restrictions and memory and synchronisation overheads. This overhead is reduced when using greenlets, but requires the use of an external library. aPyCSP is an experimental implementation of PyCSP that uses recent support for coroutines in Python and the asyncio library. This paper explores two implementation approaches for aPyCSP where the most recent version is shorter, simpler to read and understand and supports more functionality than the original version, including generic channel support and input and output guards. We believe some of the identified implementation techniques can be useful for other CSP libraries. The aPyCSP implementation is also faster and uses less memory per CSP process, allowing us to scale to 15 million processes on a computer with 64 GB memory.</p
Event-driven, collaborative and adaptive scientific workflows on the grid
No full textExponential growth in scientific data set sizes and corresponding computation needs, forces scientists and engineers to structure and automate experiments in workflows running on distributed architectures. In eScience the flows typically evolve gradually from intensive experimentation and often involve multiple participants from separate organisations. Thus, there's a need for infrastructures supporting such highly dynamic and collaborative workflows. Despite much attention to scientific workflows in recent years, most existing systems tend to be single-user top-down approaches, which are inherently best suited for static and fixed flows with all steps known up front. In this work we introduce a simple general rule-based model for event-driven workflows based on data change triggers. A bottom-up workflow approach, that enables a high level of automation and allows dynamically changing flows - with or without manual user interaction. It is realised with an implementation on top of the Minimum intrusion Grid (MiG), which helps de-couple workflow design from underlying execution concerns, and provides built-in collaboration and sharing across organisation boundaries. However, the model itself applies to much wider range of scenarios, and other such possible implementation methods are briefly outlined.</p
Towards automatic program specification using SME models
No full textThis paper introduces a method to simplify hardware modeling and verification there of in order for software programmers to, more easily, meet the demands of the growing embedded device industry. We describe a simple method for transpiling from the new SME Implementation Language into CSPM and using formal verification to verify properties within the generated program. We present a small example consisting of a seven segment display clock network and introduce how to verify the widths of the channels in the network.</p
- …
