1,720,996 research outputs found
The AstroHDF Effort
No full textHere we update the astronomy community on our effort to deal with the demands of ever-increasing astronomical data size and complexity, using the Hierarchical Data Format, version 5 (HDF5) format (Wise et al. 2011). NRAO, LOFAR and VAO have joined forces with The HDF Group to write an NSF grant, requesting funding to assist in the effort. This paper briefly summarizes our motivation for the proposed project, an outline of the project itself, and some of the material discussed at the ADASS Birds of a Feather (BoF) discussion. Topics of discussion included: community experiences with HDF5 and other file formats; toolsets which exist and/or can be adapted for HDF5; a call for development towards visualizing large (> 1 TB) image cubes; and, general lessons learned from working with large and complex data
Query Driven Visualization
No full textThe request driven way of deriving data in Astro-WISE is extended to a query driven way of visualization. This allows scientists to focus on the science they want to perform, because all administration of their data is automated. This can be done over an abstraction layer that enhances control and flexibility for the scientist
Astronomical data processing using SciQL, an SQL based query language for array data
Get PDFSciQL (pronounced as ‘cycle’) is a novel SQL-based array query language for scientific applications with both tables and arrays as first class citizens. SciQL lowers the entrance fee of adopting relational DBMS (RDBMS) in scientific domains, because it includes functionality often only found in mathematics software packages. In this paper, we demonstrate the usefulness of SciQL for astronomical data processing using examples from the Transient Key Project of the LOFAR radio telescope. In particular, how the LOFAR light-curve database of all detected sources can be constructed, by correlating sources across the spatial, frequency, time and polarisation domains
Query Driven Visualization
No full textThe request driven way of deriving data in Astro-WISE is extended to a query driven way of visualization. This allows scientists to focus on the science they want to perform, because all administration of their data is automated. This can be done over an abstraction layer that enhances control and flexibility for the scientist
Integration of the MUSE Software Pipeline into the Astro-WISE System
No full textWe discuss the current state of integrating the Mutli Unit Spectroscopic Explorer (hereafter: MUSE) software pipeline (Weilbacher et al. 2006) into the Astro-WISE system (Valentijn et al. 2007a; Vriend et al. 2012). MUSE is a future integral-field spectrograph for the VLT, consisting of 24 Integral Field Units (hereafter IFU). The MUSE data reduction pipeline is built using the Common Pipeline Library (CPL) provided by ESO. The Astro-WISE technology integrates data lineage, data persistence, distributed processing, and large file storage into an information system. To integrate the MUSE pipeline, its metadata is used to build persistent objects for storage in the Astro-WISE system. It is thought that this method can provide a convenient and quick method to implement future pipelines into Astro-WISE. Current work on the integration includes handling multiple IFUs, completing the pipeline integration, and use-case development
Astro-WISE Processing of Wide-field Images and Other Data
No full textAstro-WISE (Vriend et al. 2012) is the Astronomical Wide-field Imaging System for Europe (Valentijn et al. 2007). It is a scientific information system which consists of hardware and software federated over about a dozen institutes throughout Europe. It has been developed to exploit the ever increasing avalanche of data produced by astronomical surveys and data intensive scientific experiments in general. The demo explains the architecture of the Astro-WISE information system and shows the use of Astro-WISE interfaces. Wide-field astronomical images are derived from the raw image to the final catalog according to the user's request. The demo is based on the standard Astro-WISE guided tour, which can be accessed from the Astro-WISE website. The typical Astro-WISE data processing chain is shown, which can be used for data handling for a variety of different instruments, currently 14, including OmegaCAM, MegaCam, WFI, WFC, ACS/HST, etc
Target for LOFAR Long Term Archive:Architecture and Implementation
No full textThe LOFAR Long-Term Archive (LTA) is a multi-Petabyte scale data storage for the processed data of LOFAR telescope. We describe the adaptation of the WISE concept implemented by Target consortium for the LOFAR LTA and changes we introduced to it to accommodate LOFAR data. This paper describes an example of a new information system created on the basis of Astro-WISE for a wider range and scale of data
Astro-WISE Processing of Wide-field Images and Other Data
No full textAstro-WISE (Vriend et al. 2012) is the Astronomical Wide-field Imaging System for Europe (Valentijn et al. 2007). It is a scientific information system which consists of hardware and software federated over about a dozen institutes throughout Europe. It has been developed to exploit the ever increasing avalanche of data produced by astronomical surveys and data intensive scientific experiments in general. The demo explains the architecture of the Astro-WISE information system and shows the use of Astro-WISE interfaces. Wide-field astronomical images are derived from the raw image to the final catalog according to the user's request. The demo is based on the standard Astro-WISE guided tour, which can be accessed from the Astro-WISE website. The typical Astro-WISE data processing chain is shown, which can be used for data handling for a variety of different instruments, currently 14, including OmegaCAM, MegaCam, WFI, WFC, ACS/HST, etc
Astro-WISE for KiDS survey production and quality control
No full textThe Kilo Degree Survey (KiDS) is a 1500 square degree optical imaging survey with the recently commissioned OmegaCAM wide-field imager on the VLT Survey Telescope (VST). A suite of data products will be delivered to ESO and the community by the KiDS survey team. Spread over Europe, the KiDS team uses Astro-WISE to collaborate efficiently and pool hardware resources. In Astro-WISE the team shares, calibrates and archives all survey data. The data-centric architectural design realizes a dynamic ‘live archive’ in which new KiDS survey products of improved quality can be shared with the team and eventually the full astronomical community in a flexible and controllable manner
Integration of the MUSE Software Pipeline into the Astro-WISE System
No full textWe discuss the current state of integrating the Mutli Unit Spectroscopic Explorer (hereafter: MUSE) software pipeline (Weilbacher et al. 2006) into the Astro-WISE system (Valentijn et al. 2007a; Vriend et al. 2012). MUSE is a future integral-field spectrograph for the VLT, consisting of 24 Integral Field Units (hereafter IFU). The MUSE data reduction pipeline is built using the Common Pipeline Library (CPL) provided by ESO. The Astro-WISE technology integrates data lineage, data persistence, distributed processing, and large file storage into an information system. To integrate the MUSE pipeline, its metadata is used to build persistent objects for storage in the Astro-WISE system. It is thought that this method can provide a convenient and quick method to implement future pipelines into Astro-WISE. Current work on the integration includes handling multiple IFUs, completing the pipeline integration, and use-case development
- …
