TU Wien Research Data
Not a member yet
512 research outputs found
Sort by
Mag ich unglúck nit widerston (Jud_1523-2_n11) Audio recording
No full text<h1>Audio recording of a lute piece from the E-LAUTE project</h1><h2>Overview</h2><p>This dataset contains an audio recording of the piece "Mag ich unglúck nit widerston", a 16th century lute music piece originally notated in lute tablature, created as part of the E-LAUTE project (<a href="https://e-laute.info/">https://e-laute.info/</a>). The recording preserves and makes historical lute music from the German-speaking regions during 1450-1550 accessible.</p><p>The recording is based on the work with the title "Mag ich unglúck nit widerston" and the id "Jud_1523-2_n11" in the e-lautedb. It is found on the page(s) or folio(s) 18v in the source "1.5.2.3. Ain schone kunstliche vnderweisung" with the source-id "Jud_1523-2".</p><p>The original source and multiple transcriptions of the work can be found on the E-LAUTE platform: <a href="https://edition.onb.ac.at/fedora/objects/o:lau.Jud_1523-2/methods/sdef:TEI/get?mode=n11" target="_blank">https://edition.onb.ac.at/fedora/objects/o:lau.Jud_1523-2/methods/sdef:TEI/get?mode=n11</a>.</p><p>Links to the source: <a href="http://data.onb.ac.at/rec/AC09185338" target="_blank">http://data.onb.ac.at/rec/AC09185338</a>, <a href="https://opac.rism.info/rism/Record/rism990032736" target="_blank">https://opac.rism.info/rism/Record/rism990032736</a>, <a href="https://gateway-bayern.de/VD16+J+1031" target="_blank">https://gateway-bayern.de/VD16+J+1031</a>, .</p><h2>Dataset Contents</h2><p>This dataset includes:</p><ul><li><strong>Audio file</strong>: An audio recording of the lute piece in .wav format</li> <li><strong>Metadata file</strong>: A metadata file with detailed information about the recording in .json format</li></ul><h2>About the E-LAUTE Project</h2><p><strong>E-LAUTE: Electronic Linked Annotated Unified Tablature Edition - The Lute in the German-Speaking Area 1450-1550</strong></p><p>The E-LAUTE project creates innovative digital editions of lute tablatures from the German-speaking area between 1450 and 1550. This interdisciplinary "open knowledge platform" combines musicology, music practice, music informatics, and literary studies to transform traditional editions into collaborative research spaces.</p><p>For more information, visit the project website: <a href="https://e-laute.info/">https://e-laute.info/</a></p>
Von edler art (New_1536-7_n49) Audio recording
No full text<h1>Audio recording of a lute piece from the E-LAUTE project</h1><h2>Overview</h2><p>This dataset contains an audio recording of the piece "Von edler art", a 16th century lute music piece originally notated in lute tablature, created as part of the E-LAUTE project (<a href="https://e-laute.info/">https://e-laute.info/</a>). The recording preserves and makes historical lute music from the German-speaking regions during 1450-1550 accessible.</p><p>The recording is based on the work with the title "Von edler art" and the id "New_1536-7_n49" in the e-lautedb. It is found on the page(s) or folio(s) Bbiiijr-Ccv in the source "Der ander theil des Lautenbúchs" with the source-id "New_1536-7".</p><p>The original source and multiple transcriptions of the work can be found on the E-LAUTE platform: <a href="https://edition.onb.ac.at/fedora/objects/o:lau.New_1536-7/methods/sdef:TEI/get?mode=n49" target="_blank">https://edition.onb.ac.at/fedora/objects/o:lau.New_1536-7/methods/sdef:TEI/get?mode=n49</a>.</p><p>Links to the source: <a href="https://nbn-resolving.org/urn:nbn:de:gbv:3:1-336930-p0001-4" target="_blank">https://nbn-resolving.org/urn:nbn:de:gbv:3:1-336930-p0001-4</a>, <a href="https://opac.rism.info/rism/Record/rism993104151" target="_blank">https://opac.rism.info/rism/Record/rism993104151</a>, <a href="https://gateway-bayern.de/VD16+ZV+11665" target="_blank">https://gateway-bayern.de/VD16+ZV+11665</a>, .</p><h2>Dataset Contents</h2><p>This dataset includes:</p><ul><li><strong>Audio file</strong>: An audio recording of the lute piece in .wav format</li> <li><strong>Metadata file</strong>: A metadata file with detailed information about the recording in .json format</li></ul><h2>About the E-LAUTE Project</h2><p><strong>E-LAUTE: Electronic Linked Annotated Unified Tablature Edition - The Lute in the German-Speaking Area 1450-1550</strong></p><p>The E-LAUTE project creates innovative digital editions of lute tablatures from the German-speaking area between 1450 and 1550. This interdisciplinary "open knowledge platform" combines musicology, music practice, music informatics, and literary studies to transform traditional editions into collaborative research spaces.</p><p>For more information, visit the project website: <a href="https://e-laute.info/">https://e-laute.info/</a></p>
Auff erd lebt nit ein schóners weyb (Ger_1533-1_n30) Audio recording
No full text<h1>Audio recording of a lute piece from the E-LAUTE project</h1><h2>Overview</h2><p>This dataset contains an audio recording of the piece "Auff erd lebt nit ein schóners weyb", a 16th century lute music piece originally notated in lute tablature, created as part of the E-LAUTE project (<a href="https://e-laute.info/">https://e-laute.info/</a>). The recording preserves and makes historical lute music from the German-speaking regions during 1450-1550 accessible.</p><p>The recording is based on the work with the title "Auff erd lebt nit ein schóners weyb" and the id "Ger_1533-1_n30" in the e-lautedb. It is found on the page(s) or folio(s) XXVIIIv-XXIXr in the source "Tabulatur auff die Laudten" with the source-id "Ger_1533-1".</p><p>The original source and multiple transcriptions of the work can be found on the E-LAUTE platform: <a href="https://edition.onb.ac.at/fedora/objects/o:lau.Ger_1533-1/methods/sdef:TEI/get?mode=n30" target="_blank">https://edition.onb.ac.at/fedora/objects/o:lau.Ger_1533-1/methods/sdef:TEI/get?mode=n30</a>.</p><p>Links to the source: <a href="http://resolver.staatsbibliothek-berlin.de/SBB0001F5B500000000" target="_blank">http://resolver.staatsbibliothek-berlin.de/SBB0001F5B500000000</a>, <a href="https://opac.rism.info/rism/Record/rism993104093" target="_blank">https://opac.rism.info/rism/Record/rism993104093</a>, <a href="https://gateway-bayern.de/VD16+G+1578" target="_blank">https://gateway-bayern.de/VD16+G+1578</a>, .</p><h2>Dataset Contents</h2><p>This dataset includes:</p><ul><li><strong>Audio file</strong>: An audio recording of the lute piece in .wav format</li> <li><strong>Metadata file</strong>: A metadata file with detailed information about the recording in .json format</li></ul><h2>About the E-LAUTE Project</h2><p><strong>E-LAUTE: Electronic Linked Annotated Unified Tablature Edition - The Lute in the German-Speaking Area 1450-1550</strong></p><p>The E-LAUTE project creates innovative digital editions of lute tablatures from the German-speaking area between 1450 and 1550. This interdisciplinary "open knowledge platform" combines musicology, music practice, music informatics, and literary studies to transform traditional editions into collaborative research spaces.</p><p>For more information, visit the project website: <a href="https://e-laute.info/">https://e-laute.info/</a></p>
LongEval 2025 CORE Retrieval Test Collection
No full text<div>
<p>This collection consists of queries and documents extracted from the CORE scholarly literature search engine (<a href="https://core.ac.uk/" target="_blank" rel="noreferrer noopener">https://core.ac.uk/</a>). The queries in this collection were issued by CORE users, and a specific pipeline was designed to capture user queries, their corresponding search results, and user interactions. Using this pipeline, the dataset for this task was extracted, consisting of two main components: </p>
</div>
<div>
<ol>
<li>Search Information – includes unique (anonymous) identifiers for individual user sessions, search queries, and the returned results. </li>
<li>Click Information – includes unique (anonymous) identifiers for individual user sessions, the links clicked within the results list, and the positions of the clicked links. </li>
</ol>
</div>
<div>
<p>The documents in the collection were selected based on the search results of user queries. In addition to these selected documents, the collection also contains randomly chosen documents from the CORE index. </p>
</div>
<div>
<p>The test data was collected in January 2025. In total, the collection comprises 99 queries which were held out from the training queries (November 2024) and 492 test queries fopr the test data in January 2025. The document set consists of 124,045 downloaded, cleaned, and filtered scholarly articles. </p>
</div>
<div>
<p>This collection serves as the official testing dataset for the 2025 LongEval Information Retrieval Lab (<a href="https://clef-longeval.github.io/" target="_blank" rel="noreferrer noopener">https://clef-longeval.github.io/</a>), organized at CLEF. It includes the relevance assessments for the test queries. </p>
</div>
Wo soll ich mich hin keren ich armes (Jud_1523-2_n10) Audio recording
No full text<h1>Audio recording of a lute piece from the E-LAUTE project</h1><h2>Overview</h2><p>This dataset contains an audio recording of the piece "Wo soll ich mich hin keren ich armes", a 16th century lute music piece originally notated in lute tablature, created as part of the E-LAUTE project (<a href="https://e-laute.info/">https://e-laute.info/</a>). The recording preserves and makes historical lute music from the German-speaking regions during 1450-1550 accessible.</p><p>The recording is based on the work with the title "Wo soll ich mich hin keren ich armes" and the id "Jud_1523-2_n10" in the e-lautedb. It is found on the page(s) or folio(s) 18v in the source "1.5.2.3. Ain schone kunstliche vnderweisung" with the source-id "Jud_1523-2".</p><p>The original source and multiple transcriptions of the work can be found on the E-LAUTE platform: <a href="https://edition.onb.ac.at/fedora/objects/o:lau.Jud_1523-2/methods/sdef:TEI/get?mode=n10" target="_blank">https://edition.onb.ac.at/fedora/objects/o:lau.Jud_1523-2/methods/sdef:TEI/get?mode=n10</a>.</p><p>Links to the source: <a href="http://data.onb.ac.at/rec/AC09185338" target="_blank">http://data.onb.ac.at/rec/AC09185338</a>, <a href="https://opac.rism.info/rism/Record/rism990032736" target="_blank">https://opac.rism.info/rism/Record/rism990032736</a>, <a href="https://gateway-bayern.de/VD16+J+1031" target="_blank">https://gateway-bayern.de/VD16+J+1031</a>, .</p><h2>Dataset Contents</h2><p>This dataset includes:</p><ul><li><strong>Audio file</strong>: An audio recording of the lute piece in .wav format</li> <li><strong>Metadata file</strong>: A metadata file with detailed information about the recording in .json format</li></ul><h2>About the E-LAUTE Project</h2><p><strong>E-LAUTE: Electronic Linked Annotated Unified Tablature Edition - The Lute in the German-Speaking Area 1450-1550</strong></p><p>The E-LAUTE project creates innovative digital editions of lute tablatures from the German-speaking area between 1450 and 1550. This interdisciplinary "open knowledge platform" combines musicology, music practice, music informatics, and literary studies to transform traditional editions into collaborative research spaces.</p><p>For more information, visit the project website: <a href="https://e-laute.info/">https://e-laute.info/</a></p>
Die drúncke[n] pinter (A-Wn_Mus.Hs._18688_n58) Audio recording
No full text<h1>Audio recording of a lute piece from the E-LAUTE project</h1>
<h2>Overview</h2>
<p>This dataset contains an audio recording of the piece "Die drúncke[n] pinter", a 16th century lute music piece originally notated in lute tablature, created as part of the E-LAUTE project (<a href="https://e-laute.info/">https://e-laute.info/</a>). The recording preserves and makes historical lute music from the German-speaking regions during 1450-1550 accessible.</p>
<p>The recording is based on the work with the title "Die drúncke[n] pinter" and the id "A-Wn_Mus.Hs._18688_n58" in the e-lautedb. It is found on the page(s) or folio(s) 33r in the source "[Lautentabulatur des Stephan Craus]" with the source-id "A-Wn_Mus.Hs._18688".</p>
<p>The original source and multiple transcriptions of the work can be found on the E-LAUTE platform: <a href="https://edition.onb.ac.at/fedora/objects/o:lau.A-Wn_Mus.Hs._18688/methods/sdef:TEI/get?mode=n58" target="_blank" rel="noopener">https://edition.onb.ac.at/fedora/objects/o:lau.A-Wn_Mus.Hs._18688/methods/sdef:TEI/get?mode=n58</a>.</p>
<p>Links to the source: <a href="http://data.onb.ac.at/rec/AC14316391" target="_blank" rel="noopener">http://data.onb.ac.at/rec/AC14316391</a>, <a href="https://rism.online/sources/600141880" target="_blank" rel="noopener">https://rism.online/sources/600141880</a>, .</p>
<h2>Dataset Contents</h2>
<p>This dataset includes:</p>
<ul>
<li><strong>Audio file</strong>: An audio recording of the lute piece in .wav format</li>
<li><strong>Metadata file</strong>: A metadata file with detailed information about the recording in .json format</li>
</ul>
<h2>About the E-LAUTE Project</h2>
<p><strong>E-LAUTE: Electronic Linked Annotated Unified Tablature Edition - The Lute in the German-Speaking Area 1450-1550</strong></p>
<p>The E-LAUTE project creates innovative digital editions of lute tablatures from the German-speaking area between 1450 and 1550. This interdisciplinary "open knowledge platform" combines musicology, music practice, music informatics, and literary studies to transform traditional editions into collaborative research spaces.</p>
<p>For more information, visit the project website: <a href="https://e-laute.info/">https://e-laute.info/</a></p>
Der stifl Tantz (A-Wn_Mus.Hs._18688_n55) Audio recording
No full text<h1>Audio recording of a lute piece from the E-LAUTE project</h1><h2>Overview</h2><p>This dataset contains an audio recording of the piece "Der stifl Tantz", a 16th century lute music piece originally notated in lute tablature, created as part of the E-LAUTE project (<a href="https://e-laute.info/">https://e-laute.info/</a>). The recording preserves and makes historical lute music from the German-speaking regions during 1450-1550 accessible.</p><p>The recording is based on the work with the title "Der stifl Tantz" and the id "A-Wn_Mus.Hs._18688_n55" in the e-lautedb. It is found on the page(s) or folio(s) 32v in the source "[Lautentabulatur des Stephan Craus]" with the source-id "A-Wn_Mus.Hs._18688".</p><p>The original source and multiple transcriptions of the work can be found on the E-LAUTE platform: <a href="https://edition.onb.ac.at/fedora/objects/o:lau.A-Wn_Mus.Hs._18688/methods/sdef:TEI/get?mode=n55" target="_blank">https://edition.onb.ac.at/fedora/objects/o:lau.A-Wn_Mus.Hs._18688/methods/sdef:TEI/get?mode=n55</a>.</p><p>Links to the source: <a href="http://data.onb.ac.at/rec/AC14316391" target="_blank">http://data.onb.ac.at/rec/AC14316391</a>, <a href="https://rism.online/sources/600141880" target="_blank">https://rism.online/sources/600141880</a>, .</p><h2>Dataset Contents</h2><p>This dataset includes:</p><ul><li><strong>Audio file</strong>: An audio recording of the lute piece in .wav format</li> <li><strong>Metadata file</strong>: A metadata file with detailed information about the recording in .json format</li></ul><h2>About the E-LAUTE Project</h2><p><strong>E-LAUTE: Electronic Linked Annotated Unified Tablature Edition - The Lute in the German-Speaking Area 1450-1550</strong></p><p>The E-LAUTE project creates innovative digital editions of lute tablatures from the German-speaking area between 1450 and 1550. This interdisciplinary "open knowledge platform" combines musicology, music practice, music informatics, and literary studies to transform traditional editions into collaborative research spaces.</p><p>For more information, visit the project website: <a href="https://e-laute.info/">https://e-laute.info/</a></p>
[Elselein] (A-Wn_Mus.Hs._18688_n06) Audio recording
No full text<h1>Audio recording of a lute piece from the E-LAUTE project</h1><h2>Overview</h2><p>This dataset contains an audio recording of the piece "[Elselein]", a 16th century lute music piece originally notated in lute tablature, created as part of the E-LAUTE project (<a href="https://e-laute.info/">https://e-laute.info/</a>). The recording preserves and makes historical lute music from the German-speaking regions during 1450-1550 accessible.</p><p>The recording is based on the work with the title "[Elselein]" and the id "A-Wn_Mus.Hs._18688_n06" in the e-lautedb. It is found on the page(s) or folio(s) 7v in the source "[Lautentabulatur des Stephan Craus]" with the source-id "A-Wn_Mus.Hs._18688".</p><p>The original source and multiple transcriptions of the work can be found on the E-LAUTE platform: <a href="https://edition.onb.ac.at/fedora/objects/o:lau.A-Wn_Mus.Hs._18688/methods/sdef:TEI/get?mode=n06" target="_blank">https://edition.onb.ac.at/fedora/objects/o:lau.A-Wn_Mus.Hs._18688/methods/sdef:TEI/get?mode=n06</a>.</p><p>Links to the source: <a href="http://data.onb.ac.at/rec/AC14316391" target="_blank">http://data.onb.ac.at/rec/AC14316391</a>, <a href="https://rism.online/sources/600141880" target="_blank">https://rism.online/sources/600141880</a>, .</p><h2>Dataset Contents</h2><p>This dataset includes:</p><ul><li><strong>Audio file</strong>: An audio recording of the lute piece in .wav format</li> <li><strong>Metadata file</strong>: A metadata file with detailed information about the recording in .json format</li></ul><h2>About the E-LAUTE Project</h2><p><strong>E-LAUTE: Electronic Linked Annotated Unified Tablature Edition - The Lute in the German-Speaking Area 1450-1550</strong></p><p>The E-LAUTE project creates innovative digital editions of lute tablatures from the German-speaking area between 1450 and 1550. This interdisciplinary "open knowledge platform" combines musicology, music practice, music informatics, and literary studies to transform traditional editions into collaborative research spaces.</p><p>For more information, visit the project website: <a href="https://e-laute.info/">https://e-laute.info/</a></p>
Material Parameter Estimation for Sionna RT
No full text<h2>Material Parameter Estimation for Sionna RT</h2>
<p>Sionna is an open-source library for research in communication systems developed by Nvidia. One of the main components of the library is a differentiable ray-tracer (RT). Ray-tracing is a method of modeling radio propagation. The environment in which we want to model the radio propagation is represented as a scene, that is a 3D model with assigned radio materials.</p>
<p>This repository is related to the article <em>Validation and Calibration of Ray-Tracing Systems in an Urban Railway Scenario </em>and contains the code that was used to calibrate the Sionna ray-tracer, i.e. to estimate the parameters of the materials used in the scene.</p>
<p>The estimation of the material parameters is based on the measurements of the signal strength. The simulated signal strenght is compared with the measurements and we aim to minimize the mean absolute error.</p>
<p>The .zip folder contains the code as well as an example scene and example files with RX and TX data. A sionna.yml file that can be used to create a conda environment is also enclosed.</p>
<p><strong>More details about the code and its use can be found in ReadMe.txt file.</strong></p>
Transformer network trained on simulated X-ray photoelectron spectroscopy data for organic and inorganic compounds
No full text<h2>Dataset Description</h2>
<p>This data repository provides the underlying data and neural network training scripts associated with the manuscript titled <em>"A Transformer Network for High-Throughput Materials Characterization with X-ray Photoelectron Spectroscopy" </em>by Simperl and Werner published in the Journal of Applied Physics (https://doi.org/10.1063/5.0296600) (2025)</p>
<p>All data files are released under the Creative Commons Attribution 4.0 International (CC-BY) license, while all code files are distributed under the MIT license.</p>
<p>The repository contains simulated X-ray photoelectron spectroscopy (XPS) spectra stored as hdf5 files in the zipped (h5_files.zip) folder, which was generated using the software developed by the authors. The <em>NIST Standard Reference Database 100 – Simulation of Electron Spectra for Surface Analysis (SESSA)</em> is freely available at <a href="https://www.nist.gov/srd/nist-standard-reference-database-100" target="_new" rel="noopener">https://www.nist.gov/srd/nist-standard-reference-database-100</a>. </p>
<p>The neural network architecture is implemented using the PyTorch Lightning framework and is fully available within the attached materials as <em>Transformer_SimulatedSpectra.py </em>contained in the <em>python_scripts.zip. </em></p>
<p>The trained model and the list of materials for the train, test and validation sets are contained in the <em>models.zip</em> folder.</p>
<p>The repository contains all the data necessary to replot the figures from the manuscript. These data are available in the form of .csv files or .h5 files for the spectra. In addition, the repository also contains a Python script (<em>Plot_Data_Manuscript.ipynb</em>) which is contained in the <em>python_scripts.zip</em> file.</p>
<h3>Context and methodology</h3>
<p>The dataset and accompanying Python code files included in this repository were used to train a transformer-based neural network capable of directly inferring chemical concentrations from simulated survey X-ray photoelectron spectroscopy (XPS) spectra of bulk compounds.</p>
<p>The spectral dataset provided here represents the raw output from the SESSA software (version 2.2.2), prior to the normalization procedure described in the associated manuscript. This step of normalisation is of paramount importance for the effective training of the neural network.</p>
<p>The repository contains the Python scripts utilised to execute the spectral simulations and the neural network training on the Vienna Scientific Cluster (VSC5) which is part of the Austrian Scientific Computing Infrastructure (ASC). In order to obtain guidance on the proper configuration of the Command Line Interface (CLI) tools required for SESSA, users are advised to consult the official SESSA manual, which is available at the following address: <a href="https://nvlpubs.nist.gov/nistpubs/NSRDS/NIST.NSRDS.100-2024.pdf" target="_new" rel="noopener">https://nvlpubs.nist.gov/nistpubs/NSRDS/NIST.NSRDS.100-2024.pdf</a>.</p>
<p>To run the neural network training we provided the <em>requirements_nn_training.txt </em>file that contains all the necessary python packages and version numbers. All other python scripts can be run locally with the python libraries listed in <em>requirements_data_analysis.txt.</em> </p>
<h3>Data details</h3>
<p><strong>HDF5 (in zip folder): </strong>As described in the manuscript, we simulate X-ray photoelectron spectra for each of the 7,587 inorganic [1] and organic [2] materials in our dataset. To reflect realistic experimental conditions, each simulated spectrum was augmented by systematically varying parameters such as peak width, peak shift, and peak type—all configurable within the SESSA software—as well as by applying statistical Poisson noise to simulate varying signal-to-noise ratios. These modifications account for experimentally observed and material-specific spectral broadening, peak shifts, and detector-induced noise. Each material is represented by an individual HDF5 (.h5) file, named according to its chemical formula and mass density (in g/cm³). For example, the file for SiO2 with a density of 2.196 gcm-3 is named SiO2_2.196.h5. For more complex chemical formulas, such as Co(ClO4)2 with a density of 3.33 gcm-3, the file is named Co_ClO4_2_3.33.h5. Within each HDF5 file, the metadata for each spectrum is stored alongside a fixed energy axis and the corresponding intensity values. The spectral data are organized hierarchically by augmentation parameters in the following directory structure, e.g. for Ac_10.0.h5 we have SNR_0/WIDTH_0.3/SHIFT_-3.0/PEAK_gauss/Ac_10.0/. These files can be easily inspected with H5Web in Visual Studio Code or using h5py in Python or any other h5 interpretable program.</p>
<p><strong>Session Files:</strong> The .ses files are SESSA specific input files that can be directly loaded into SESSA to specify certain input parameters for the initilization (ini), the geometry (geo) and the simulation parameters (sim_para) and are required by the python script <em>Simulation_Script_VSC_json.py </em>to run the simulation on the cluster.</p>
<p><strong>Json Files: </strong>The two json files (MaterialsListVSC_gauss.json, MaterialsListVSC_lorentz.json) are used as the input files to the Python script <em>Simulation_Script_VSC_json.py.</em> These files contain all the material specific information for the SESSA simulation.</p>
<p><strong>csv files:</strong> The csv files are used to generate the plots from the manuscript described in the section "<strong>Plotting Scripts"</strong>.</p>
<p><strong>npz files: </strong>The two .npz files (<em>element_counts.npz, single_elements.npz</em>) are python arrays that are needed by the <em>Transformer_SimulatedSpectra.py </em>script and contain the number of each single element in the dataset and an array of each single element present, respectively.</p>
<h2>SESSA Simulation Script</h2>
<p>There is one python file that sets the communication with SESSA:</p>
<ul>
<li><em>Simulation_Script_VSC_json.py:</em> This script is the heart of the simulation as it controls the communication through the CLI with SESSA using the specified input paramters in the .json and .ses files together with external functions specified in <em>VSC_function.py</em></li>
</ul>
<h3>Technical Details</h3>
<p><strong><em>Simulation_Script_VSC_json.py:</em></strong><em> </em>This script uses the functions of the <em>VSC_function.py </em>script (therefore needs to be placed in the same directory as this script) and can be called with the following command:</p>
<p> <code>python3 Simulation_Script_VSC_json.py MaterialsListVSC_gauss.json 0</code></p>
<p>It simulates the spectrum for the material at index 0 in the .json file and with the corresponding parameters specified in the .json file.</p>
<p>It is important that before running this script the following paths need to be specified:</p>
<ul>
<li>sessa_path:<em> </em>The path to their SESSA installation in <em>sessa_path </em>and the path to their session files in</li>
<li>folder_path: The path to their .ses files. In this directory an output folder will be generated where all the output files, including the simulated spectra, are written to. </li>
</ul>
<p>To run SESSA on a computing cluster it is important to have a working Xvfb (virtual frame buffer) or a similar tool available to which any graphical output from SESSA can be written to. </p>
<h2>Neural Network Training Script</h2>
<p>Before running the training script it is important to normalize the data such that the squared integral of the spectrum is 1 (as described in the manuscript) and shown in the code: <em>normalize_spectra.py</em> </p>
<p>For the neural network training we use the <em>Transformer_SimulatedSpectra.py </em>where the external functions used are specified in <em>external_functions.py. </em>This script contains the full description of the neural network architecture, the hyperparameter tuning and the Wandb logging. </p>
<p>In the <em>models.zip </em>folder the fully trained network <em>final_trained_model.ckpt</em> presented in the manuscript is available as well as the list of training, validation and testing materials <em>(test_materials_list.pt, train_materials_list.pt, val_materials_list.pt)</em> where the corresponding spectra are extracted from the hdf5 files. The file types .ckpt and .pt can be read in by using the pytorch specific load functions in Python, e.g.</p>
<pre><code>torch.load(train_materials_list)</code></pre>
<h3>Technical Details</h3>
<p><strong><em>normalize_spectra.py:</em> </strong>To run this script properly it is important to set up a python environment with the necessary libraries specified in the <em>requirements_data_analysis.txt </em>file. Then it can be called with</p>
<p><code>python3 normalize_spectra.py</code></p>
<p>where it is important to specify the path to the .h5 files containing the unnormalized spectra.</p>
<p><strong><em>Transformer_SimulatedSpectra.py:</em></strong><em> </em>To run this script properly on the cluster it is important to set up a python environment with the necessary libraries specified in the <em>requirements_nn_training.txt </em>file. This script also relies on <em>external_functions.py</em>, <em>single_elements.npz </em>and <em>element_counts.npz</em> (that should be placed in the same directory as the python script) file. This is important for creating the datasets for training, validation and testing and ensures that all the single elements appear in the testing set. You can call this script (on the cluster) within a slurm script to start the GPU training. </p>
<p><code>python3 Transformer_SimulatedSpectra.py</code></p>
<p>It is important that before running this script the following paths need to be specified:</p>
<ul>
<li>data_path: General path where all the data is stored</li>
<li>neural_network_data: The location where you keep your normalized hdf5 files</li>
<li>wandb_api_key: The api key to use wandb</li>
<li>ray_results: The location where you want to save your tuning results</li>
<li>checkpoints: The location where you want to save your ray checkpoints</li>
<li>saved_model_path: The location where you want to save your final models</li>
</ul>
<p>The following parameters can be set to true or false:</p>
<ul>
<li>ray_tune_activate: This activates the hyperparameter tuning</li>
<li>full_training_activate: This activates the full training where the hyperparameters need to be set manually </li>
<li>continue_training: If we want to train the same model for more epochs. In that case, it is important to set the data to the same train, validation and testing materials as the initially training when importing the dataloaders at the beginning. </li>
</ul>
<h2>Plotting Scripts</h2>
<p>With the script <em>Plot_Data_Manuscript.ipynb </em>we can plot all the figures in the manuscript using the .csv files and the normalized .h5 files. In order to use this code it is necessary to define the path variables at the beginning of the script accordingly to where the data (the .csv and .h5 files) are saved locally. The necessary Python libraries can be installed from <em>requirements_data_analysis.txt. </em> </p>
<h3>Figure Data</h3>
<p><strong>Figure 2: </strong>The data for figure 2 showing the chemical distribution in the dataset can directly be plotted with the file <em>elements_counts_periodic_table.csv</em></p>
<p><strong>Figure 3: </strong>The data for figure 3 showing the survey spectrum of SiO2 and the augmented data features for noise, shift and width can be extracted from SiO2_2.196.h5. </p>
<p><strong>Figure 5: </strong>The data for the validation and training loss curve can be retrieved from the <em>metrics.csv</em> file that contains the data for the training and validation loss. </p>
<p><strong>Figure 6: </strong>The polar plot indicating the distriubtion of the mean absolute error for the chemical concentration for each correctly predicted material can be generated from the <em>tp_elements_only_metrics.csv.</em></p>
<p><strong>Figure 7:</strong> The confusion matrix can be produced from the <em>confusion_matrix.csv</em> containing for each element the values for true negative (tn), false negative (fn), false positive (fp) and true positive (tp).</p>
<p><strong>Figure 8a and 8c: </strong>The two example spectra can be produced from the hdf5 files for CoCr2O4_5.14.h5 and Pb2SiO4_7.6.h5 both for snr:10, width: 0.6eV, shift: -3.0eV and peak type: Lorentz.</p>
<p><strong>Figure 8b and 8d: </strong>The predicted and true concentrations for each element can be extracted from the <em>concentration_prediction_CoCr2O4.csv </em>and <em>concentration_prediction_Pb2SiO4.csv</em>, respectively. <em> </em> </p>
<p><strong>Figure 9: </strong>The variation of the predicted concentration as a function of the background contribution in the spectrum can be extracted from the <em>background_variation_predictions.csv</em>.</p>