Common Language Resources and Technology Infrastructure - Slovenia
Not a member yet
840 research outputs found
Sort by
Text classification model SloBERTa-Trendi-Topics 1.0
No full textThe SloBerta-Trendi-Topics model is a text classification model for categorizing news texts with one of 13 topic labels. It was trained on a set of approx. 36,000 Slovene texts from various Slovene news sources included in the Trendi Monitor Corpus of Slovene (http://hdl.handle.net/11356/1590) such as "rtvslo.si", "sta.si", "delo.si", "dnevnik.si", "vecer.com", "24ur.com", "siol.net", "gorenjskiglas.si", etc.
The texts were semi-automatically categorized into 13 categories based on the sections under which they were published (i.e. URLs). The set of labels was developed in accordance with related categorization schemas used in other corpora and comprises the following topics: "črna kronika" (crime and accidents), "gospodarstvo, posel, finance" (economy, business, finance), "izobraževanje" (education), "okolje" (environment), "prosti čas" (free time), "šport" (sport), "umetnost, kultura" (art, culture), "vreme" (weather), "zabava" (entertainment), "zdravje" (health), "znanost in tehnologija" (science and technology), "politika" (politics), and "družba" (society). The categorization process is explained in more detail in Kosem et al. (2022): https://nl.ijs.si/jtdh22/pdf/JTDH2022_Kosem-et-al_Spremljevalni-korpus-Trendi.pdf
The model was trained on the labeled texts using the SloBERTa 2.0 contextual embeddings model (http://hdl.handle.net/11356/1397; also available at HuggingFace: https://huggingface.co/EMBEDDIA/sloberta) and validated on a development set of 1,293 texts using the simpletransformers library and the following hyperparameters:
Train batch size: 8
Learning rate: 1e-5
Max. sequence length: 512
Number of epochs: 2
The model achieves a macro-F1-score of 0.94 on a test set of 1,295 texts (best for "črna kronika", "politika", "šport", and "vreme" at 0.98, worst for "prosti čas" at 0.83).
Please note that the fastText-Trendi-Topics 1.0 text classification model is also available (http://hdl.handle.net/11356/1710) that is faster and computationally less demanding, but achieves lower classification accuracy
Frequency list of words from the Trendi corpus 2021
No full textThis frequency list of words was prepared by extracting words (i.e. lemmas with their lexical features) from the Trendi Monitor Corpus of Slovene (http://hdl.handle.net/11356/1590) covering the period between 1 January 2021 and 31 December 2021 using the LIST corpus extraction tool (http://hdl.handle.net/11356/1227). The Trendi frequency list was then compared to the frequency list of words from the Gigafida 2.0 Corpus of Slovene (http://hdl.handle.net/11356/1320), which covers the period between 1991 and 2018, and the frequency list of words from Trendi for 2019-2020. The words were compared using the simple maths formula implemented by SketchEngine (see https://www.sketchengine.eu/documentation/simple-maths/).
The final list contains lemmas, their lexical features, their absolute and relative frequencies from the first (1991–2020) and second periods (2021), and the simple maths value indicating if the word is more frequent in 2021 (simple maths > 1.00) or in 1991–2020 (simple maths < 1.00).
For frequency lists of words that are typical of previous years according to the simple maths measure (e.g. 2019 vs. 1991-2018), please refer to earlier versions of this entry
ASR training dataset for Serbian JuzneVesti-SR v1.0
No full textThe JuzneVesti-SR dataset consists of audio recordings and manual transcripts from the Južne Vesti website and its host show called '15 minuta' (https://www.juznevesti.com/Tagovi/Intervju-15-minuta.sr.html). The processing of the audio and its alignment to the manual transcripts followed the pipeline of the ParlaSpeech-HR dataset (http://hdl.handle.net/11356/1494) as closely as possible. Segments in this dataset range from 2 to 30 seconds. Train-dev-test split has been performed with 80:10:10 ratio. As with the ParlaSpeech-HR dataset, two transcriptions are provided; one with transcripts in their raw form (with punctuation, capital letters, numerals) and another normalised with the same rule-based normaliser as was used in ParlaSpeech-HR dataset creation, which is lowercased, punctuation is removed and numerals are replaced with words. The speaker_info attribute is less abundant due to the fact that compared to parliamentary corpora less data is available in this domain, so it covers only the guest name, guest description, host name, and speaker breakdown (when the host or the guest are speaking).
Original transcripts were collected with the help of the ReLDI Centre Belgrade
Database of the Western South Slavic Verb HyperVerb 1.0
No full textThe verbal Western South Slavic database (WeSoSlaV) contains 3000 most frequent Slovenian and 5300 most frequent BCS verbs which are all coded for a number of properties spanning from their phonology, morphology to their semantic and syntactic properties. The database is a table where each verb is given a row of its own. The coded properties are organized in columns
Glossary for academic integrity
No full textThe Glossary for academic integrity comprises 212 terms related to academic integrity presented in alphabetical order. Each term is accompanied by a definition, and if relevant, common synonyms.
The Glossarywas developed by an international team within an Erasmus+ project. As the main sources for the definitions, the authors utilized international and national projects, organizations with expertise in ethics, academic publishers, legislation and other relevant resources
Slovene Text Denormalizator RSDO-DS2-DENORM 1.0
No full textThis Text Denormalisator converts Slovene spoken-form text into written-form text. Typically it is used as a post-processing step in Automatic Speech Recognition, which traditionally outputs spoken-form text. As input it accepts text in either string form, list of tokens, or a list of dictionaries with a mandatory "text" field. The output is a dictionary. Example of use:
denormalize("Danes, osmega sedmega dva tisoč dvaindvajset, je lep sončen dan, saj je zunaj prijetnih petindvajset stopinj Celzija.")
{'denormalized_content': [{'text': 'Danes', 'index': [0]}, {'text': ',', 'index': [1]}, {'text': '8.', 'index': [2]}, {'text': '7.', 'index': [3]}, {'text': '2022', 'index': [4, 5, 6]}, {'text': ',', 'index': [7]}, {'text': 'je', 'index': [8]}, {'text': 'lep', 'index': [9]}, {'text': 'sončen', 'index': [10]}, {'text': 'dan', 'index': [11]}, {'text': ',', 'index': [12]}, {'text': 'saj', 'index': [13]}, {'text': 'je', 'index': [14]}, {'text': 'zunaj', 'index': [15]}, {'text': 'prijetnih', 'index': [16]}, {'text': '25', 'index': [17]}, {'text': '°C', 'index': [18, 19]}, {'text': '.', 'index': [20]}], 'denormalized_string': 'Danes, 8. 7. 2022, je lep sončen dan, saj je zunaj prijetnih 25 °C.'
Corpus of academic Slovene KAS 2.0
No full textThe KAS corpus of Slovene academic writing consists of almost 65,000 BSc/BA, 16,000 MSc/MA and 1,600 PhD theses (82 thousand texts, 5 million pages or 1,5 billion tokens) written 2000 - 2018 and gathered from the digital libraries of Slovene higher education institutions via the Slovene Open Science portal (http://openscience.si/).
The theses have associated with them significant metadata, while each thesis in the corpus contains its textual body, i.e. without their front and back matter. The body is divided into chapters, then into pages, these into paragraphs, and then into sentences. The sentence tokens are tagged with morphosyntactically descriptions (detailed part-of-speech tags) and the words lemmatised.
As opposed to the previous version 1.0, the KAS corpus of Slovene academic writing 2.0 is cleaner and contains segmentations into chapters. The metadata also contains more information about research fields of each work. Both versions consist of the same number of BSc/BA, MSc/MA, and PhD theses, however, the processing was done from scratch for 2.0, so the number of e.g. pages and tokens is different. Note also that the new version does not contain links to the PNG pictures of individual pages , nor does it contain annotated terms, both present in version 1.0. It is, unlike 1.0, also not mounted on the CLARIN.SI concordancers.
The new version is distributed in the canonical TEI encoding, JSON, and as plain text files. In the TEI format, chapter names are denoted with the tag. Each entry in JSON files have a string ID and a list containing names of chapters as its first element and texts as its second element. Chapters without text are represented as an empty string. The plain text files contain only text bodies without segmentation information.
References:
Žagar, A., Kavaš, M., & Robnik Šikonja, M. (2021). Corpus KAS 2.0: cleaner and with new datasets. In Information Society - IS 2021: Proceedings of the 24th International Multiconference. https://doi.org/10.5281/zenodo.556222
Abstracts from the KAS corpus KAS-Abs 2.0
No full textThe KAS-abs 2.0 corpus contains 125,202 automatically identified Slovenian and/or English abstracts from BSc/BA, MSc/MA, and PhD theses included in the KAS Corpus of Academic Slovene 2.0 (http://hdl.handle.net/11356/1448).
The abstracts are either in Slovenian (*-abs-sl.txt, 71,567 files) or English (*-abs-en.txt, 53,635 files); note that some theses have the abstract only in one language. The abstracts are stored in the same manner as the KAS corpus, i.e. as individual files in 1,000 directories, each with the same file identifier suffix. The file with the metadata of the theses is also included.
The corpus can be useful for research in e.g. machine translations and terminology extraction, or, using also the full texts from the KAS corpus, for studies in automatic summarisation. Note that there exists the related corpus http://hdl.handle.net/11356/1447, which contains the automatically sentence aligned bi-lingual abstract, and http://hdl.handle.net/11356/1446, which contains a pre-prepared summarization dataset.
The corpus is further described in:
Žagar, A., Kavaš, M., & Robnik Šikonja, M. (2021). Corpus KAS 2.0: cleaner and with new datasets. In Information Society - IS 2021: Proceedings of the 24th International Multiconference. https://doi.org/10.5281/zenodo.5562228
As opposed to the previous version 1.0, KAS-Abs 2.0 has been extended and most of the errors in abstract identification have been removed, e.g having the Slovene and English abstract in the same file, or the extracted abstracts also containing other parts of the thesis. The metadata file has also been improved, e.g. by giving the theses better CERIF identifiers
The news dataset for discriminating between Bosnian, Croatian and Serbian SETimes.HBS 1.0
No full textThe SETimes.HBS dataset consists of parallel documents written in Bosnian, Croatian and Serbian, harvested from the already inactive setimes.com website publishing news in the languages of South-Eastern Europe. While the writing process of the documents is not known, they are quite likely independent translations from English. The main intended usage of this dataset is closely-related language discrimination. This dataset is not a traditional parallel dataset as there are no explicit links between parallel documents. Special care was taken that the training, development and testing bins of the dataset contain the same documents in all three languages as data leakage between the three bins, given the similarity of the three languages, could be problematic for benchmarking
The CLASSLA-StanfordNLP model for morphosyntactic annotation of standard Slovenian 1.3
No full textThis model for morphosyntactic annotation of standard Slovenian was built with the CLASSLA-StanfordNLP tool (https://github.com/clarinsi/classla-stanfordnlp) by training on the ssj500k training corpus (http://hdl.handle.net/11356/1210) and using the CLARIN.SI-embed.sl word embeddings (http://hdl.handle.net/11356/1204). The model produces simultaneously UPOS, FEATS and XPOS (MULTEXT-East) labels. The estimated F1 of the XPOS annotations is ~97.06.
The difference to the previous version of the model is that the model now also includes the Sloleks inflectional lexicon