Common Language Resources and Technology Infrastructure - Slovenia
Not a member yet
840 research outputs found
Sort by
Serbian linguistic training corpus SETimes.SR 2.0
No full textThe SETimes.SR training corpus contains around 100,000 tokens manually annotated on the levels of tokenisation, sentence segmentation, morphosyntactic tagging, lemmatisation, syntactic dependencies, and named entities.
The annotation formalisms followed in the SETimes.SR corpus are (1) the MULTEXT-East V6 morphosyntactic specifications for the Serbo-Croatian macro-language, http://nl.ijs.si/ME/V6/msd/, (2) the UDv2 Guidelines, http://universaldependencies.org/guidelines.html, and (3) the Janes annotation guidelines for named entities, http://nl.ijs.si/janes/wp-content/uploads/2017/09/SlovenianNER-eng-v1.1.pdf.
The difference to the previous version of the corpus are (1) the extension of the corpus with 502 sentences from various news sources and (2) improvements in the annotations of the corpus.
The continuous improvement of this dataset is led by the CLASSLA knowledge centre for South Slavic languages (https://www.clarin.si/info/k-centre/) and the ReLDI Centre Belgrade
Corpus for identifying sex education concepts SemSex 1.0
No full textThe SemSex corpus is designed to facilitate the automated recognition of sexual education concepts within curriculum description documents. The corpus contains two components: PDF documents detailing Slovene school subjects and a structured JSON file named curriculums.json.
The first part of the corpus contains the PDF documents describing various school subjects. Annotations within these documents show specific phrases that pertain to one or more sexual education concepts. These annotations serve as markers, aiding in the extraction of relevant information.
The second part of the corpus, curriculums.json, is a structured file presenting the extracted texts from the PDF documents in the JSON format. This file encapsulates the textual content extracted from the PDF documents as well as annotations corresponding to the sections that describe sexual education concepts. Each annotation in curriculums.json comprises a list of concepts that the particular description is referencing. The documents were annotated manually using the concepts in the SemSex ontology. The ontology is included in the submission as an additional file in the turtle format and is described in detail at https://github.com/TimotejK/SemSex.
The JSON structure of the corpus unfolds as follows:
- At the first level, individual objects represent each PDF document.
- Under 'pages,' a list is provided for each document, encapsulating various pages.
- 'content': This section contains a string representing all the text extracted from the corresponding PDF page.
- 'page_number': Each page is tagged with the original page number from the PDF document.
- 'annotations': A list associated with each page, outlining specific annotations.
- 'start_index': Denotes the starting index of the annotated phrase within the page content.
- 'end_index': Specifies the concluding index of the annotated phrase within the page content.
- 'labels': This list encompasses the concepts to which the annotated phrase refers.
The models that have been trained on this corpus are available at http://hdl.handle.net/11356/1894
Corpus of scientific texts from the Open Science Slovenia portal OSS 1.0
No full textOSS is a large collection of scientific writing in the Slovenian language gathered from the Open Science Slovenia portal (https://openscience.si). It consists of over 150 thousand monographs, articles, diploma, master's and doctoral theses, advanced textbooks, reviews etc. mostly published between 2000 and 2022 by Slovenian universities, research institutions, etc.
Texts are accompanied by metadata, i.e. author, supervisor (for theses), year of publication, publisher (mostly faculties of the various universities), type of publication (according to SICRIS classification), keywords, and CERIF and UDC codes. The texts were obtained directly from PDFs, so it should be noted that they can contain various types of character noise.
The texts are linguistically annotated with the CLASSLA pipeline (https://github.com/clarinsi/classla) on the levels lemmatisation, MULTEXT-East Version 6 morphosyntactic descriptions, Universal Dependencies part-of-spech and morphological features, and named entities.
The corpus is distributed in CoNLL-U and vertical file formats, one file for each text. The text metadata is given as a TSV file.
Note that there exist similar, but older and smaller corpora KAS 2.0 (http://hdl.handle.net/11356/1448) and KAS 1.0 (http://hdl.handle.net/11356/1244). These contain only theses and only up to 2018, but are cleaner and with more metadata. The repository also archives a number of KAS-derived datasets; pls. search for "KAS" to find them
Slovenian Word in Context dataset SloWiC 1.0
No full textThe SloWIC dataset is a Slovenian dataset for the Word in Context task. Each example in the dataset contains a target word with multiple meanings and two sentences that both contain the target word. Each example is also annotated with a label that shows if both sentences use the same meaning of the target word. The dataset contains 1808 manually annotated sentence pairs and additional 13150 automatically annotated pairs to help with training larger models.
The dataset is stored in the JSON format following the format used in the SuperGLUE version of the Word in Context task (https://super.gluebenchmark.com/).
Each example contains the following data fields:
- word: The target word with multiple meanings
- sentence1: The first sentence containing the target word
- sentence2: The second sentence containing the target word
- idx: The index of the example in the dataset
- label: Label showing if the sentences contain the same meaning of the target word
- start1: Start of the target word in the first sentence
- start2: Start of the target word in the second sentence
- end1: End of the target word in the first sentence
- end2: End of the target word in the second sentence
- version: The version of the annotation
- manual_annotation: Boolean showing if the label was manually annotated
- group: The group of annotators that labelled the exampl
The CLASSLA-Stanza model for morphosyntactic annotation of non-standard Slovenian 2.1
No full textThis model for morphosyntactic annotation of non-standard Slovenian was built with the CLASSLA-Stanza tool (https://github.com/clarinsi/classla) by training on the SUK training corpus (http://hdl.handle.net/11356/1747) and the Janes-Tag corpus (http://hdl.handle.net/11356/1732), using the CLARIN.SI-embed.sl word embeddings (http://hdl.handle.net/11356/1204) that were expanded with the MaCoCu-sl Slovene web corpus (http://hdl.handle.net/11356/1517). These corpora were additionally augmented for handling missing diacritics by repeating parts of the corpora with diacritics removed. The model produces simultaneously UPOS, FEATS and XPOS (MULTEXT-East) labels. The estimated F1 of the XPOS annotations is ~92.17.
The difference to the previous version of the model is that the model was trained on the SUK training corpus and the 3.0 version of Janes-tag, uses new embeddings and the new version of the Slovene morphological lexicon Sloleks 3.0 (http://hdl.handle.net/11356/1745)
Multilingual comparable corpora of parliamentary debates ParlaMint 4.0
No full textParlaMint 4.0 is a set of comparable corpora containing transcriptions of parliamentary debates of 29 European countries and autonomous regions, mostly starting in 2015 and extending to mid-2022. The individual corpora comprise between 9 and 126 million words and the complete set contains over 1.1 billion words.
The transcriptions are divided by days with information on the term, session and meeting, and contain speeches marked by the speaker and their role (e.g. chair, regular speaker). The speeches also contain marked-up transcriber comments, such as gaps in the transcription, interruptions, applause, etc. The corpora have extensive metadata, most importantly on speakers (name, gender, MP and minister status, party affiliation), the political parties and parliamentary groups (name, coalition/opposition status, Wikipedia-sourced left-to-right political orientation, and CHES variables, https://www.chesdata.eu/). Note that some corpora have further metadata, e.g. the year of birth of the speakers, links to their Wikipedia articles, their membership in various committees, etc. The transcriptions are also marked with the subcorpus they belong to ("reference", until 2020-01-30, "covid", from 2020-01-31, and "war", from 2022-02-24).
The corpora are encoded according to the Parla-CLARIN TEI recommendation (https://clarin-eric.github.io/parla-clarin/), but have been encoded against the compatible, but much stricter ParlaMint encoding guidelines (https://clarin-eric.github.io/ParlaMint/) and schemas (included in the distribution).
This entry contains the ParlaMint TEI-encoded corpora and their derived plain text versions along with TSV metadata of the speeches. Also included is the 4.0 release of the sample data and scripts available at the GitHub repository of the ParlaMint project at https://github.com/clarin-eric/ParlaMint.
Note that there also exists the linguistically marked-up version of the 4.0 ParlaMint corpus, also linked with concordancers, which is available at http://hdl.handle.net/11356/1860. Another related resource is the Linguistically annotated multilingual comparable corpora of parliamentary debates in English ParlaMint-en.ana 4.0 (http://hdl.handle.net/11356/1864).
As opposed to the previous version 3.0, this version adds corpora for Spain (ES), Finland (FI) and the Basque Country (ES-PV); extends the corpora for Austria (AT), Czechia (CZ), Hungary (HU), and Ukraine (UA) with more recent data; adds metadata to political parties and parliamentary groups on left-to-right political orientation from Wikipedia as well as CHES variables; and adds the information on whether a speaker was a minister and when for the corpora that previously lacked this information. The TEI encoding of some details has also changed, and many errors found in 3.0 corpora have been corrected
Corpus of textbooks for learning Slovenian as L2 ccKUUS 2.0
No full textThe ccKUUS 2.0 corpus consists of a set of two textbooks and two workbooks for learning Slovenian as a second and foreign language, aimed at adolescents. Published by the Centre for Slovene as a Second and Foreign Language (Faculty of Arts, University of Ljubljana), these textbooks and workbooks were the most frequently used teaching materials for teaching Slovenian as a second and foreign language to adolescents in Slovenia and abroad at the time of the corpus creation. The metadata for each text includes its title, subtitle, authors, year of publication, publisher, CEFR level, target group and, for the textbooks, the number of estimated hours of the lessons.
The ccKUUS 2.0 corpus is subset of the KUUS 2.0 corpus (http://hdl.handle.net/11356/1877) which is released under the Creative Commons licence and contains approximately 10% of the complete KUUS 2.0 corpus
The CLASSLA-Stanza model for UD dependency parsing of standard Serbian 2.1
No full textThe model for UD dependency parsing of standard Serbian was built with the CLASSLA-Stanza tool (https://github.com/clarinsi/classla) by training on the SETimes.SR training corpus (http://hdl.handle.net/11356/1200) and using the CLARIN.SI-embed.sr word embeddings (http://hdl.handle.net/11356/1789). The estimated LAS of the parser is ~89.83.
The difference to the previous version of the model is that this version uses the new version of Serbian word embeddings
Ukrainian web corpus MaCoCu-uk 1.0
No full textThe Ukrainian web corpus MaCoCu-uk 1.0 was built by crawling the ".ua" and ".укр" internet top-level domains in 2022, extending the crawl dynamically to other domains as well. The crawler is available at https://github.com/macocu/MaCoCu-crawler.
Considerable effort was devoted into cleaning the extracted text to provide a high-quality web corpus. This was achieved by removing boilerplate (https://corpus.tools/wiki/Justext) and near-duplicated paragraphs (https://corpus.tools/wiki/Onion), discarding very short texts as well as texts that are not in the target language. The dataset is characterized by extensive metadata which allows filtering the dataset based on text quality and other criteria (https://github.com/bitextor/monotextor), making the corpus highly useful for corpus linguistics studies, as well as for training language models and other language technologies.
In XML format, each document is accompanied by the following metadata: title, crawl date, url, domain, file type of the original document, distribution of languages inside the document, and a fluency score based on a language model. The text of each document is divided into paragraphs that are accompanied by metadata on the information whether a paragraph is a heading or not, metadata on the paragraph quality (labels, such as “short” or “good”, assigned based on paragraph length, URL and stopword density via the jusText tool - https://corpus.tools/wiki/Justext) and fluency (score between 0 and 1, assigned with the Monocleaner tool - https://github.com/bitextor/monocleaner), the automatically identified language of the text in the paragraph, and information whether the paragraph contains sensitive information (identified via the Biroamer tool - https://github.com/bitextor/biroamer).
The corpus can be easily read with the prevert parser (https://pypi.org/project/prevert/).
Notice and take down: Should you consider that our data contains material that is owned by you and should therefore not be reproduced here, please: (1) Clearly identify yourself, with detailed contact data such as an address, telephone number or email address at which you can be contacted. (2) Clearly identify the copyrighted work claimed to be infringed. (3) Clearly identify the material that is claimed to be infringing and information reasonably sufficient in order to allow us to locate the material. (4) Please write to the contact person for this resource whose email is available in the full item record. We will comply with legitimate requests by removing the affected sources from the next release of the corpus.
This action has received funding from the European Union's Connecting Europe Facility 2014-2020 - CEF Telecom, under Grant Agreement No. INEA/CEF/ICT/A2020/2278341. This communication reflects only the author’s view. The Agency is not responsible for any use that may be made of the information it contains.
A newer version of the corpus is available as part of the MaCoCu-Genre corpora collection at http://hdl.handle.net/11356/1969. The main novelty of the MaCoCu-Genre version is that the texts have been automatically annotated with genre categories. Additionally, the corpus underwent additional post-processing and has been transformed to the JSONL format
The CLASSLA-Stanza model for lemmatisation of standard Bulgarian 2.1
No full textThe model for lemmatisation of standard Bulgarian was built with the CLASSLA-Stanza tool (https://github.com/clarinsi/classla) by training on the BulTreeBank training corpus (https://clarino.uib.no/korpuskel/corpora) and using the Bulgarian inflectional lexicon (Popov, Simov, and Vidinska 1998). The estimated F1 of the lemma annotations is ~98.93.
The difference to the previous version of the lemmatizer is that this version was trained using the new version of the Bulgarian word embeddings