Common Language Resources and Technology Infrastructure - Slovenia
Not a member yet
840 research outputs found
Sort by
The CLASSLA-Stanza model for morphosyntactic annotation of standard Croatian 2.1
No full textThe model for morphosyntactic annotation of standard Croatian was built with the CLASSLA-Stanza tool (https://github.com/clarinsi/classla) by training on the hr500k training corpus (http://hdl.handle.net/11356/1792) and using the CLARIN.SI-embed.hr word embeddings (http://hdl.handle.net/11356/1790). The model produces simultaneously UPOS, FEATS and XPOS (MULTEXT-East) labels. The estimated F1 of the XPOS annotations is ~94.87.
The difference to the previous version of the model is that this version was trained using the new version of the hr500k corpus and the new version of the Croatian word embeddings
Slovenian Definition Extraction evaluation datasets RSDO-def 1.0
No full textThe Slovene Definition Extraction evaluation datasets RSDO-def contains sentences extracted from the Corpus of term-annotated texts RSDO5 1.1 (http://hdl.handle.net/11356/1470), which contains texts with annotated terms from four different domains: biomechanics, linguistics, chemistry, and veterinary science.
The file and sentence identifiers are the same as in the original RSDO corpus.
The labels added to the sentences included in the dataset denote:
0: Non-definition
1: Weak definition
2: Definition
The dataset consists of two parts:
1. RSDO-def-random employed a random sampling strategy, with 14 definitions, 98 weak-definitions and 849 non-definitions.
2. RSDO-def-larger added sentences to the random one by the pattern-based definition extraction as presented in Pollak et al. (2014). It contains 169 definitions, 214 weak-definitions and 872 non-definitions.
Both parts were manually annotated by five terminographers. In case of discrepancies between annotators, a consensus was reached and the final label was confirmed by all five annotators. Duplicates were removed in both parts.
The criteria for annotation are based on the standard ISO 1087-1:2000 (E/F) Terminology Work - Vocabulary, Part 1, Theory and Application, which explains a definition as follows: "Representation of a concept by a descriptive statement which serves to differentiate it from related concepts". Weak definition labels were assigned if the extracted sentences contained a term and at least one delimiting feature without a superordinate concept, or sentences consisting of superordinate concepts without delimiting features but with some typical examples. Instances were labeled as Non-definition if the sentence with the extracted concept did not contain any information about the concept or its delimiting features.
The dataset is described in more detail in Tran et al. 2023, where it was used for evaluating definition extraction approaches. If you use this resource, please cite:
Tran, T.H.H., Podpečan, V., Jemec Tomazin, M., Pollak, Senja (2023). Definition Extraction for Slovene: Patterns, Transformer Classifiers and ChatGPT. Proceedings of the ELEX 2023: Electronic lexicography in the 21st century. Invisible lexicography: everywhere lexical data is used without users realizing they make use of a “dictionary” (accepted)
Reference to the pattern-based definition extraction method used for creating RSDO-def-larger:
Pollak, S. (2014). Extracting definition candidates from specialized corpora. Slovenščina 2.0: empirical, applied and interdisciplinary research, 2(1), pp. 1–40. https://doi.org/10.4312/slo2.0.2014.1.1-40
Related resources:
- Jemec Tomazin, M. et al. (2021). Corpus of term-annotated texts RSDO5 1.1, Slovenian language resource repository CLARIN.SI, ISSN 2820-4042, http://hdl.handle.net/11356/1470.
- Podpečan et al. (2023). DF_NDF_wiki_slo: Definition extraction training sets from Wikipedia, Slovenian language resource repository CLARIN.SI, http://hdl.handle.net/11356/1840
Terminological multiword expressions lexicon
No full textThe Terminological Multiword Expressions Lexicon contains multiword terms extracted from various terminological sources. The entries were lemmatized and tagged according to the MULTEXT-East specifications for Slovenian (https://nl.ijs.si/ME/V6/msd/html/msd-sl.html), and manually checked and corrected.
The lexicon is distributed as a TSV file, one line for each multiword term. The first column contains the term in its canonical form, the second the lemmas of its words, and the third the MULTEXT-East morphosyntactic descriptions
Thesaurus of Modern Slovene 2.0
No full textThesaurus of Modern Slovene is the largest automatically generated open-access collection of Slovene synonyms. It is sourced from the data in two principal language resources: The Oxford®-DZS Comprehensive English-Slovenian Dictionary and the Gigafida 1.0 corpus of written Slovene. The links identified between synonyms were additionally confirmed using the Dictionary of Standard Slovenian Language (SSKJ). The data extraction and structure for the Thesaurus were based on the frequency and manner in which words co-occur in translation strings of the Oxford-DZS Dictionary. This information is the basis for discriminating between ‘core’ and ‘near’ synonyms, with ‘core’ synonyms exhibiting a greater connection to the keyword. In the following step, an approach combining balanced co-occurrence graphs and the Personal PageRank algorithm automatically divides the synonyms into subgroups and ranks them according to the degree of semantic relatedness to the keyword, as well as their frequency in language use. For the creation methodology, see Krek et al. (2017) in the provided references.
The database includes dictionary entries: single- and multiword headwords, their part-of-speech and other linguistic features, as well as automatically extracted synonyms, their type (core or near) and relevancy rank. In version 2.0, 4,544 manually revised antonyms were added to the database. Additionally, for a part of the database, synonyms were distributed under the corresponding word senses. Pertaining to how much lexicographic revision was involved in their preparation, database entries can have one of the following three statuses: (a) ssss-automatic (96,064 entries): no manual revision was conducted; (b) ssss-manual (3,421 entries): word senses and semantic indicators were prepared by lexicographers, and synonyms were manually distributed under each corresponding sense; (c) ssss-hybrid (1,352 entries): manually revised senses are combined with data compiled automatically. For novelties of v2.0, see Arhar Holdt et al. (2023) in the provided references
eSSKJ collocations 1.0
No full textThe database of eSSKJ Collocations 1.0 contains entries for 1797 headwords (1186 nouns, 140 verbs, 421 adjectives, and 48 adverbs) and 167 multi-word expressions with 3098 senses, and 19,181 manually curated collocates based on Gigafida 1.0 corpus and specially adapted Sketch grammar for Slovenian. The MSD tagset follows the MULTEXT-East morphosyntactic specifications for Slovenian (https://nl.ijs.si/ME/V6/msd/html/msd-sl.html). 744 collocates are linked to their corresponding dictionary senses. The selection of collocates follows the eSSKJ dictionary guidelines (https://fran.si/179/novi-slovar-slovenskega-knjiznega-jezika/datoteke/Potrjeni_koncept_NoviSSKJ.pdf). An early version of eSSKJ is available at http://hdl.handle.net/11356/1249.
Semantically close collocations are grouped into 12,619 sets belonging to 5976 syntactic structures. Where applicable, information on use of prepositions or conjunctions in collocations, and use of additional words (extended collocations) are also provided
Slovene learner corpus KOST 2.0
No full textThe corpus of Slovene as a foreign language KOST (Korpus slovenščine kot tujega jezika) contains 8,347 texts (almost 1.3 million words) written by adult speakers for whom Slovene is not their first language. This corpus offers insights into Slovene language as produced by those who are still learning it as a second or foreign language, and in particular into the most common errors that occur in this process. KOST therefore aims at all those working with Slovene as a second or foreign language.
The texts were mainly written at lectorates and Slovene as a L2/FL courses. Most of the authors of these texts speak Serbian, Bosnian and Macedonian as their first language, but texts by speakers of other languages are also included. The authors are at different proficiency levels in Slovene, from beginners to advanced.
For each contributor, information is available on gender, year of birth, country, first language and other languages they speak, employment status and education, and prior experience of learning Slovene. For each text, there is also information on the time and circumstances of creation (exam or homework), the programme in which it was produced, input type (digital or hand-written), language level and the grade.
A part of the corpus has also texts available in their corrected version. The tokens of the original and corrected texts are linked (one group of link per paragraph) and the links categorised into 23 error types.
The corpus is availabe in two formats: (1) TEI encoding of the complete corpus (texts, links), including contributor and text metadata in the TEI header, and (2) the corpus in the original and corrected variants as vertical and registry files, suitable for mounting on CQP-type concordancers. Note that the vertical format does not retain the connection between the original and corrected tokens
Dataset for evaluation of Slovene spell- and grammar-checking tools Šolar-Eval 1.0
No full textŠolar-Eval is a specialized dataset designed for the evaluation of Slovene spell- and grammar-checking tools and methodologies. It encompasses 109 essays authored by Slovene primary and secondary school students, featuring 9,808 language corrections meticulously annotated based on the identified language problems.
The essays are sourced from the Šolar 3.0 corpus, which integrates authentic corrections from language teachers (http://hdl.handle.net/11356/1589). However, inconsistencies and heterogeneity are common in teacher corrections, particularly in style improvements, making this corpus suboptimal for evaluation tasks. For Šolar-Eval, the corrections were conducted by researchers aiming to ensure consistency, homogeneity, and minimal language intervention.
The corrections are annotated according to the reference guidelines found in the attached document. The codes for language errors are structured hierarchically, facilitating robust or fine-grained evaluation.
The dataset is accessible in JSON format as generated by the CJVT Svala 1.1 annotation tool (https://orodja.cjvt.si/svala/). The source and target text is also available in the CoNLL-U format (https://universaldependencies.org/format.html). Furthermore, linguistic annotations were applied using the CLASSLA pipeline (https://github.com/clarinsi/classla/) across various levels, including tokenization, sentence segmentation, lemmatization, MULTEXT-East v6 MSD-tags, JOS-SYN dependency syntax, Universal Dependencies, and named entities (more about specific annotation layers: https://wiki.cjvt.si/shelves/linguistic-annotation-of-slovene-corpora). For better accessibility and wider usability, we provide versions with JOS-SYN as well as Universal Dependencies, and English as well as Slovene tags
Corpus of Slovenian periodicals (1771-1914) sPeriodika 1.0
No full textThe corpus of Slovenian periodicals sPeriodika contains linguistically annotated periodicals published during the 18th, 19th, and beginning of 20th century (1771-1914). The periodical issues were retreived from Slovenia's national library's digital library service (https://dlib.si) in the form of OCR-ed PDF and TXT files.
Before linguistically annotating the documents (lemmatisation, part-of-speech tagging, and named entity recognition) with CLASSLA-Stanza (https://github.com/clarinsi/classla), the OCR-ed texts were corrected with a lightweight and robust approach using cSMTiser (https://github.com/clarinsi/csmtiser), a text normalisation tool based on character-level machine translation. This OCR post-correction model was trained on a set of manually corrected samples (300 random paragraphs at least 100 characters in length) from the original texts, cf. http://hdl.handle.net/11356/1907.
The documents in the collection are enriched with the following metadata obtained from dLib:
- Document ID (URN)
- Periodical name
- Document (periodical issue) title
- Volume number (if available)
- Issue number (if available)
- Year of publication
- Date of publication (of varying granularity, based on original metadata available)
- Source (URL of the original digitised document available at dlib.si)
- Image (see below)
- Quality (see below)
The majority of documents are pagewise aligned with the scanned images of original prints. Using a concordancer the metadata allows for a single-click route to the image of the page in question for further investigation and checking of the OCR results. In other cases, only links to full document PDFs are provided. For custom use without a concordancer, the images are available at https://nl.ijs.si/inz/speriodika/ with individual files having the form -.jpg.
In addition to metadata published by dLib, the corpus contains estimates of the quality of OCR-ed text for individual pages. Pages are classified as "low" quality when mistakes are common and the text itself is mostly suitable for close reading. Otherwise, with pages appropriate for distant reading tasks, the quality metadatum is set to "good".
The corpus is available in vertical format with linguistic annotations, as well as JSON files, which contain the corpus texts in all stages of processing - see the sample JSON for README explaining the format and a sample file
Collection of Slovenian paremiological units Pregovori 1.1
No full textThis corpus collects and annotates the extensive and highly valuable diachronic collection of 37,390 Slovenian proverbs, 50 years and more in the making at the ZRC SAZU Institute of Slovenian Ethnology. Each proverb is linked to its source, and the sources comprise 2,630 bibliographical items (1578-2010): printed books, journals, calendars, collecting campaigns in different journals, folklore collecting field-works, personal notes, etc.
Each proverb is represented in two ways: in its diplomatic transcription faithful to its source (due to the technical difficulties of the transcribers and human errors in transcription, the transcription of older texts is inconsistent) and as the critical transcription which modernises the alphabet used.
The words of the critical transcriptions have also been automatically modernised to contemporary spelling using cSMTiser (https://github.com/clarinsi/csmtiser) trained on the goo300k corpus of historical Slovenian (http://hdl.handle.net/11356/1025), and these words further annotated with lemmas, MULTEXT-East morphosyntactic descriptions (https://nl.ijs.si/ME/V6/msd/html/msd-sl.html) and Universal dependencies (https://universaldependencies.org/) with the CLASSLA toolchain (https://github.com/clarinsi/classla).
The canonical encoding of the corpus is TEI, but the corpus is also distributed in two derived encodings. One is the proverbs and the bibliography as two TSV files, and the other the vertical file with the proverbs, as used by CQP-type concordancers, such as Sketch Engine.
As opposed to the previous version 1.0, this version includes 1,183 more proverbs and 115 more bibliographical items and corrects some errors
Monitor corpus of Slovene Trendi 2023-09
No full textThe Trendi corpus is a monitor corpus of Slovenian. It contains news articles from 106 media websites, published by 70 publishers. Trendi 2023-09 covers the period from January 2019 to September 2023, complementing the Gigafida 2.0 reference corpus of written Slovene (http://hdl.handle.net/11356/1320).
The contents of the Trendi corpus are obtained using the Jožef Stefan Institute Newsfeed service (http://newsfeed.ijs.si/). The texts have been annotated using the CLASSLA-Stanza pipeline (https://github.com/clarinsi/classla), including syntactic parsing according to the Universal Dependencies (https://universaldependencies.org/sl/) and Named Entities (https://nl.ijs.si/janes/wp-content/uploads/2017/09/SlovenianNER-eng-v1.1.pdf).
An important addition are topics or thematical categories, which have been automatically assigned to each text. There are 13 categories altogether: Arts and culture, Crime and accidents, Economy, Environment, Health, Leisure, Politics and Law, Science and Technology, Society, Sports, Weather, Entertainment, and Education. The text classification uses the following models: Text classification model SloBERTa-Trendi-Topics 1.0 (http://hdl.handle.net/11356/1709), Text classification model fastText-Trendi-Topics 1.0 (http://hdl.handle.net/11356/1710), and the SloBERTa model (https://huggingface.co/cjvt/sloberta-trendi-topics).
The corpus is currently not available as a downloadable dataset due to copyright restrictions but we hope to make at least some of it available in the near future. The corpus is accessible through CLARIN.SI concordancers.
As opposed to the previous version of the corpus, this version adds texts from March to September 2023, adds topic classification to files previous mistakenly without them, and corrects some other minor errors