Common Language Resources and Technology Infrastructure - Slovenia
Not a member yet
840 research outputs found
Sort by
Fine-tuned models for extractive question answering in the Slovenian language
No full text6 different fine-tuned Transformer-based models that solve the downstream task of extractive question answering in the Slovenian language. The fine-tuned models included are: bert-base-cased-squad2-SLO, bert-base-multilingual-cased-squad2-SLO, electra-base-squad2-SLO, roberta-base-squad2-SLO, sloberta-squad2-SLO and xlm-roberta-base-squad2-SLO. The models were trained and evaluated using the Slovene translation of the SQuAD2.0 dataset (https://www.clarin.si/repository/xmlui/handle/11356/1756).
The models achieve these metric values:
sloberta-squad2-SLO: EM=67.1, F1=73.56
xlm-roberta-base-squad2-SLO: EM=62.52, F1=69.51
bert-base-multilingual-cased-squad2-SLO: EM=61.37, F1=68.1
roberta-base-squad2-SLO: EM=58.23, F1=64.62
bert-base-cased-squad2-SLO: EM=55.12, F1=60.52
electra-base-squad2-SLO: EM=53.69, F1=60.8
Corpus of metaphorical expressions in spoken Slovene language G-KOMET 1.0
No full textG-KOMET (a corpus of metaphorical expressions in spoken Slovene language) is an upgrade of the hand-annotated written corpus for metaphorical expressions KOMET (http://hdl.handle.net/11356/1293) with transcriptions of speech and conversation that covers 50,000 lexical units. The corpus contains samples from the Gos corpus of spoken Slovene (http://hdl.handle.net/11356/1438) and includes a balanced set of transcriptions of informative, educational, entertaining, private, and public discourse. It contains hand-annotated metaphor-related words, i.e. linguistic expressions that have the potential for people to interpret them as metaphors, idioms, i.e. multi-word units in which at least one word has been used metaphorically, and metonymies, expressions that we use to express something else.
The annotation scheme was based on the MIPVU metaphor identification process. This protocol was modified and adapted to the specifics of the Slovene language and the specifics of the spoken language. Corpus was annotated for the following relations to metaphor: indirect metaphor, direct metaphor, borderline cases and metaphor signals. In addition, the corpus introduces a new ‘frame’ tag, which gives information about a concept to which it refers. This conceptual frame allows us to search for figurative expressions within a specific context category (e.g. time, spatial orientation, emotions etc.). Metonymies were furthermore categorized based on the specific metonymic mapping. Corpus of metaphorical expressions in spoken Slovene language G-KOMET allows an objective and systematic analysis of metaphorical expressions, metaphors and metonymies in various Slovene texts
Q-CAT Corpus Annotation Tool 1.3
No full textThe Q-CAT (Querying-Supported Corpus Annotation Tool) is a computational tool for manual annotation of language corpora, which also enables advanced queries on top of these annotations. The tool has been used in various annotation campaigns related to the ssj500k reference training corpus of Slovenian (http://hdl.handle.net/11356/1210), such as named entities, dependency syntax, semantic roles and multi-word expressions, but it can also be used for adding new annotation layers of various types to this or other language corpora. Q-CAT is a .NET application, which runs on Windows operating system.
Version 1.1 enables the automatic attribution of token IDs and personalized font adjustments.
Version 1.2 supports the CONLL-U format and working with UD POS tags.
Version 1.3 supports adding new layers of annotation on top of CONLL-U (and then saving the corpus as XML TEI)
Macedonian-English parallel corpus MaCoCu-mk-en 1.0
No full textThe Macedonian-English parallel corpus MaCoCu-mk-en 1.0 was built by crawling the ".mk" and ".мкд" internet top-level domains in 2021, extending the crawl dynamically to other domains as well.
All the crawling process was carried out by the MaCoCu crawler (https://github.com/macocu/MaCoCu-crawler). Websites containing documents in both target languages were identified and processed using the tool Bitextor (https://github.com/bitextor/bitextor). Considerable efforts were devoted into cleaning the extracted text to provide a high-quality parallel corpus. This was achieved by removing boilerplate and near-duplicated paragraphs and documents that are not in one of the targeted languages. Document and segment alignment as implemented in Bitextor were carried out, and BicleanerAI (https://github.com/bitextor/bicleaner-ai) and Bifixer (https://github.com/bitextor/bifixer) were used for fixing, cleaning, and deduplicating the final version of the corpus.
While the TXT format consists solely of pairs of source and target segments (consisting of one or several sentences), each segment pair in the TMX format is accompanied by the following metadata:
- source and target document URL;
- quality score as provided by the tool BicleanerAI;
- translation direction identification: the source segment in each segment pair was identified by using a probabilistic model;
- personal information identification (“biroamer-entities”): segments containing personal information are flagged, so final users of the corpus can decide whether to use these segments;
- language variants: the language variant of English (British or American) was identified for every segment pair on document and domain level.
Notice and take down: Should you consider that our data contains material that is owned by you and should therefore not be reproduced here, please: (1) Clearly identify yourself, with detailed contact data such as an address, telephone number or email address at which you can be contacted. (2) Clearly identify the copyrighted work claimed to be infringed. (3) Clearly identify the material that is claimed to be infringing and information reasonably sufficient in order to allow us to locate the material. (4) Please write to the contact person for this resource whose email is available in the full item record. We will comply with legitimate requests by removing the affected sources from the next release of the corpus.
This action has received funding from the European Union's Connecting Europe Facility 2014-2020 - CEF Telecom, under Grant Agreement No. INEA/CEF/ICT/A2020/2278341. This communication reflects only the author’s view. The Agency is not responsible for any use that may be made of the information it contains
Bulgarian web corpus MaCoCu-bg 1.0
No full textThe Bulgarian web corpus MaCoCu-bg 1.0 was built by crawling the ".bg" and ".бг" internet top-level domains in 2021, extending the crawl dynamically to other domains as well. The crawler is available at https://github.com/macocu/MaCoCu-crawler.
Considerable efforts were devoted into cleaning the extracted text to provide a high-quality web corpus. This was achieved by removing boilerplate (https://corpus.tools/wiki/Justext) and near-duplicated paragraphs (https://corpus.tools/wiki/Onion), discarding very short texts as well as texts that are not in the target language. The dataset is characterized by extensive metadata which allows filtering the dataset based on text quality and other criteria (https://github.com/bitextor/monotextor), making the corpus highly useful for corpus linguistics studies, as well as for training language models and other language technologies.
In the XML format, each document is accompanied by the following metadata: title, crawl date, url, domain, file type of the original document, distribution of languages inside the document, and a fluency score based on a language model. The text of each document is divided into paragraphs that are accompanied by metadata on the information whether a paragraph is a heading or not, metadata on the paragraph quality and fluency, the automatically identified language of the text in the paragraph, and information whether the paragraph contains sensitive information (identified via the Biroamer tool - https://github.com/bitextor/biroamer).
The TSV format delivers sentence-level data, and contains the following metadata: sentence URL, paragraph and sentence ID within the document, a simhash and a quality score, which allow filtering out near-duplicate sentences (all sentences with the same simhash can be deleted, except for the one with the highest quality score), the language of the sentence, information on sentence fluency, and information whether the sentence contains personal or sensitive information (identified via the Biroamer sensitive data and named entity recognizer).
Notice and take down: Should you consider that our data contains material that is owned by you and should therefore not be reproduced here, please: (1) Clearly identify yourself, with detailed contact data such as an address, telephone number or email address at which you can be contacted. (2) Clearly identify the copyrighted work claimed to be infringed. (3) Clearly identify the material that is claimed to be infringing and information reasonably sufficient in order to allow us to locate the material. (4) Please write to the contact person for this resource whose email is available in the full item record. We will comply with legitimate requests by removing the affected sources from the next release of the corpus.
This action has received funding from the European Union's Connecting Europe Facility 2014-2020 - CEF Telecom, under Grant Agreement No. INEA/CEF/ICT/A2020/2278341. This communication reflects only the author’s view. The Agency is not responsible for any use that may be made of the information it contains
Dictionary of Romanian Language (Letter N) - DLR (ELEXIS)
No full textDicționarul Limbii Române, abbreviated DLR, also called Thesaurus Dictionary of the Romanian Language, is the most important lexicographical work of the Romanian language, developed under the aegis of the Romanian Academy during more than a century. It was compiled and edited in two stages (known under the brand DA series during 1906–1944 and new series, DLR, from 1965 to 2010), in 37 volumes and contains about 175,000 words and variations, with more than 1,300,000 quotes. The development of electronic version was made in 2007–2010
Dictionary of Dubrovnik Dialect (ELEXIS)
No full textРјечнк дубровачког говора (Dictionary of the Dubrovnik Dialect) is a dialectological dictionary by Mihailo Bojanić and Rastislava Trivunac, which was first published in Srpski dijalektološki zbornik (Serbian Dialectological Journal) by the Serbian Academy of Arts and Sciences in 2002.
Note that the status of the Dubrovnik dialect is contested. While the dictionary authors take it to be Serbian, the view in Croatian linguistics is that this štokavian dialect is Croatian. For this reason it is here categorised into the Serbo-Croatian macrolanguage
Dictionary of Kosovo-Metohija Dialect (ELEXIS)
No full textРечник косовско-метохиског дијалекта (Dictionary of the Dialect of Kosovo and Metohija) by Gliša Elezović was originally published in two volumes in 1932 and 1935 in Srpski dijalektološki zbornik (Serbian Dialectological Journal) based on the sources collected between 1902 and 1928
Corpus of Slovenian texts for pedagogical purposes ccMAKS 1.0
No full textMAKS (MlAdinski KorpuS, i.e. the Youth Corpus) includes texts from literature, newspapers, and, to a lesser extent, the web. The corpus was designed for the needs of the e-learning environment "Slovenščina na dlani", where it served as a source of grammar and spelling exercises. The texts have therefore been selected to be as style-neutral as possible, proofread, and thematically interesting for the learner population. Some texts originate from the Slovenian Reference corpus Gigafida, while many texts (primarily literary) were newly gathered.
The corpus as a whole is available in the CLARIN.SI concordances, while the openly available ccMAKS dataset includes 10% of the texts, sampled in accordance with the authorship agreements. In the project "Empirical foundations for digitally-supported development of writing skills", the corpus was linguistically annotated with the CLASSLA v1.1.1 pipeline (https://github.com/clarinsi/classla/) at the levels of tokenization, sentence segmentation, lemmatization, MULTEXT-East v6 MSD-tags (https://nl.ijs.si/ME/V6/msd/html/msd-sl.html), JOS dependency syntax (https://nl.ijs.si/jos/bib/jos-skladnja-navodila.pdf), and named entities (https://nl.ijs.si/janes/wp-content/uploads/2017/09/SlovenianNER-eng-v1.1.pdf). The idea is to provide comparably annotated pedagogically-relevant corpora that can be used for different tasks in language didactics and NLP.
The corpus is available in CoNLL-U and vertical formats. The CoNLL-U format contains one document per file (and separately text metadata as a TSV file) and the vertical format contains concatenated documents in one large file. The registry file ccmaks.regi for the vertical format is compatible with the LIST 1.2 corpus extraction tool (http://hdl.handle.net/11356/1276) and the ccmaks.noske.regi file is needed for SketchEngine-type concordancers
ASR database ARTUR 0.1 (transcriptions)
No full textARTUR is a speech database designed for the needs of automatic speech recognition for the Slovenian language. The database includes 1,035 hours of speech, although only 840 hours are transcribed, while the remaining 195 hours are without transcription.
The data is divided into 4 parts:
(1) approx. 520 hours of read speech, which includes the reading of pre-defined sentences, selected from the corpus Gigafida; each sentence is contained in one file; speakers are demographically balanced; spelling is included in special files; all with manual transcriptions;
(2) approx. 204 hours of public speech, which includes media recordings, online recordings of conferences, workshops, education videos, etc.; 56 hours are manually transcribed;
(3) approx. 110 hours of private speech, which includes monologues and dialogues between two persons, recorded for the purposes of the speech database; the speakers are demographically balanced; two subsets for domain-specific ASR (i.e., smart-home and face-description) are included; 63 hours are manually transcribed;
(4) approx. 201 hours of parliamentary speech, which includes recordings from the Slovene National Assembly, all with manual transcriptions.
This repository entry includes transcriptions in Transcriber 1.5.1 TRS format only; audio recordings are available at http://hdl.handle.net/11356/1717