Common Language Resources and Technology Infrastructure - Slovenia
Not a member yet
840 research outputs found
Sort by
Slovene instruction-following dataset for large language models GaMS-Instruct-GEN 1.0
No full textGaMS-Instruct-GEN is an instruction-following dataset designed to fine-tune Slovene large language models to follow instructions. It consists of pairs of prompts and responses, some of which contain an additional input field.
The dataset was generated automatically using GPT-4 by using 225 manually compiled seed prompts from SelfInstruct (Wang et al. 2022), an instruction-following dataset for English (https://huggingface.co/datasets/yizhongw/self_instruct). The seed prompts were manually translated into Slovene (see "seed_tasks_sl.jsonl") and used as part of a prompt to generate additional similar examples (see 00README.txt for more details).
The automatically generated examples were manually validated by 9 annotators (linguists). Version 1.0 contains only prompt-response pairs that are adequately formatted and free of LLM-hallucinations. Most of the prompt-response pairs deal with general topics (e.g. essay writing, event organization, text corrections, creative tasks), while some deal with Slovene-specific topics (e.g. planning trips around Slovenia, prompts referring to Slovene literature or culture)
Corpus of 1968 Slovenian literature Maj68 3.0
No full textMaj68 corpus contains 1,521 texts (about a million words) by 198 known authors published between 1964 and 1972 in the periodicals "Tribuna", "Problemi" and "Problemi. Literatura." The texts contain complete bibliographical data, are classified according to text and language type, degree of presence of non-standard Slovenian, foreign languages, modernism, and visual elements. The data about the authors of the texts are provided with their gender and year of birth. The presence of visual elements is marked in the corpus; note that 48 texts have only visual elements, i.e. do not contain any text.
The corpus is available as facsimiles (PDFs), in TEI encoding, as plain text files accompanied by metadata files, as a linguistically annotated TEI corpus, and the derived vertical files and registry file, for mounting on CWB-type concordancers. The TEI encoding follows the CLARIN.SI TEI customisation (https://github.com/clarinsi/TEI-schema).
The automatic linguistic annotation includes lemmas, MULTEXT-East morphosyntactic descriptions and Universal Dependencies morphological features and syntactic annotation. and was performed by the CLASSLA-Stanza pipeline (https://github.com/clarinsi/classla), using the models for standard Slovenian.
As opposed to to the previous version, this corpus also includes manually assigned linguistic categories, in particular for names and for spans of text not written in standard Slovenian. Both of these categories are further subdivided, and the typology is given in the accompanying PDF file of this entry
CorefUD conversion of Slovene coreference resolution corpus coref149
No full textThis corpus is the CorefUD conversion of the coref149 corpus for coreference resolution in Slovene (http://hdl.handle.net/11356/1182). It contains 149 documents annotated with coreference information. Coreference in Universal Dependencies (CorefUD) is an initiative to collect coreference corpora in various languages and harmonize them to the same scheme and data format (CoNLL-U). The coreference information is stored in the MISC column. More concretely, the start and end of each coreference mention is marked with the "Entity=" attribute. For example, "Entity=(e0" marks the start of the entity e0 at the current token while "Entity=e0) marks the end of the entity e0 at the current token. For full details on the format, please see http://hdl.handle.net/11234/1-5478. To ensure compliance with the CoNLL-U format, the corpus was automatically annotated with trankit v1.1.2 to obtain lemmas, part of speech tags (UPOS, XPOS - MULTEXT-East V6), features, and dependencies (head, dependency relation). To enable implementation into the SloBENCH evaluation framework (https://slobench.cjvt.si/), we release the labeled training set (containing 100 documents) and the unlabeled test set (containing 49 documents) in the CorefUD format. Please note that the labels are available in the original coref149 corpus but omitted here to deter misuse of the test set labels. In comparison to the original coref149 corpus, this contains the same texts and coreference information in a different (more universal) format
Monitor corpus of Slovene Trendi 2023-11
No full textThe Trendi corpus is a monitor corpus of Slovenian. It contains news articles from 106 media websites, published by 70 publishers. Trendi 2023-11 covers the period from January 2019 to November 2023, complementing the Gigafida 2.0 reference corpus of written Slovene (http://hdl.handle.net/11356/1320).
The contents of the Trendi corpus are obtained using the Jožef Stefan Institute Newsfeed service (http://newsfeed.ijs.si/). The texts have been annotated using the CLASSLA-Stanza pipeline (https://github.com/clarinsi/classla), including syntactic parsing according to the Universal Dependencies (https://universaldependencies.org/sl/) and Named Entities (https://nl.ijs.si/janes/wp-content/uploads/2017/09/SlovenianNER-eng-v1.1.pdf).
An important addition are topics or thematical categories, which have been automatically assigned to each text. There are 13 categories altogether: Arts and culture, Crime and accidents, Economy, Environment, Health, Leisure, Politics and Law, Science and Technology, Society, Sports, Weather, Entertainment, and Education. The text classification uses the following models: Text classification model SloBERTa-Trendi-Topics 1.0 (http://hdl.handle.net/11356/1709), Text classification model fastText-Trendi-Topics 1.0 (http://hdl.handle.net/11356/1710), and the SloBERTa model (https://huggingface.co/cjvt/sloberta-trendi-topics).
The corpus is currently not available as a downloadable dataset due to copyright restrictions but we hope to make at least some of it available in the near future. The corpus is accessible through CLARIN.SI concordancers.
This version adds texts from October to November 2023
Collocations Dictionary of Modern Slovene KSSS 2.0
No full textThe database of the Collocations Dictionary of Modern Slovene 2.0 contains 4,491,958 collocations in 81,443 entries. Collocations occur in 81 different syntactic relations. Collocations are labelled according to their status as "automatic" (automatically extracted, not yet manually validated) and "manual" (manually validated). In total, there are 2,090 completed entries (all collocations manually validated) and 11,227 entries with sense division and a combination of manual and automatic collocations. The IDs, provided for headwords, senses and collocations, come from the Digital Dictionary Database for Slovene.
Collocations were obtained from the Gigafida 2.0 corpus, using a method for extracting collocation data from text corpora based on a formal definition of syntactic structures, which takes into account not only the POS-tagging level of annotation but also syntactic parsing (syntactic treebank model) and introduces the possibility of controlling the canonical form of extracted collocations based on statistical data on forms with different properties in the corpus. The link to the paper describing the procedure (Krek et al. 2022) is listed as a reference in this entry.
The dictionary is split into 41 files of 2000 entries to keep the file size manageable
The CLASSLA-Stanza model for UD dependency parsing of standard Slovenian 2.0
No full textThis model for UD dependency parsing of standard Slovenian was built with the CLASSLA-Stanza tool (https://github.com/clarinsi/classla) by training on the SUK training corpus (http://hdl.handle.net/11356/1747) and using the CLARIN.SI-embed.sl word embeddings (http://hdl.handle.net/11356/1204) expanded with the MaCoCu-sl Slovene web corpus (http://hdl.handle.net/11356/1517). The estimated LAS of the parser is ~91.11.
The difference to the previous version of the model is that the model was trained using the SUK training corpus and uses the updated embeddings
Croatian-English parallel corpus MaCoCu-hr-en 2.0
No full textThe Croatian-English parallel corpus MaCoCu-hr-en 2.0 was built by crawling the “.hr” internet top-level domain in 2021 and 2022, extending the crawl dynamically to other domains as well.
All the crawling process was carried out by the MaCoCu crawler (https://github.com/macocu/MaCoCu-crawler). Websites containing documents in both target languages were identified and processed using the tool Bitextor (https://github.com/bitextor/bitextor). Considerable effort was devoted into cleaning the extracted text to provide a high-quality parallel corpus. This was achieved by removing boilerplate and near-duplicated paragraphs and documents that are not in one of the targeted languages. Document and segment alignment as implemented in Bitextor were carried out, and Bifixer (https://github.com/bitextor/bifixer) and BicleanerAI (https://github.com/bitextor/bicleaner-ai) were used for fixing, cleaning, and deduplicating the final version of the corpus.
The corpus is available in three formats: two sentence-level formats, TXT and TMX, and a document-level TXT format. TMX is an XML-based format and TXT is a tab-separated format. They both consist of pairs of source and target segments (one or several sentences) and additional metadata. The following metadata is included in both sentence-level formats:
- source and target document URL;
- paragraph ID which includes information on the position of the sentence in the paragraph and in the document (e.g., “p35:77s1/3” which means “paragraph 35 out of 77, sentence 1 out of 3”);
- quality score as provided by the tool Bicleaner AI (a likelihood of a pair of sentences being mutual translations, provided with a score between 0 and 1);
- similarity score as provided by the sentence alignment tool Bleualign (value between 0 and 1);
- personal information identification (“biroamer-entities-detected”): segments containing personal information are flagged, so final users of the corpus can decide whether to use these segments;
- translation direction and machine translation identification (“translation-direction”): the source segment in each segment pair was identified by using a probabilistic model (https://github.com/RikVN/TranslationDirection), which also determines if the translation has been produced by a machine-translation system;
- a DSI class (“dsi”): information whether the segment is connected to any of Digital Service Infrastructure (DSI) classes (e.g., cybersecurity, e-health, e-justice, open-data-portal), defined by the Connecting Europe Facility (https://github.com/RikVN/DSI);
- English language variant: the language variant of English (British or American, using a lexicon-based English variety classifier - https://pypi.org/project/abclf/) was identified on document and domain level.
Furthermore, the sentence-level TXT format provides additional metadata:
- web domain of the text;
- source and target document title;
- the date when the original file was retrieved;
- the original type of the file (e.g., “html”), from which the sentence was extracted;
- paragraph quality (labels, such as “short” or “good”, assigned based on paragraph length, URL and stopword density via the jusText tool - https://corpus.tools/wiki/Justext);
- information whether the sentence is a heading or not in the original document.
The document-level TXT format provides pairs of documents identified to contain parallel data. In addition to the parallel documents (in base64 format), the corpus includes the following metadata: source and target document URL, a DSI category and the English language variant (British or American).
As opposed to the previous version, this version has more accurate metadata on languages of the texts, which was achieved by using Google's Compact Language Detector 2 (CLD2) (https://github.com/CLD2Owners/cld2), a high-performance language detector supporting many languages. Other tools, used for web corpora creation and curation, have been updated as well, resulting in an even cleaner corpus. The new version also provides additional metadata, such as the position of the sentence in the paragraph and document, and information whether the sentence is related to a DSI. Moreover, the corpus is now also provided in a document-level format.
Notice and take down: Should you consider that our data contains material that is owned by you and should therefore not be reproduced here, please: (1) Clearly identify yourself, with detailed contact data such as an address, telephone number or email address at which you can be contacted. (2) Clearly identify the copyrighted work claimed to be infringed. (3) Clearly identify the material that is claimed to be infringing and information reasonably sufficient in order to allow us to locate the material. (4) Please write to the contact person for this resource whose email is available in the full item record. We will comply with legitimate requests by removing the affected sources from the next release of the corpus.
This action has received funding from the European Union's Connecting Europe Facility 2014-2020 - CEF Telecom, under Grant Agreement No. INEA/CEF/ICT/A2020/2278341. This communication reflects only the author’s view. The Agency is not responsible for any use that may be made of the information it contains
Catalan-English parallel corpus MaCoCu-ca-en 1.0
No full textThe Catalan-English parallel corpus MaCoCu-ca-en 1.0 was built by crawling the ".cat", ".es", ".ad", ".fr", ".it" and ".eu” internet top-level domain in 2022, extending the crawl dynamically to other domains as well.
All the crawling process was carried out by the MaCoCu crawler (https://github.com/macocu/MaCoCu-crawler). Websites containing documents in both target languages were identified and processed using the tool Bitextor (https://github.com/bitextor/bitextor). Considerable effort was devoted into cleaning the extracted text to provide a high-quality parallel corpus. This was achieved by removing boilerplate and near-duplicated paragraphs and documents that are not in one of the targeted languages. Document and segment alignment as implemented in Bitextor were carried out, and Bifixer (https://github.com/bitextor/bifixer) and BicleanerAI (https://github.com/bitextor/bicleaner-ai) were used for fixing, cleaning, and deduplicating the final version of the corpus.
The corpus is available in three formats: two sentence-level formats, TXT and TMX, and a document-level TXT format. TMX is an XML-based format and TXT is a tab-separated format. They both consist of pairs of source and target segments (one or several sentences) and additional metadata. The following metadata is included in both sentence-level formats:
- source and target document URL;
- paragraph ID which includes information on the position of the sentence in the paragraph and in the document (e.g., “p35:77s1/3” which means “paragraph 35 out of 77, sentence 1 out of 3”);
- quality score as provided by the tool Bicleaner AI (a likelihood of a pair of sentences being mutual translations, provided with a score between 0 and 1);
- similarity score as provided by the sentence alignment tool Bleualign (value between 0 and 1);
- personal information identification (“biroamer-entities-detected”): segments containing personal information are flagged, so final users of the corpus can decide whether to use these segments;
- translation direction and machine translation identification (“translation-direction”): the source segment in each segment pair was identified by using a probabilistic model (https://github.com/RikVN/TranslationDirection), which also determines if the translation has been produced by a machine-translation system;
- a DSI class (“dsi”): information whether the segment is connected to any of Digital Service Infrastructure (DSI) classes (e.g., cybersecurity, e-health, e-justice, open-data-portal), defined by the Connecting Europe Facility (https://github.com/RikVN/DSI);
- English language variant: the language variant of English (British or American, using a lexicon-based English variety classifier - https://pypi.org/project/abclf/) was identified on document and domain level.
Furthermore, the sentence-level TXT format provides additional metadata:
- web domain of the text;
- source and target document title;
- the date when the original file was retrieved;
- the original type of the file (e.g., “html”), from which the sentence was extracted;
- paragraph quality (labels, such as “short” or “good”, assigned based on paragraph length, URL and stopword density via the jusText tool - https://corpus.tools/wiki/Justext);
- information whether the sentence is a heading or not in the original document.
The document-level TXT format provides pairs of documents identified to contain parallel data. In addition to the parallel documents (in base64 format), the corpus includes the following metadata: source and target document URL, a DSI category and the English language variant (British or American).
Notice and take down: Should you consider that our data contains material that is owned by you and should therefore not be reproduced here, please: (1) Clearly identify yourself, with detailed contact data such as an address, telephone number or email address at which you can be contacted. (2) Clearly identify the copyrighted work claimed to be infringed. (3) Clearly identify the material that is claimed to be infringing and information reasonably sufficient in order to allow us to locate the material. (4) Please write to the contact person for this resource whose email is available in the full item record. We will comply with legitimate requests by removing the affected sources from the next release of the corpus.
This action has received funding from the European Union's Connecting Europe Facility 2014-2020 - CEF Telecom, under Grant Agreement No. INEA/CEF/ICT/A2020/2278341. This communication reflects only the author’s view. The Agency is not responsible for any use that may be made of the information it contains
Corpus of combined Slovenian corpora metaFida 1.0
No full textSlovenia has a large number of diverse corpora available for online analysis via the CLARIN.SI concordancers. However, if users are interested in the same queries across different corpora they have to search for relevant information in each corpus separately, and then combine this information manually, which is time-consuming and also prone to analysis errors. An additional problem is that corpora typically have different metadata and may also be labeled at different linguistic levels, which further complicates identical searches across different corpora.
For these reasons we combined a number of existing Slovenian corpora available through the CLARIN.SI concordances into the metaFida corpus. Here it was first necessary to unify the metadata and harmonize the linguistic and structural annotations between the corpora, and to create conversions of individual corpora from their vertical formats, which are used as input by the CLARIN.SI concordances, into the metaFida vertical format. As the source corpora are not completely distinct, metaFida is also deduplicated on the paragraph level.
In the metaFida corpus we keep only information that is common to most of the selected corpora. The structure is nested very shallowly (text and paragraph), as it is then easier to create subcorpora or limit the search to individual text types. All metaFida positional attributes (word, normalised form, lemma, MULTEXT-East MSD in Slovenian and English) are considered to have multiple values, separated by a space. Multiple values are needed because some corpora have normalized words (older Slovenian, user-generated content), where one original word can be mapped to several normalized ones or vice versa.
metaFida contains over 4,7 billion words or 6 billion tokens from 15 million text published 1584 - 2022 from the following 34 corpora, of which many, but not all, are also availiable for download, as indicated by their handle:
* eltec_slv: ELTeC-slv (100 romanov), https://doi.org/10.5281/zenodo.4662600; 5,596,656 words
* prilit: PriLit (starejša pripovedna proza), http://hdl.handle.net/11356/1319; 1,060,538 words
* imp: IMP (starejša besedila), http://hdl.handle.net/11356/1031; 14,348,452 words
* maj68: Maj68 (Maj 1968 v literaturi), http://hdl.handle.net/11356/1491; 1,033,971 words
* vayna: VAYNA (napadi na JNA), http://hdl.handle.net/11356/1237; 256,429 words
* gos20: Gos 2.0 (referenčni, govorni), http://hdl.handle.net/11356/1771; 2,436,386 words
* janes_norm30: Janes Norm 3.0 (ročno normaliziran), http://hdl.handle.net/11356/1733; 249,576 words
* janes_tweet: Janes Tweet (tviti 2013-2017), http://hdl.handle.net/11356/1142; 108,769,902 words
* janes_wiki: Janes Wiki (Wikipedija komentarji), http://hdl.handle.net/11356/1137; 3,917,428 words
* janes_blog: Janes Blog (blogi s komentarji), http://hdl.handle.net/11356/1138; 27,596,463 words
* janes_forum: Janes Forum (spletni forumi), http://hdl.handle.net/11356/1139; 37,654,809 words
* janes_news: Janes News (komentarji na novice), http://hdl.handle.net/11356/1140; 11,908,481 words
* lemonde_sl: LeMonde: slovensko; 506,358 words
* konji: Konji (konjeništvo); 395,718 words
* filmi: FILMI (filmske kritike); 764,764 words
* maks: MAKS (mladinska književnost); 9,881,294 words
* ispac_sl: ISPAC: slovensko; 1,169,486 words
* jaslo_sl: jaSlo: slovensko; 425,434 words
* siparl30: siParl 3.0 (parlament 1990-2022), http://hdl.handle.net/11356/1748; 205,441,411 words
* kost10_orig: KOST: izvorni (L2), https://hdl.handle.net/11356/1753; 1,020,509 words
* jezkor: JezKor (jezikoslovje), http://hdl.handle.net/11356/1755; 6,243,898 words
* solar30_orig: Šolar: učenci (razvojni), https://hdl.handle.net/11356/1589; 1,621,527 words
* sbsj: SBSJ (šolska besedila), http://hdl.handle.net/11356/1413; 1,424,887 words
* rsdo5: RSDO5 (s termini označena besedila), http://hdl.handle.net/11356/1470; 241,797 words
* dsi: DSI (informatika), http://hdl.handle.net/11356/1239; 4,254,177 words
* korp: KoRP (odnosi z javnostmi); 1,756,731 words
* suss: ŠUSS (jezikovna vprašanja), http://hdl.handle.net/11356/1242; 272,541 words
* trans5_sl: TRANS5: slovensko; 1,297,269 words
* dgt15_sl: EU DGT 2015: Slovene; 48,454,851 words
* gfida20_dedup: Gigafida v2.0 (referenčni, dedupliciran), http://hdl.handle.net/11356/1320; 1,105,200,611 words
* oss10: OSS (znanstvena dela), http://hdl.handle.net/11356/1774; 2,342,855,598 words
* classlawiki_sl: CLASSLAWiki-sl (Slovenian Wikipedia), http://hdl.handle.net/11356/1427; 41,543,793 words
* slwac: slWaC (Slovene Web); 749,372,269 words
* tweet_sl: Tweet-sl (stari tviti); 4,854,229 words
Σ 34 corpora, 4,743,828,243 words before deduplication, which removes about 0.3% of words, 1.3% tokens, 7% texts and 11% paragraphs
ASR database ARTUR 1.0 (audio)
No full textArtur 1.0 is a speech database designed for the needs of automatic speech recognition for the Slovenian language. The database includes 1,067 hours of speech. 884 hours are transcribed, while the remaining 183 hours are recordings only. This repository entry includes audio files only, the transcriptions are available on http://hdl.handle.net/11356/1772.
The data are structured as follows:
(1) Artur-B, read speech, 573 hours in total. It includes:
(1a) Artur-B-Brani, 485 hours: Readings of sentences which were pre-selected from a 10% increment in the Gigafida 2.0 corpus. The sentences were chosen in such a way that they reflect the natural or the actual distribution of triphones in the words. They were distributed between 1,000 speakers, so that we recorded approx. 30 min in read form from each speaker. The speakers were balanced according to gender, age, region, and a small proportion of speakers were non-native speakers of Slovene. Each sentence is its own audio file and has a corresponding transcription file.
(1b) Artur-B-Crkovani, 10 hours: Spellings. Speakers were asked to spell abbreviations and personal names and surnames, all chosen so that all Slovene letters were covered, plus the most common foreign letters.
(1c) Artur-B-Studio, 51 hours: Designed for the development of speech synthesis. The sentences were read in a studio by a single speaker. Each sentence is its own audio file and has a corresponding transcription file.
(1d) Artur-B-Izloceno, 27 hours: The recordings include different types of errors, typically, incorrect reading of sentences or a noisy environment.
(2) Artur-J, public speech, 62 hours in total. It includes:
(2a) Artur-J-Splosni, 62 hours: media recordings, online recordings of conferences, workshops, education videos, etc.
(3) Artur-N, private speech, 74 hours in total. It includes:
(3a) Artur-N-Obrazi, 6 hours: Speakers were asked to describe faces on pictures. Designed for a face-description domain-specific speech recognition.
(3b) Artur-N-PDom, 7 hours: Speakers were asked to read pre-written sentences, as well as to express instructions for a potential smart-home system freely. Designed for a smart-home domain-specific speech recognition.
(3c) Artur-N-Prosti, 61 hours: Monologues and dialogues between two persons, recorded for the purposes of the Artur database creation. Speakers were asked to conversate or explain freely on casual topics.
(4) Artur-P, parliamentary speech, 201 hours in total. It includes:
(4a) Artur-P-SejeDZ, 201 hours: Speech from the Slovene National Assembly.
Further information on the database are available in the Artur-DOC file, which is part of this repository entry