Common Language Resources and Technology Infrastructure - Slovenia
Not a member yet
840 research outputs found
Sort by
Corpus of Bosnia and Herzegovina language-related news articles MetaLangNEWS-Bs
No full textA comprehensive corpus of news articles on the topic of language, published in major daily newspapers and news portals in Bosnia and Herzegovina in the five-year period of January 1, 2015 - January 1, 2020. The corpus is designed to facilitate research on metalanguage (‘language about language’), linguistic ideologies, language policy and planning, as well as the specific contemporary debates on language defining, naming, and standardisation, ongoing in post-Yugoslav societies. The corpus is available in plain text version and XML with full metadata.
MetaLangNEWS-Bs is complemented with a separate corpus of citizen metalanguage comments, i.e. online comments to the news articles, available as MetaLangNEWS-COMMENTS-Bs (http://hdl.handle.net/11356/1691). Parallel versions from Serbia (http://hdl.handle.net/11356/1371), Montenegro (http://hdl.handle.net/11356/1688), North Macedonia (http://hdl.handle.net/11356/1652), Slovenia (http://hdl.handle.net/11356/1360) and Croatia (http://hdl.handle.net/11356/1369) are also available
Slovene Text Normalizator RSDO-DS2-NORM 1.0
No full textThis Text Normalisator converts Slovene text from written-form into its spoken-form. Traditionally it is an essential preprocessing step before text-to-speech (TTS). As input it accepts text as a string, and returns a dictionary with fields "input_text", "normalised_text", "status" and "logs". Example:
normalize_text("Sodobna definicija Celzijeve temperaturne lestvice, ki velja od leta 1954, je, da je temperatura trojne točke vode enaka 0,01 °C.")
{'input_text': 'Sodobna definicija Celzijeve temperaturne lestvice, ki velja od leta 1954, je, da je temperatura trojne točke vode enaka 0,01 °C.', 'normalized_text': 'Sodobna definicija Celzijeve temperaturne lestvice, ki velja od leta tisoč devetsto štiriinpetdeset, je, da je temperatura trojne točke vode enaka nič celih nič ena stopinje Celzija.', 'status': 1, 'logs': [('1954', 'tisoč devetsto štiriinpetdeset'), ('0,01', 'nič celih nič ena'), ('°C', 'stopinje Celzija')]}
For further details see README.md
Morphological lexicon Sloleks 3.0
No full textSloleks is a reference morphological lexicon of Slovene that was developed to be used in various NLP applications and language manuals. It contains Slovene lemmas, their inflected or derivative word forms and the corresponding grammatical description. In addition to the approx. 100,000 entries already available in Sloleks 2.0 (http://hdl.handle.net/11356/1230), Sloleks 3.0 contains an additional cca. 265,000 newly generated entries from the most frequent lemmas in Gigafida 2.0 (http://hdl.handle.net/11356/1320) not yet included in previous versions of Sloleks. For verbs, adjectives, adverbs, and common nouns, the lemmas were checked manually by three annotators and included in Sloleks only if confirmed as legitimate by at least one annotator. No manual checking was performed on proper nouns.
Lemmatization rules, part-of-speech categorization and the set of feature-value pairs follow the MULTEXT-East morphosyntactic specifications for Slovenian (https://nl.ijs.si/ME/V6/msd/html/msd-sl.html). In addition to grammatical information, each word form is also given the information on its absolute corpus frequency and its compliance with the reference language standard. In addition, most entries contain information on their morphological patterns (see http://hdl.handle.net/11356/1411 for more on morphological patterns).
The lexicon also includes accentuated word forms automatically generated through neural networks (Krsnik 2017). For the 100,000 entries from Sloleks 2.0, the accentuated forms were manually corrected, whereas the accentuated forms for the other 265,000 entries are fully automatic. IPA and SAMPA phonetic transcriptions were generated automatically using an improved G2P system for Slovene developed within the RSDO project (see https://github.com/clarinsi/slovene_g2p).
Version 3.0 is encoded in XML, but unlike 2.0, which used the LMF format, the new version uses a custom XML format developed for the morphological lexicon by the Centre for Language Resources and Technologies of the University of Ljubljana (see the included .xsd files and "00README.txt" for details).
Reference:
Krsnik, Luka. Napovedovanje naglasa slovenskih besed z metodami strojnega učenja: magistrsko delo: magistrski program druge stopnje Računalništvo in informatika. Ljubljana: [L. Krsnik], 2017. http://eprints.fri.uni-lj.si/3978
Bulgarian-English parallel corpus MaCoCu-bg-en 1.0
No full textThe Bulgarian-English parallel corpus MaCoCu-bg-en 1.0 was built by crawling the ".bg" and ".бг" internet top-level domains in 2021, extending the crawl dynamically to other domains as well.
All the crawling process was carried out by the MaCoCu crawler (https://github.com/macocu/MaCoCu-crawler). Websites containing documents in both target languages were identified and processed using the tool Bitextor (https://github.com/bitextor/bitextor). Considerable efforts were devoted into cleaning the extracted text to provide a high-quality parallel corpus. This was achieved by removing boilerplate and near-duplicated paragraphs and documents that are not in one of the targeted languages. Document and segment alignment as implemented in Bitextor were carried out, and BicleanerAI (https://github.com/bitextor/bicleaner-ai) and Bifixer (https://github.com/bitextor/bifixer) were used for fixing, cleaning, and deduplicating the final version of the corpus.
While the TXT format consists solely of pairs of source and target segments (one or several sentences), each segment pair in the TMX format is accompanied by the following metadata:
- source and target document URL;
- quality score as provided by the tool BicleanerAI;
- translation direction identification: the source segment in each segment pair was identified by using a probabilistic model;
- personal information identification (“biroamer-entities”): segments containing personal information are flagged, so final users of the corpus can decide whether to use these segments;
- language variants: the language variant of English (British or American) was identified for every segment pair on document and domain level.
Notice and take down: Should you consider that our data contains material that is owned by you and should therefore not be reproduced here, please: (1) Clearly identify yourself, with detailed contact data such as an address, telephone number or email address at which you can be contacted. (2) Clearly identify the copyrighted work claimed to be infringed. (3) Clearly identify the material that is claimed to be infringing and information reasonably sufficient in order to allow us to locate the material. (4) Please write to the contact person for this resource whose email is available in the full item record. We will comply with legitimate requests by removing the affected sources from the next release of the corpus.
This action has received funding from the European Union's Connecting Europe Facility 2014-2020 - CEF Telecom, under Grant Agreement No. INEA/CEF/ICT/A2020/2278341. This communication reflects only the author’s view. The Agency is not responsible for any use that may be made of the information it contains
Icelandic-English parallel corpus MaCoCu-is-en 1.0
No full textThe Icelandic-English parallel corpus MaCoCu-is-en 1.0 was built by crawling the ".is" internet top-level domain in 2021, extending the crawl dynamically to other domains as well.
All the crawling process was carried out by the MaCoCu crawler (https://github.com/macocu/MaCoCu-crawler). Websites containing documents in both target languages were identified and processed using the tool Bitextor (https://github.com/bitextor/bitextor). Considerable efforts were devoted into cleaning the extracted text to provide a high-quality parallel corpus. This was achieved by removing boilerplate and near-duplicated paragraphs and documents that are not in one of the targeted languages. Document and segment alignment as implemented in Bitextor were carried out, and BicleanerAI (https://github.com/bitextor/bicleaner-ai) and Bifixer (https://github.com/bitextor/bifixer) were used for fixing, cleaning, and deduplicating the final version of the corpus.
While the TXT format consists solely of pairs of source and target segments (one or several sentences), each segment pair in the TMX format is accompanied by the following metadata:
- source and target document URL;
- quality score as provided by the tool BicleanerAI;
- translation direction identification: the source segment in each segment pair was identified by using a probabilistic model;
- personal information identification (“biroamer-entities”): segments containing personal information are flagged, so final users of the corpus can decide whether to use these segments;
- language variants: the language variant of English (British or American) was identified for every segment pair on document and domain level.
Notice and take down: Should you consider that our data contains material that is owned by you and should therefore not be reproduced here, please: (1) Clearly identify yourself, with detailed contact data such as an address, telephone number or email address at which you can be contacted. (2) Clearly identify the copyrighted work claimed to be infringed. (3) Clearly identify the material that is claimed to be infringing and information reasonably sufficient in order to allow us to locate the material. (4) Please write to the contact person for this resource whose email is available in the full item record. We will comply with legitimate requests by removing the affected sources from the next release of the corpus.
This action has received funding from the European Union's Connecting Europe Facility 2014-2020 - CEF Telecom, under Grant Agreement No. INEA/CEF/ICT/A2020/2278341. This communication reflects only the author’s view. The Agency is not responsible for any use that may be made of the information it contains
Monitor corpus of Slovene Trendi 2022-05
No full textThe Trendi corpus is a monitor corpus of Slovene. It contains news from 107 different media websites, published by 48 different publishers. Trendi 2022-05 covers the period from January 2019 to May 2022, complementing the Gigafida 2.0 reference corpus of written Slovene. All the contents of the Trendi corpus are at the moment obtained using the Jožef Stefan Institute Newsfeed service (http://newsfeed.ijs.si/).
The texts have been annotated using the classla-stanza pipeline (https://github.com/clarinsi/classla), including syntactic parsing according to the Universal Dependencies (https://universaldependencies.org/sl/) and Named Entities (https://nl.ijs.si/janes/wp-content/uploads/2017/09/SlovenianNER-eng-v1.1.pdf).
At the moment, the corpus is not available as a dataset due to copyright restrictions, we hope to make at least some of it available in the near future. The corpus is accessible through CLARIN.SI concordancers
Frequency list of language problems from Šolar 3.0
No full textThe dataset comprises 36570 examples of student writing from Slovenian primary and secondary schools, together with authentic (teacher-provided) corrections of language problems in these sentences.
Teacher corrections are categorised into 180 types, using a hierarchically structured system of labels described in the attached document (in Slovenian). Every entry is equipped with corresponding metadata, such as the type of the source text, the educational stage of the author, and the type and the region of the school, where the text was produced (see README for more information).
The data is exported from the Šolar 3.0 corpus (http://hdl.handle.net/11356/1589). The purpose of the dataset is to facilitate easier access for didactical purposes, statistical analyses of language problems in Slovenian primary and secondary education, and machine learning purposes
Slovene Translation of the Atomic 2020 data set SloATOMIC 2020
No full textThe SloATOMIC 2020 corpus contains the Slovene translations of the ATOMIC 2020 data set, a commonsense knowledge graph with 1.33M everyday inferential knowledge tuples about entities and events. The translations were acquired using the DeepL translation service, where a selection of about 10k examples was also manually inspected and appropriately fixed. The corpus consists of 1.331.114 examples distributed across the train, validation, and test data sets. The corpus was created as part of work package 4 of the Slovene in the Digital Environment project.
The corpus consists of the following files:
- sloatomic_train.tsv: The training set.
- sloatomic_dev.tsv: The validation set.
- sloatomic_test.tsv.automatic_all: The test set containing all of the automatically translated examples.
- sloatomic_test.tsv.automatic_10k: The selection of 10k examples from the complete test set.
- sloatomic_test.tsv.manual_10k: The manually inspected and fixed examples of the automatic 10k subset.
The data is in the tsv (tab-seperated) format. Each line contains one example. The columns are:
- head_event: The head event of the example.
- relation: The relation between the head event and the tail event. The relation can be one of the 23 different descriptors.
- tail_event: The tail event of the example
Corpus of political party programs Programi2022
No full textA corpus of political party programs for the 2022 parliamentary elections in Slovenia. Included are political programs for 19 parties with candidates at the election. The programs were extracted from party-published sources (websites, PDF files) and linguistically annotated with CLASSLA (https://github.com/clarinsi/classla).
The corpus is split by party, extracted text files are available for all 19 programs, while PDF files are available for parties that published them. It contains 330559 tokens in total, with the longest party program containing about 80 thousand and the shortest about 300 tokens
Corpus of 1968 Slovenian literature Maj68 2.0
No full textMaj68 corpus contains 1,521 texts by 198 known authors published between 1964 and 1972 in the periodicals "Tribuna", "Problemi" and "Problemi. Literatura." The texts contain complete bibliographical data, are classified according to text and language type, degree of presence of non-standard Slovenian, foreign languages, modernism, and visual elements. The data about the authors of the texts are provided with their gender and year of birth. The presence of visual elements is marked in the corpus; note that 48 texts have only visual elements, i.e. do not contain any text.
The corpus is available as facsimiles (PDFs), in the TEI encoding, as plain text files accompanied by metadata files, as the linguistically annotated TEI corpus, and the derived vertical files and registry file, for mounting on CWB-type concordancers. The TEI encoding follows the CLARIN.SI TEI customisation (https://github.com/clarinsi/TEI-schema).
The automatic linguistic annotation includes lemmas, MULTEXT-East morphosyntactic descriptions and Universal Dependencies morphological features and syntactic annotation.
As opposed to version 1 of this corpus, 647 new text from Tribuna and Problemi have been added, and some mistakes in metadata corrected