Common Language Resources and Technology Infrastructure - Slovenia
Not a member yet
840 research outputs found
Sort by
A1 core vocabulary with lexical information for Slovenian 1.0
No full textThe A1 Core Vocabulary with Lexical Information for Slovenian 1.0 includes vocabulary that users of Slovenian as L2 are expected to acquire at level A1 according to the CEFR. The vocabulary items are supplemented with user-tailored grammatical, semantic and multimedia information, including the translation of the entry into three foreign languages – English, Hungarian (a language unrelated to Slovenian and spoken in the Slovenian cross-border area) and Albanian (a language distant from Slovenian and often spoken by immigrant pupils in Slovenia). Some translations for entries or meanings are still missing and will be added during the upgrade process. This growing resource, the first of its kind, is designed to cater to the specific needs of users learning Slovenian as L2. In the current version, it contains a limited number of entries (991). It has been developed with the idea that in the future it can be systematically supplemented and upgraded with more complex semantic information relevant for users of Slovenian as L2 at higher levels of language proficiency (A2-C1). A description of this resource can be found in:
KLEMEN, Matej, ARHAR HOLDT, Špela, POLLAK, Senja, KOSEM, Iztok, PORI, Eva, GANTAR, Polona, KNEZ, Mihaela. Building a CEFR-Labeled Core Vocabulary and Developing a Lexical Resource for Slovenian as a Second and Foreign Language. In Proceedings of the eLex 2023 conference: Electronic lexicography in the 21st century: Invisible Lexicography. Brno: Lexical Computing CZ s.r.o., 2023, pp. 664-678, https://elex.link/elex2023/proceedings-download/
Font ZRCalo 1.0
No full textZRCalo is an open font meant to gradually phase out the ZRCola font as one of the components of the ZRCola 2 input system (http://hdl.handle.net/11356/1090). The current version is a baseline variant covering the basic Latin Unicode blocks. Future versions will aim to build on Unicode's combining characters mechanic to replace ZRCola's extensive use of the Private Use Area
Pretrained models for recognising sex education concepts SemSEX 1.0
No full textPretrained language models for detecting and classifying the presence of sex education concepts in Slovene curriculum documents. The models are PyTorch neural network models, intended for usage with the HuggingFace transformers library (https://github.com/huggingface/transformers).
The models are based on the Slovenian RoBERTa contextual embeddings model SloBERTa 2.0 (http://hdl.handle.net/11356/1397) and on the CroSloEngual BERT model (http://hdl.handle.net/11356/1330). The source code of the model and example usage is available in GitHub repository https://github.com/TimotejK/SemSex.
The models and tokenizers can be loaded using the AutoModelForSequenceClassification.from_pretrained() and the AutoTokenizer.from_pretrained() functions from the transformers library. An example of such usage is available at https://github.com/TimotejK/SemSex/blob/main/Concept%20detection/Classifiers/full_pipeline.py.
The corpus on which these models have been trained is available at http://hdl.handle.net/11356/1895
Word embeddings CLARIN.SI-embed.sr 2.0
No full textCLARIN.SI-embed.sr contains word embeddings induced from the srWaC and MaCoCu-sr web corpora. The embeddings are based on the skip-gram model of fastText trained on 3,434,602,575 tokens of running text for 2,676,036 lowercased surface forms.
The difference to the previous version of the embeddings is that this version was trained on the original dataset expanded with the MaCoCu-sr web crawl corpus
Word embeddings CLARIN.SI-embed.bg 1.0
No full textCLARIN.SI-embed.bg contains word embeddings for Bulgarian induced from the MaCoCu-bg web crawl corpus (http://hdl.handle.net/11356/1515). The embeddings are based on the skip-gram model of fastText trained on 4,120,343,820 tokens of running text for 2,746,640 lowercased surface forms
Turkish-English parallel corpus MaCoCu-tr-en 2.0
No full textThe Turkish-English parallel corpus MaCoCu-tr-en 2.0 was built by crawling the “.tr” and “.cy” internet top-level domains in 2021, extending the crawl dynamically to other domains as well.
All the crawling process was carried out by the MaCoCu crawler (https://github.com/macocu/MaCoCu-crawler). Websites containing documents in both target languages were identified and processed using the tool Bitextor (https://github.com/bitextor/bitextor). Considerable effort was devoted into cleaning the extracted text to provide a high-quality parallel corpus. This was achieved by removing boilerplate and near-duplicated paragraphs and documents that are not in one of the targeted languages. Document and segment alignment as implemented in Bitextor were carried out, and Bifixer (https://github.com/bitextor/bifixer) and BicleanerAI (https://github.com/bitextor/bicleaner-ai) were used for fixing, cleaning, and deduplicating the final version of the corpus.
The corpus is available in three formats: two sentence-level formats, TXT and TMX, and a document-level TXT format. TMX is an XML-based format and TXT is a tab-separated format. They both consist of pairs of source and target segments (one or several sentences) and additional metadata. The following metadata is included in both sentence-level formats:
- source and target document URL;
- paragraph ID which includes information on the position of the sentence in the paragraph and in the document (e.g., “p35:77s1/3” which means “paragraph 35 out of 77, sentence 1 out of 3”);
- quality score as provided by the tool Bicleaner AI (a likelihood of a pair of sentences being mutual translations, provided with a score between 0 and 1);
- similarity score as provided by the sentence alignment tool Bleualign (value between 0 and 1);
- personal information identification (“biroamer-entities-detected”): segments containing personal information are flagged, so final users of the corpus can decide whether to use these segments;
- translation direction and machine translation identification (“translation-direction”): the source segment in each segment pair was identified by using a probabilistic model (https://github.com/RikVN/TranslationDirection), which also determines if the translation has been produced by a machine-translation system;
- a DSI class (“dsi”): information whether the segment is connected to any of Digital Service Infrastructure (DSI) classes (e.g., cybersecurity, e-health, e-justice, open-data-portal), defined by the Connecting Europe Facility (https://github.com/RikVN/DSI);
- English language variant: the language variant of English (British or American, using a lexicon-based English variety classifier - https://pypi.org/project/abclf/) was identified on document and domain level.
Furthermore, the sentence-level TXT format provides additional metadata:
- web domain of the text;
- source and target document title;
- the date when the original file was retrieved;
- the original type of the file (e.g., “html”), from which the sentence was extracted;
- paragraph quality (labels, such as “short” or “good”, assigned based on paragraph length, URL and stopword density via the jusText tool - https://corpus.tools/wiki/Justext);
- information whether the sentence is a heading or not in the original document.
The document-level TXT format provides pairs of documents identified to contain parallel data. In addition to the parallel documents (in base64 format), the corpus includes the following metadata: source and target document URL, a DSI category and the English language variant (British or American).
As opposed to the previous version, this version has more accurate metadata on languages of the texts, which was achieved by using Google's Compact Language Detector 2 (CLD2) (https://github.com/CLD2Owners/cld2), a high-performance language detector supporting many languages. Other tools, used for web corpora creation and curation, have been updated as well, resulting in an even cleaner corpus. The new version also provides additional metadata, such as the position of the sentence in the paragraph and document, and information whether the sentence is related to a DSI. Moreover, the corpus is now also provided in a document-level format.
Notice and take down: Should you consider that our data contains material that is owned by you and should therefore not be reproduced here, please: (1) Clearly identify yourself, with detailed contact data such as an address, telephone number or email address at which you can be contacted. (2) Clearly identify the copyrighted work claimed to be infringed. (3) Clearly identify the material that is claimed to be infringing and information reasonably sufficient in order to allow us to locate the material. (4) Please write to the contact person for this resource whose email is available in the full item record. We will comply with legitimate requests by removing the affected sources from the next release of the corpus.
This action has received funding from the European Union's Connecting Europe Facility 2014-2020 - CEF Telecom, under Grant Agreement No. INEA/CEF/ICT/A2020/2278341. This communication reflects only the author’s view. The Agency is not responsible for any use that may be made of the information it contains
Q-CAT Corpus Annotation Tool 1.5
No full textThe Q-CAT (Querying-Supported Corpus Annotation Tool) is a tool for manual linguistic annotation of corpora, which also enables advanced queries on top of these annotations. The tool has been used in various annotation campaigns related to the ssj500k reference training corpus of Slovenian (http://hdl.handle.net/11356/1210), such as named entities, dependency syntax, semantic roles and multi-word expressions, but it can also be used for adding new annotation layers of various types to this or other language corpora. Q-CAT is a .NET application, which runs on Windows operating system.
Version 1.1 enables the automatic attribution of token IDs and personalized font adjustments.
Version 1.2 supports the CONLL-U format and working with UD POS tags.
Version 1.3 supports adding new layers of annotation on top of CONLL-U (and then saving the corpus as XML TEI).
Version 1.4 introduces new features in command line mode (filtering by sentence ID, multiple link type visualizations)
Version 1.5 supports listening to audio recordings (provided in the # sound_url comment line in CONLL-U
CORDEX inflectional lookup data 1.0
No full textThe inflectional data lookup module serves as an optional component within the cordex library (https://github.com/clarinsi/cordex/) that significantly improves the quality of the results. The module consists of a pickled dictionary of 111,660 lemmas, and maps these lemmas to their corresponding word forms.
Each word form in the dictionary is accompanied by its MULTEXT-East morphosytactic descriptions, relevant features (custom features extracted from morphosytactic descriptions with the help of https://gitea.cjvt.si/generic/conversion_utils and its frequency within the Gigafida 2.0 corpus (http://hdl.handle.net/11356/1320), or Gigafida 1.0 when other information is unavailable. The dictionary is used to select the most frequent word form of a lemma that satisfies additional filtering conditions (ie. find the most utilized word form of lemma "centralen" in singular, i.e."centralni")
Macedonian linguistic training corpus SETimes.MK 0.1
No full textThe SETimes.MK corpus is a sample of 570 sentences from the now unavailable setimes.com website of news articles on topics of South-Eastern Europe. The sentences were manually corrected for sentence splitting and tokenisation, while the morphosyntactic labels (following the MULTEXT-East standard for Macedonian https://nl.ijs.si/ME/V6/msd/html/msd-mk.html) and lemmas were automatically annotated with two iterations of preliminary models for Macedonian in the CLASSLA-Stanza tool (https://pypi.org/project/classla/), after which they were manually corrected. The UPOS+UFEATS morphosyntactic description has been assigned with the mapper available at https://github.com/clarinsi/macedonian-tagset-mapping.
The included sentences have their parallel counterparts inside the Croatian hr500k dataset (http://hdl.handle.net/11356/1792) and the Serbian SETimes.SR dataset (http://hdl.handle.net/11356/1843), and the sentence identifiers can be used to match corresponding sentences.
Please note that the dataset does not completely follow the Universal Dependencies specifications for Macedonian (https://universaldependencies.org/mk/index.html), as the UPOS+FEATS features in the dataset take as their basis the MULTEXT-East specifications, which differ in certain respects from the Universal Dependencies for Macedonian one
Slovene learner corpus KOST 1.0
No full textThe corpus of Slovene as a foreign language KOST (Korpus slovenščine kot tujega jezika) contains 6,311 texts (just over 1 million words) written by adult speakers for whom Slovene is not their first language. This corpus offers insights into Slovene language as produced by those who are still learning it as a second or foreign language, and in particular into the most common errors that occur in this process. KOST therefore aims at all those working with Slovene as a second or foreign language.
The texts were mainly written at lectorates and Slovene as a L2/FL courses. Most of the authors of these texts speak Serbian, Bosnian and Macedonian as their first language, but texts by speakers of other languages are also included. The authors are at different proficiency levels in Slovene, from beginners to advanced.
For each contributor, information is available on gender, year of birth, country, first language and other languages they speak, employment status and education, and prior experience of learning Slovene. For each text, there is also information on the time and circumstances of creation (exam or homework), the programme in which it was produced, input type (digital or hand-written), language level and the grade.
A part of the corpus has also texts available in their corrected version. The tokens of the original and corrected texts are linked (one group of link per paragraph) and the links categorised into 23 error types.
The corpus is availabe in three formats: (1) CoNLL-U for the texts and JSON for the links, (2) TEI encoding of the complete corpus (texts, links), including contributor and text metadata in the TEI header, and (3) the corpus in the original and corrected variants and vertical and registry files, suitable for mounting on CQP-type concordancers. Note that the vertical format does not retain the connection between the original and corrected tokens