Common Language Resources and Technology Infrastructure - Slovenia
Not a member yet
840 research outputs found
Sort by
Croatian-English parallel corpus MaCoCu-hr-en 1.0
No full textThe Croatian-English parallel corpus MaCoCu-hr-en 1.0 was built by crawling the ".hr" internet top-level domain in 2021, extending the crawl dynamically to other domains as well.
All the crawling process was carried out by the MaCoCu crawler (https://github.com/macocu/MaCoCu-crawler). Websites containing documents in both target languages were identified and processed using the tool Bitextor (https://github.com/bitextor/bitextor). Considerable efforts were devoted into cleaning the extracted text to provide a high-quality parallel corpus. This was achieved by removing boilerplate and near-duplicated paragraphs and documents that are not in one of the targeted languages. Document and segment alignment as implemented in Bitextor were carried out, and BicleanerAI (https://github.com/bitextor/bicleaner-ai) and Bifixer (https://github.com/bitextor/bifixer) were used for fixing, cleaning, and deduplicating the final version of the corpus.
While the TXT format consists solely of pairs of source and target segments (one or several sentences), each segment pair in the TMX format is accompanied by the following metadata:
- source and target document URL;
- quality score as provided by the tool BicleanerAI;
- translation direction identification: the source segment in each segment pair was identified by using a probabilistic model;
- personal information identification (“biroamer-entities”): segments containing personal information are flagged, so final users of the corpus can decide whether to use these segments;
- language variants: the language variant of English (British or American) was identified for every segment pair on document and domain level.
Notice and take down: Should you consider that our data contains material that is owned by you and should therefore not be reproduced here, please: (1) Clearly identify yourself, with detailed contact data such as an address, telephone number or email address at which you can be contacted. (2) Clearly identify the copyrighted work claimed to be infringed. (3) Clearly identify the material that is claimed to be infringing and information reasonably sufficient in order to allow us to locate the material. (4) Please write to the contact person for this resource whose email is available in the full item record. We will comply with legitimate requests by removing the affected sources from the next release of the corpus.
This action has received funding from the European Union's Connecting Europe Facility 2014-2020 - CEF Telecom, under Grant Agreement No. INEA/CEF/ICT/A2020/2278341. This communication reflects only the author’s view. The Agency is not responsible for any use that may be made of the information it contains
Choice of plausible alternatives dataset in Macedonian COPA-MK
No full textThe COPA-MK dataset (Choice of plausible alternatives in Macedonian) is a translation of the English COPA dataset (https://people.ict.usc.edu/~gordon/copa.html) by following the XCOPA dataset translation methodology (https://arxiv.org/abs/2005.00333).
The dataset consists of 1,000 premises (My body cast a shadow over the grass), each given a question (What is the cause? / What happened as a result?), and two choices (The sun was rising; The grass was cut), with a label encoding which of the choices is more plausible given the annotator or translator (The sun was rising).
The dataset follows the same format as the Croatian COPA-HR dataset (http://hdl.handle.net/11356/1404). It is split into training (400 instances), validation (100 instances) and test (500 instances) JSONL files.
Translation quality was ensured with the help of the ReLDI Centre Belgrade
Corpus of Slovenian textbooks ccUčbeniki 1.0
No full textccUčbeniki includes 32 openly available texbooks for Slovenian primary and secondary education, published by the Slovenian National Education Institute in 2014-2015. The textbooks, prepared by various authors, cover different subjects as is documented in the ccucbeniki-metadata file.
The corpus was linguistically annotated with the CLASSLA v1.1.1 pipeline (https://github.com/clarinsi/classla/) at the levels of tokenization, sentence segmentation, lemmatization, MULTEXT-East v6 MSD-tags (https://nl.ijs.si/ME/V6/msd/html/msd-sl.html), JOS dependency syntax (https://nl.ijs.si/jos/bib/jos-skladnja-navodila.pdf), and named entities (https://nl.ijs.si/janes/wp-content/uploads/2017/09/SlovenianNER-eng-v1.1.pdf). The idea is to provide comparably annotated pedagogically-relevant corpora that can be used for different tasks in the field of language didactics and NLP.
The corpus is available in CoNLL-U and vertical formats. The CoNLL-U format contains one document per file (and separately text metadata as a TSV file) and the vertical format contains concatenated documents in one large file. The registry file ccucbeniki.regi for the vertical format is compatible with the LIST 1.2 corpus extraction tool (http://hdl.handle.net/11356/1276)
Service for querying dependency treebanks Drevesnik 1.0
No full textDrevesnik (https://orodja.cjvt.si/drevesnik/) is an online service for querying syntactically parsed corpora in Slovenian using the Universal Dependencies annotation scheme with easy-to-use query language on the one hand and user-friendly graph visualizations on the other. It is based on the open-source dep_search tool (https://github.com/TurkuNLP/dep_search), which was localized and modified so as to also support querying by JOS morphosyntactic tags, random distribution of results, and filtering by sentence length.
The source code and the documentation for the search backend and the web user interface are publicly available on the CLARIN.SI GitHub repository https://github.com/clarinsi/drevesnik. This submission corresponds to release 1.0: https://github.com/clarinsi/drevesnik/releases/tag/1.0
ASR database ARTUR 0.1 (audio)
No full textARTUR is a speech database designed for the needs of automatic speech recognition for the Slovenian language. The database includes 1,035 hours of speech, although only 840 hours are transcribed, while the remaining 195 hours are without transcription.
The data is divided into 4 parts:
(1) approx. 520 hours of read speech, which includes the reading of pre-defined sentences, selected from the Gigafida 2.0 corpus (http://hdl.handle.net/11356/1320); each sentence is contained in one file; speakers are demographically balanced; spelling is included in special files; all with manual transcriptions;
(2) approx. 204 hours of public speech, which includes media recordings, online recordings of conferences, workshops, education videos, etc.; 56 hours are manually transcribed;
(3) approx. 110 hours of private speech, which includes monologues and dialogues between two persons, recorded for the purposes of the speech database; the speakers are demographically balanced; two subsets for domain-specific ASR (i.e., smart-home and face-description) are included; 63 hours are manually transcribed;
(4) approx. 201 hours of parliamentary speech, which includes recordings from the Slovene National Assembly, all with manual transcriptions.
Audio files are WAV 44,1 kHz, pcm, 16-bit, mono.
This entry includes the recordings only; transcriptions are available at http://hdl.handle.net/11356/1718
Slovenian commonsense reasoning model SloMET-ATOMIC 2020
No full textThe SloMET-ATOMIC 2020 is a Slovene commonsense reasoning model that is able to predict commonsense descriptions in a natural language for a given input sentence.
The model is an adaptation of the Slovene GPT-2 model (https://huggingface.co/cjvt/gpt-sl-base) that has been finetuned using the SloATOMIC 2020 corpus (http://hdl.handle.net/11356/1724), consisting of 1.33M everyday interence knowledge tuples about entities and events.
The released model is a pytorch neural network model, intended for usage with the transformers library (https://github.com/huggingface/transformers)
Slovene corpus for general relation extraction SloREL 1.1
No full textThe SloREL corpus contains annotations for training relation extraction models on Slovene documents. It contains documents from Slovene Wikipedia with annotated entities and relations. We constructed the annotations using a semi-supervised process based on linking the documents to the WikiData knowledge graph. The corpus contains 244,437 sentences from Slovene Wikipedia pages. We also provide 896 additional sentences collected from the 24ur.com news website with annotated and linked entities, which do not contain annotated relations and are meant for additional testing of the models. The entities in our corpus are linked to the entities in the WikiData knowledge graph which is useful for models that take advantage of additional knowledge from a knowledge graph. Altogether the corpus comprises 245,333 sentences with 813,952 relations and 1,616,193 entities.
The corpus comprises of multiple documents:
- schema-definition.xsd: defines the structure of the xml documents containing relation annotations.
- SloREL/train.xml: training portion of the SloREL corpus containing
Wikipedia documents
- SloREL/test.xml: testing portion of the SloREL corpus containing
Wikipedia documents
- SloREL/validation.xml: validation portion of the SloREL corpus containing
Wikipedia documents
- 24ur/24ur.xml: additional sentences from the 24ur.com news articles
Changes in version 1.1:
We fixed the mislabeled relation types that were present in the previous version of the dataset. We also rearranged the archive to make the structure more understandable
Slovenian-English glossary of education
No full textThe glossary, which contains 1028 dictionary entries, is a terminological collection of Slovenian terms with English equivalents in the field of education. The collection was created with the aim of unifying the use of English terms when describing the Slovenian education system in a non-Slovene environment. Terms that are not listed as stand-alone entries are less appropriate for use
NeMo Punctuation and Capitalisation service RSDO-DS2-P&C-API 1.0
No full textPunctuation and Capitalisation service for NeMo models. For more details about building such models, see the official NVIDIA NeMo documentation (https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/nlp/punctuation_and_capitalization.html) and NVIDIA NeMo GitHub (https://github.com/NVIDIA/NeMo). A model for punctuation and capitalisation restoration in lowercased non-punctuated Slovene text can be downloaded from http://hdl.handle.net/11356/1735.
The service accepts as input either a single string or list of strings for which punctuation and capitalisation should be restored. The result will be in the same format as the request, either a single string or list of strings. The maximal accepted text length is 5000c. Note that punctuation and capitalization of one 5000c text block on cpu will take advantage of all available cores and may take ~30s (on a system with 24 vCPU). See the service README.md for further details
Slovenian text summarization models
No full textA text summarisation task aims to convert a longer text into a shorter text while preserving the essential information of the source text. In general, there are two approaches to text summarization. The extractive approach simply rewrites the most important sentences or parts of the text, whereas the abstractive approach is more similar to human-made summaries. We release 5 models that cover extractive, abstractive, and hybrid types:
Metamodel: a neural model based on the Doc2Vec document representation that suggests the best summariser.
Graph-based model: unsupervised graph-based extractive approach that returns the N most relevant sentences.
Headline model: a supervised abstractive approach (T5 architecture) that returns returns headline-like abstracts.
Article model: a supervised abstract approach (T5 architecture) that returns short summaries.
Hybrid-long model: unsupervised hybrid (graph-based and transformer model-based) approach that returns short summaries of long texts.
Details and instructions to run and train the models are available at https://github.com/clarinsi/SloSummarizer.
The web service with a demo is available at https://slovenscina.eu/povzemanje