Common Language Resources and Technology Infrastructure - Slovenia
Not a member yet
840 research outputs found
Sort by
Linguistically annotated multilingual comparable corpora of parliamentary debates in English ParlaMint-en.ana 4.0
No full textParlaMint-en.ana 4.0 is the English machine translation of the ParlaMint.ana 4.0 (http://hdl.handle.net/11356/1860) set of corpora of parliamentary debates across Europe. The translation is linguistically annotated similarly to the original language corpora (but without UD syntax), and with the addition of USAS semantic tags (https://ucrel.lancs.ac.uk/usas/). Because of the addition of semantic tags the UK corpus (ParlaMint-GB) is also included.
The translation to English was done with EasyNMT (https://github.com/UKPLab/EasyNMT) using OPUS-MT models (https://github.com/Helsinki-NLP/Opus-MT). Machine translation was done on the sentence level, and includes both speeches and transcriber notes, including headings. Note that corpus metadata is mostly available both in the source language and in English. The linguistic annotation of the speeches, i.e. tokenisation, tagging with UD PoS and morphological features, lemmatisation, and NER annotation was done with Stanza (https://stanfordnlp.github.io/stanza/) using the conll03 model (4 classes). The annotation of MWEs (phrases) and tokens with USAS tags was done with pyMusas (https://github.com/ucrel/pymusas).
Note that the English in the corpora contains typical NMT errors, including factual errors even when high fluency is achieved, and any use of this corpus should take the machine translation limitations into account.
The files associated with this entry include the machine translated and linguistically annotated corpora in several formats: the corpora in the canonical ParlaMint TEI XML encoding; the corpora in the derived vertical format (for use with CQP-based concordancers, such as CWB, noSketch Engine or KonText); and the corpora in the CoNLL-U format with TSV speech metadata. The CoNLL-U files include MT-generated word-alignment and pyMusas USAS tags, as well as the tags and lemmas produced for the purposes of semantic tagging by Spacy (https://spacy.io/), when they are different from the default annotations. Also included is the 4.0 release of the sample data and scripts available at the GitHub repository of the ParlaMint project at https://github.com/clarin-eric/ParlaMint and the log files produced in the process of building the corpora for this release. The log files show e.g. known errors in the corpora, while more information about known problems is available in the (open) issues at the GitHub repository of the project.
As opposed to the previous version 3.0, this version adds corpora for United Kingdom (GB), Spain (ES), Finland (FI) and the Basque Country (ES-PV); extends the corpora for Austria (AT), Czechia (CZ), Hungary (HU), and Ukraine (UA) with more recent data; adds USAS semantic tags to all corpora; adds metadata to political parties and parliamentary groups on left-to-right political orientation from Wikipedia, as well as CHES variables; adds the information on whether a speaker was a minister and when for the corpora that previously lacked this information. The TEI encoding of some details has also changed, and many errors found in 3.0 corpora have been corrected
List of Linked Senses from Open Slovene WordNet and the Digital Dictionary Database of Slovene 1.0
No full textThis is a list containing sense IDs from Open Slovene WordNet 1.0 (OSWN; http://hdl.handle.net/11356/1888) and the Digital Dictionary Database of Slovene (DDDS) developed by the Centre for Language Resources and Technologies of the University of Ljubljana.
The file consists of four columns containing the following data:
- synset ID from the Open Slovene WordNet;
- sense ID from the Digital Dictionary Database of Slovene;
- lexical unit ID from the Digital Dictionary Database of Slovene;
- form of the lexical unit from the Digital Dictionary Database of Slovene.
The list allows the end user to access OSWN data through the DDDS API (documented at https://wiki.cjvt.si/books/digital-dictionary-database/chapter/rest-api), namely which senses and lexical units from DDDS are assigned to a certain synset ID in OSWN
Annotated collocation candidates for three common syntactic structures in Slovene
No full textThis resource contains 713,310 collocation candidates, which were automatically extracted from the Gigafida 2.0 corpus (http://hdl.handle.net/11356/1320) and annotated whether they are legitimate collocations or not. The collocation candidates belong to three syntactic structures that are among the most common and semantically most informative collocational structures in the Slovenian language:
- Verb + Noun in accusative (Structure_ID = 23; Structure_name = gg-s4;#_1_#-1_2_dve). Contains 163,229 annotated collocation candidates.
- Adjective + Noun (Structure_ID = 34; Structure_name = p0-s0;2_1_dol-#_2_#). Contains 342,714 annotated collocation candidates.
- Noun + Noun in genitive (Structure_ID = 53; Structure_name = s0-s2;#_1_#-1_2_dol). Contains 207,367 collocation candidates.
Structure IDs and structure names are provided as used in the Digital Dictionary Database at the Centre for Language Resources and Technologies at the University of Ljubljana (https://www.cjvt.si/en/).
In the annotation, three types of decision were possible:
a) YES. The collocation candidate is a legitimate collocation, i.e., it is statistically relevant, represents the right syntactic structure, and shows meaningful but transparent semantic word combination.
b) EXTENDED. The collocation candidate may be considered a collocation but in most cases or always requires a third element.
c) NO. The collocation candidate is not a collocation. This can be for example because of a problem in lemmatisation, morphosyntactic annotation etc., or because the candidate is a compound, phrase etc., i.e., some other multiword unit.
It should be noted that the annotation did not consider the criterion of collocation relevance, e.g., which collocations would make it into a dictionary or a related source. We consider this as a next step in using this data. However, part of the relevance has been included in the selection method, as the collocation candidates were selected using noun, adjective and verb headwords from Collocation Dictionary of Modern Slovene 1.0 (http://hdl.handle.net/11356/1250), taking up to top 30 collocations with a minimum frequency of 4 for each headword per syntactic structure
Face-domain-specific automatic speech recognition models
No full textThis entry contains all the files required to implement face-domain-specific automatic speech recognition (ASR) applications using the Kaldi ASR toolkit (https://github.com/kaldi-asr/kaldi), including the acoustic model, language model, and other relevant files. It also includes all the scripts and configuration files needed to use these models for implementing face-domain-specific automatic speech recognition.
The acoustic model was trained using the relevant Kaldi ASR tools (https://github.com/kaldi-asr/kaldi) and the Artur speech corpus (http://hdl.handle.net/11356/1776; http://hdl.handle.net/11356/1772). The language model was trained using the domain-specific text data involving face descriptions obtained by translating the Face2Text English dataset (https://github.com/mtanti/face2text-dataset) into the Slovenian language. These models, combined with other necessary files like the HCLG.fst and decoding scripts, enable the implementation of face-domain-specific ASR applications.
Two speech corpora ("test" and "obrazi") and two Kaldi ASR models ("graph_splosni" and "graph_obrazi") can be selected for conducting speech recognition tests by setting the variable "graph" and "test_sets" in the "local/test_recognition.sh" script.
Acoustic speech features can be extracted and speech recognition tests can be conducted using the "local/test_recognition.sh" script.
Speech recognition test results can be obtained using the "results.sh" script.
The KALDI_ROOT environment variable also needs to be set in the script "path.sh" to set the path to the Kaldi ASR toolkit installation folder
Serbian Web Corpus PDRS 1.0
No full textPDRS 1.0 is a web corpus based on crawling the .rs domain. Crawling has been done in September and October 2022 with BootCat. As search terms, appr. 2,800 word forms with a frequency between 5,000 and 500,000 in srWaC have been used.
The texts are deduplicated, cyrillic texts have been transliterated into the Latin alphabet. The linguistic processing was done with the CLASSLA package (https://github.com/clarinsi/classla) for tokenization, lemmatization and morpho-syntactic tagging (both MULTEXT-East and Universal Dependencies).
In addition, some 80% of the URLs are manually tagged for 10 different types of sources ("area"): media (media outlets with several posts daily), inform (topic-centered sites with infrequent posts - maximum 3 per day), company (presentations of companies), state (websites of government bodies on nationa, regional and local level), forum (forum posts), portal (topic-centered portals without daily coverage), science (scientific publications), shop (with descriptions of products), database (knowledge bases, dictionaries, databases and similar) and community (NGOs, fan clubs, associations and other).
The corpus is distributed in the CoNLL-U format in batches of appr. 2x50 mio. tokens
The CLASSLA-Stanza model for lemmatisation of non-standard Slovenian 2.1
No full textThis model for lemmatisation of non-standard Slovenian was built with the CLASSLA-Stanza tool (https://github.com/clarinsi/classla) by training on the SUK training corpus (http://hdl.handle.net/11356/1747) and the Janes-Tag corpus (http://hdl.handle.net/11356/1732), using the CLARIN.SI-embed.sl word embeddings (http://hdl.handle.net/11356/1204) that were expanded with the MaCoCu-sl Slovene web corpus (http://hdl.handle.net/11356/1517). These corpora were additionally augmented for handling missing diacritics by repeating parts of the corpora with diacritics removed. The estimated F1 of the lemma annotations is ~91.45.
The difference to the previous version of the model is that the model was trained on the SUK training corpus and the 3.0 version of Janes-tag, uses new embeddings and the new version of the Slovene morphological lexicon Sloleks 3.0 (http://hdl.handle.net/11356/1745)
Croatian linguistic training corpus hr500k 2.0
No full textThe hr500k training corpus contains about 500,000 tokens manually annotated on the levels of
tokenisation, sentence segmentation, morphosyntactic tagging, lemmatisation and named entities. About half of the corpus is also manually annotated with syntactic dependencies. A subset of the syntactically annotated corpus is also annotated for multi-word expressions. Furthermore, about a fifth of the corpus is annotated with semantic role labels.
The annotation formalisms followed in the hr500k corpus are (1) the MULTEXT-East V6 morphosyntactic specifications for the Serbo-Croatian macro-language, https://nl.ijs.si/ME/V6/msd/, (2) the UDv2 Guidelines, http://universaldependencies.org/guidelines.html, (3) the Janes annotation guidelines for named entities, https://nl.ijs.si/janes/wp-content/uploads/2017/09/SlovenianNER-eng-v1.1.pdf, (4) the PARSEME guidelines for annotating multi-word expressions, https://parsemefr.lis-lab.fr/parseme-st-guidelines/1.3/ and (4) the semantic role labelling annotation protocol for Slovenian and Croatian, https://www.sdjt.si/wp/wp-content/uploads/2018/09/JTDH-2018_Gantar-et-al_Towards-Semantic-Role-Labeling-in-Slovene-and-Croatian.pdf.
Different to the previous version of the dataset, it is now encoded in the conllup format, as are other linguistic training datasets for Croatian and Serbian. The PARSEME multi-word expression annotation layer was added as well, together with countless corrections of labels on all available levels.
The continuous improvement of this dataset is led by the CLASSLA knowledge centre for South Slavic languages (https://www.clarin.si/info/k-centre/) and the ReLDI Centre Belgrade
Montenegrin web corpus MaCoCu-cnr 1.0
No full textThe Montenegrin web corpus MaCoCu-cnr 1.0 was built by crawling the ".me" internet top-level domain in 2021 and 2022, extending the crawl dynamically to other domains as well. The crawler is available at https://github.com/macocu/MaCoCu-crawler.
Considerable effort was devoted into cleaning the extracted text to provide a high-quality web corpus. This was achieved by removing boilerplate (https://corpus.tools/wiki/Justext) and near-duplicated paragraphs (https://corpus.tools/wiki/Onion), discarding very short texts as well as texts that are not in the target language. The dataset is characterized by extensive metadata which allows filtering the dataset based on text quality and other criteria (https://github.com/bitextor/monotextor), making the corpus highly useful for corpus linguistics studies, as well as for training language models and other language technologies.
In XML format, each document is accompanied by the following metadata: title, crawl date, url, domain, file type of the original document, distribution of languages inside the document, and a fluency score based on a language model. The text of each document is divided into paragraphs that are accompanied by metadata on the information whether a paragraph is a heading or not, metadata on the paragraph quality (labels, such as “short” or “good”, assigned based on paragraph length, URL and stopword density via the jusText tool - https://corpus.tools/wiki/Justext) and fluency (score between 0 and 1, assigned with the Monocleaner tool - https://github.com/bitextor/monocleaner), the automatically identified language of the text in the paragraph, and information whether the paragraph contains sensitive information (identified via the Biroamer tool - https://github.com/bitextor/biroamer).
The corpus can be easily read with the prevert parser (https://pypi.org/project/prevert/).
Notice and take down: Should you consider that our data contains material that is owned by you and should therefore not be reproduced here, please: (1) Clearly identify yourself, with detailed contact data such as an address, telephone number or email address at which you can be contacted. (2) Clearly identify the copyrighted work claimed to be infringed. (3) Clearly identify the material that is claimed to be infringing and information reasonably sufficient in order to allow us to locate the material. (4) Please write to the contact person for this resource whose email is available in the full item record. We will comply with legitimate requests by removing the affected sources from the next release of the corpus.
This action has received funding from the European Union's Connecting Europe Facility 2014-2020 - CEF Telecom, under Grant Agreement No. INEA/CEF/ICT/A2020/2278341. This communication reflects only the author’s view. The Agency is not responsible for any use that may be made of the information it contains
PyTorch model for Slovenian Named Entity Recognition SloNER 1.0
No full textThe SloNER is a model for Slovenian Named Entity Recognition. It is is a PyTorch neural network model, intended for usage with the HuggingFace transformers library (https://github.com/huggingface/transformers).
The model is based on the Slovenian RoBERTa contextual embeddings model SloBERTa 2.0 (http://hdl.handle.net/11356/1397). The model was trained on the SUK 1.0 training corpus (http://hdl.handle.net/11356/1747).The source code of the model is available on GitHub repository https://github.com/clarinsi/SloNER
Natural Language 2 Semantic Hypergraph Dataset NL2SH 1.0
No full textNL2SH (Natural Language to Semantic Hypergraph) dataset can be used to build and evaluate methods for knowledge extraction and representation based on a semantic hypergraph. Each sentence has natural language annotations and dedicated semantic hyperedge. Majority of the sentences used in this dataset are taken from the following sources:
* John Eastwood, Oxford Guide to English Grammar, Oxford University Press, 2002.
* Andrew Redford, An Introduction to English Sentence Structure, Cambridge University Press, 2009.
* Essential English Grammar, Philip Gucker, Dover Publications, Inc. New York, 1966
Natural language annotations are:
* sent_i - id of the sentence
* tok_i - id of the token in the sentence
* word - token text
* space - does space follows the token
* lemma - lemma of the token
* pos - Universal POS tags (https://universaldependencies.org/u/pos/)
* tag - Penn Treebank tags (https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html)
* dep - ClearNLP depedency labels (https://github.com/clir/clearnlp-guidelines/blob/master/md/specifications/dependency_labels.md)
* head - id of the token which is a dependency head of the current token
* ner - named entities (https://catalog.ldc.upenn.edu/docs/LDC2013T19/OntoNotes-Release-5.0.pdf)
* roleset - roleset of a verb frame (https://propbank.github.io/v3.4.0/frames/)
* srl - semantic role labels with IOB annotation (https://verbs.colorado.edu/propbank/EPB-Annotation-Guidelines.pdf)
* coref - coreference labels with IOB annotation * synset - WordNet's synsets (https://wordnet.princeton.edu)
The annotations for semantic hypergraph elements primarily adhere to the annotation guidelines of the Graphbrain project (https://graphbrain.net/manual/notation.html). However, atom annotations are modified and at the end contains:
* label,
* type and optional subtype,
* type specific atom roles,
* type specific additional information,
* named entit