Script consists of 2 parts:
article parser
aligner
Required software (install before using script):
yalign
additional Ubuntu packages:
mongodb
ipython
python-nose
python-werkzeug
Wiki article parser
Article parser works in 2 steps:
Extracts articles from wiki dumps
Saves extracted articles to local DB (Mongo DB)
Before using parser, wiki dumps should be downloaded and extracted to some directory (directory should contain *.xml, *.sql files). For each language 2 dump files should be downloaded - articles and language link dumps, here is examples:
PL:
http://dumps.wikimedia.org/plwiki/latest/plwiki-latest-pages-articles.xml.bz2
http://dumps.wikimedia.org/plwiki/latest/plwiki-latest-langlinks.sql.gz
EN:
http://dumps.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles.xml.bz2
http://dumps.wikimedia.org/enwiki/latest/enwiki-latest-langlinks.sql.gz
IMPORTANT NOTE: Engilsh dumps after extraction will require about 50 Gb of free space. During parsing parser can require up to 8 Gb ram.
Article parser have option "main language" - its language for which articles extracted from other languages only if it exist in main language. Eg. if main language is PL, then article extractor first extracts all article for PL, then article for other languages and only if such articles exists in PL translation. This reduces space requirements.
For help use:
pythonparsewikidumps.py−hExamplecommand: python parse_wiki_dumps.py -d ~/temp/wikipedia_dump/ -l pl -v
Wikipedia aligner
Aligner can be used when article extracted from dumps.
Aligner takes article pairs for given language pair, aligns text and saves parallel corpara to 2 files. Option "-s" can be used to limit number of symbols in file (by default size is 50000000 symbols, thats around 50-60Mb)
By default aligner tries to continue aligning where it was stopped, to force aligning from begining need to use "--restart" key
For help use:
pythonalign.py−hExamplecommand: python align.py -o wikipedia -l en-pl -v
Euronews crawler
Crawler finds links to articles using euronews archive http://euronews.com/2004/, and in parallel extracts and saves article texts to DB.
For help use:
pythonparseeuronews.py−hExamplecommand: python parse_euronews.py -l en,pl -v
Euronews aligner
Starting aligner for euronews articles:
pythonalign.py−oeuronews−len−pl−vSavingarticlesinplaintextScript"saveplaintext.py"canbeusedtosaveallarticlesinplaintextformat,itacceptspathforsavingarticles,languagesofarticlestobesaved,andsourceofarticles(euronews,wikipedia).Forhelpuse: python save_plain_text.py -h
Example command:
pythonsaveplaintext.py−len,pl−r[path]−oeuronewsYalignselectionThisscripttriesrandomparametersformodelofyaligninordertogetbestparametersforaligningprovidedtextsamples.Beforeusingyalignselectionscriptneedtopreparearticlesamplesusingpreparerandomsampling.pyscript.Creatingfolderwitharticlesamplescanbedonewiththiscommand: python prepare_random_sampling.py -o wikipedia -c 10 -l ru-en -v
-o wikipedia - source of articles can be wikipedia or euronews
-c 10 - number of articles to extract
-l ru-en - languages to extract
This script will create "article_samples" folder with articles files, then you can create manually aligned files (you need align article of second language), for this example you need to align "en" file, files named "_orig" - should be left unmodified
Then manual aligning is ready you can run selection script here is example:
pythonyalignselection.py−−samplesarticlesamples/−−lang1ru−−lang2en−−threshold0.1536422609112349e−6−−thresholdstep0.0000001−−thresholdstepcount10−−penalty0.014928930455303857−−penaltystep0.0001−−penaltystepcount1−mru−enHereiswhateachparametermeans:−−samplesarticlesamples/−pathtoarticlesamplesfolder−−lang1ru−−lang2en−languagestoalign(articlesofsecondlanguageshouldbealignedmanually,scriptwillbeusing"??orig"files,alignthemautomaticallyandwillcomparewithmanuallyaligned)−−threshold0.1536422609112349e−6−thresholdvalueofmodel,selectionwillbemadearoundthisvalue−−thresholdstep0.0000001−stepofchangingvalue−−thresholdstepcount10−numberofstepstocheckbelowandabovevaule,egifvalue10,step1,andcount2,scriptwillcheck89101112sameparametersforpenalty−mru−en−pathtoyalignmodelAlsoyoucanuse(totweakcomparisonoftextlinesinfiles):−−lengthand−−similarity−−length−mindiffirenceinlengthinordertomarklinessimilar,1−samelength,0.5−atleasthalfoflength−−similarity−similarityoftextinlines,1−exactlysame,0−completelydifferent.Forsimilaritychecksentencescomparedassequenceofcharacters.Ithasmultiprocessingsupportalready.Use−toptiontosetnumberofthreads,bydefaultitsetsnumberofthreadsequaltonumberofCPU.foradditionalparametersyoucanuse′−h′key.Thenyalignselection.pyscriptwillfinishworkitwillproducecsvfile,withfirstcolumnequaltothreshold,secondcolumnequaltopenalty,andthirdissimilarityforthisparameters.AlignwithHUNALINGmethodInordertousehunalignyouneedadd"−−hunalign"optioninalign.pyscript,hereisexample: python align.py -l li-hu -r align_result -o wikipedia --hunalign
In my empirical study it provides better results when articles are translations of each other or simillar in leghth and content.
Align From fodler
For aligning already aligned texts using hunalign:
Command exmaple is:
$ python align_aligned_using_hunalign.py source/ target/
Final info
Wołk, K., & Marasek, K. (2015, September). Tuned and GPU-accelerated parallel data mining from comparable corpora. In International Conference on Text, Speech, and Dialogue (pp. 32-40). Springer International Publishing.
http://arxiv.org/pdf/1509.08639
For more detailed usage instruction see howto.pdf.
For any questions: | Krzysztof Wolk | [email protected]
New upgraded version of the Polish-Lithuanian Parallel Corpus (http://hdl.handle.net/11321/309) with extra files and features (Including General, Medical, Technical, Legal, Business Terms … and Slang Expressions). The Polish-Lithuanian Parallel Corpus "2" contains 16 543 470 words (PL 8 697 956 + LT 7 845 514)
OSL is a markup universal language for linguistic description of any object in terms of structure and behavior. The kernel is presented and subsets for IT system, business and human-being. Special atention is given to geometrical structures suggested as new data types
WordneLoom 2.0 executable files for plWordnet 4.0.
Source code available at https://github.com/CLARIN-PL/WordnetLoom
WordnetLoom – is an wordnet editor application built for the needs of the construction of a the largest Polish wordnet called plWordNet. WordnetLoom provides two means of interaction: a form-based, implemented initially, and a visual, graph-based introduced recently. The visual, graph-based interactive presentation of the wordnet structure enables browsing and its direct editing on the structure of lexico-semantic relations and synsets. WordnetLooms works in a distributed environment, i.e. several linguists can work simulanuously from different sites on the same central database
Do you manage Open Research Online? Become a CORE Member to access insider analytics, issue reports and manage access to outputs from your repository in the CORE Repository Dashboard! 👇
No full text
No full text
No full text
No full text
No full text
No full text
No full text
No full text
No full text