338 research outputs found
Inter-rater reliability of the Italian version of the Paykel Scale of stressful life events
No full textThe inter-rater reliability of the Italian version of the Scale for evaluating stressful life events, developed by Paykel, was studied in 15 psychiatric patients and in 15 normal subjects. Agreement between the two raters was satisfactory for total number of events, impact and independence of event areas and categories. These preliminary findings suggest that the Scale is a reliable instrument for the assessment of life events in Italian samples
Maligne Lymphome und Pseudolymphome der Haut : Bericht über 25 Fälle der Abteilung Dermatologie der Medizinischen Fakultät der Rheinisch-Westfälischen Technischen Hochschule Aachen in der Zeit von 1965 - 1987
No full textLe conoscenze e i bisogni informativi dei pazienti con diagnosi di schizofrenia esplorati attraverso il metodo dei focus group
No full textAims - Psychiatric patients often are not informed about their diagnosis and their involvement in the decision making process is rare. Aim of the study was to explore the informative needs of patients with schizophrenia and the knowledge about their illness. Method - Three focus groups were conducted with 25 long-stay patients with the diagnosis of schizophrenia, attending the Mental Health Centre of the South-Verona Community-based Mental Health Service. The group discussions were audiotaped and transcribed. Results - The authors identified 18 different thematic categories which were used by two raters to classify all patient contributions. The interrater reliability was satisfactory. The qualitative analysis evidenced that patients have little knowledge about their illness. Patients had confuse and vague ideas on schizophrenia but their knowledge on drug names, dosages and side effects appeared precise and detailed. Several patients have looked for information in encyclopedias and medical dictionaries. Conclusion - The findings suggest a need of patients affected by schizophrenia for an information exchange with their psychiatrists that takes into account their informative needs, corrects wrong beliefs and actively involves them in therapeutic decisions
Do experience and text quality matter for raters' decision-making behaviors?
No full textThis study examines the decision-making behaviors of raters with varying levels of experience while assessing EFL essays of distinct qualities. The data were collected from 28 raters with varying levels of rating experience and working at the English language departments of different universities in Turkey. Using a 10-point analytic rubric, each rater voice-recorded their thoughts through think-aloud protocols (TAPs) while scoring 16 essays of distinct text qualities and provided brief score explanations. Data collected from TAPs were analyzed by using a coding scheme adapted from Cumming, Kantor, and Powers (2002). The results revealed that text quality has a larger effect than rating experience on raters' decision-making behaviors. In addition, raters prioritized aspects of style, grammar, and mechanics when rating low-quality essays, but emphasized rhetoric and their general impressions of the text for high-quality essays. Furthermore, low-experienced raters differed more in their behaviors while assessing scripts of distinct qualities than did the medium- and high-experienced groups. The findings suggest that raters' scoring behaviors might evolve with practice, resulting in less variation in their decisions. As such, this research provides implications for developing strategy-based rater training programs, which might help to increase consistency across raters of different experience levels.International Research Foundation for English Language Education (TIRF)The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The first author received financial support for the research from The International Research Foundation for English Language Education (TIRF; tirfonline.org) as a doctoral dissertation grant (DDG)
A Study of Raters’ Behavior in Scoring L2 Speaking Performance: Using Rater Discussion as a Training Tool
No full textThe studies conducted so far on the effectiveness of resolution methods including the discussion method in resolving discrepancies in rating have yielded mixed results. What is left unnoticed in the literature is the potential of discussion to be used as a training tool rather than a resolution method. The present study addresses this research gap by exploring the data coming from rating behaviors of 5 Iranian raters of English. Qualitative analysis of the data indicated that the discussion method can serve the function of training raters. It helped raters rate more easily, quickly and confidently. Furthermore, it helped them improve their understanding and application of the rating criteria and enabled them justify their scoring decisions. Many-faceted Rasch analysis also supported the beneficial effects of discussion in terms of improvement in raters’ severity, consistency in scoring, and the use of scale categories. The findings provide insight into the potential of discussion to be used as a training tool especially in EFL contexts in which lack of access to expert raters can be an obstacle to holding training programs. The author argues for future studies to focus on how discussion may function depending on the rating scale used
La comunicazione della diagnosi di schizofrenia.Il punto di vista degli psichiatri raccolto attraverso il metodo dei focus group.
No full textEvidence from the literature show that patients affected by psychosis rarely are informed about their diagnosis and/or involved in the decision making process regarding the therapeutic program. The aim of the present study is to investigate psychiatrists' clinical experiences, beliefs and attitudes towards communicating the diagnosis to patients affected by psychosis. METHOD: Three focus groups were conducted with 28 psychiatrists of different levels of expertise (10 senior psychiatrists and 18 psychiatrists in training), all working at the South-Verona Community-based Mental Health Service. The group discussions were audiotaped and transcribed. The authors derived different thematic categories which were used by two raters to classify all contributions of the psychiatrists. RESULTS: The main themes of interest which emerged were: experience with the communication of the diagnosis of psychosis and main communication strategies used; obstacles and personal difficulties in giving such information. CONCLUSIONS: In the psychiatric setting the disclosure of the diagnosis of psychosis implies different clinical and ethical issues and is felt as highly problematic. These findings suggest a need of psychiatrists to involve the psychotic patient in the information giving process in order to learn his/her informative needs which are the starting point for individually tailored information. The authors have no potential conflict of interest related to the subject of the paper
Examining the reliability of the landing error scoring system with raters using the standardized instructions and scoring sheet
No full textContext:Dynamic movement-based screens,such as the Landing Error Scoring System(LESS),are becoming more widely used in research and practical settings. Currently, 3 studies have examined the reliability of the LESS. These studies have reported good interrater and intrarater reliability. However, all 3 studies involved raters, who were founders of the LESS. Therefore, it is unclear whether the reliability reported reflects that which would be observed with practitioners without such specialized and intimate knowledge of the screen and only using the standardized set of instructions. Objective: To investigate the interrater and intrarater reliability of the final score and the individual scoring criteria of the LESS. Design: Reliability protocol. Setting: Controlled laboratory. Participants: Two raters scored 30 male participants (age=21.8 [3.9] y; height=1.75 [0.46] m; mass= 75.5 [6.6] kg) involved in a variety of college sports. Main Outcome Measure: Two raters using only the standardized scoring sheet assessed the interrater reliability of the total score and individual scoring criteria independently of each other.The principal author scored the videos again 6 weeks later for the intrarater reliability component of the study. Intervention: Participants performed a drop box landing from a 30-cm box was recorded with a video camera from the front and side views. Results: The intraclass coefficients interrater and intrarater reliability for the total scores were excellent (intraclass co efficients range=.95and .96;SEM=1.01and1.02).The individual scoring criteria of the LESShad between moderate and perfect agreementng kappa statistics (κ=.41–1.0). Conclusion: The final score and individual scoring criteria of the LESS have acceptable reliability with raters using the standardized scoring sheet. Practitioners using only the standardized scoring sheet should feel confident that the LESS is a reliable tool
Comparison of algorithm-based estimates of occupational diesel exhaust exposure to those of multiple independent raters in a population-based case-control study
No full textObjectives: Algorithm-based exposure assessments based on patterns in questionnaire responses and professional judgment can readily apply transparent exposure decision rules to thousands of jobs quickly. However, we need to better understand how algorithms compare to a one-by-one job review by an exposure assessor. We compared algorithm-based estimates of diesel exhaust exposure to those of three independent raters within the New England Bladder Cancer Study, a population-based case-control study, and identified conditions under which disparities occurred in the assessments of the algorithm and the raters.Methods: Occupational diesel exhaust exposure was assessed previously using an algorithm and a single rater for all 14 983 jobs reported by 2631 study participants during personal interviews conducted from 2001 to 2004. Two additional raters independently assessed a random subset of 324 jobs that were selected based on strata defined by the cross-tabulations of the algorithm and the first rater's probability assessments for each job, oversampling their disagreements. The algorithm and each rater assessed the probability, intensity and frequency of occupational diesel exhaust exposure, as well as a confidence rating for each metric. Agreement among the raters, their aggregate rating (average of the three raters' ratings) and the algorithm were evaluated using proportion of agreement, kappa and weighted kappa (κw). Agreement analyses on the subset used inverse probability weighting to extrapolate the subset to estimate agreement for all jobs. Classification and Regression Tree (CART) models were used to identify patterns in questionnaire responses that predicted disparities in exposure status (i.e., unexposed versus exposed) between the first rater and the algorithm-based estimates.Results: For the probability, intensity and frequency exposure metrics, moderate to moderately high agreement was observed among raters (κw = 0.50-0.76) and between the algorithm and the individual raters (κw = 0.58-0.81). For these metrics, the algorithm estimates had consistently higher agreement with the aggregate rating (κw = 0.82) than with the individual raters. For all metrics, the agreement between the algorithm and the aggregate ratings was highest for the unexposed category (90-93%) and was poor to moderate for the exposed categories (9-64%). Lower agreement was observed for jobs with a start year <1965 versus ≥1965. For the confidence metrics, the agreement was poor to moderate among raters (κw = 0.17-0.45) and between the algorithm and the individual raters (κw = 0.24-0.61). CART models identified patterns in the questionnaire responses that predicted a fair-to-moderate (33-89%) proportion of the disagreements between the raters' and the algorithm estimates.Discussion: The agreement between any two raters was similar to the agreement between an algorithm-based approach and individual raters, providing additional support for using the more efficient and transparent algorithm-based approach. CART models identified some patterns in disagreements between the first rater and the algorithm. Given the absence of a gold standard for estimating exposure, these patterns can be reviewed by a team of exposure assessors to determine whether the algorithm should be revised for future studies. © 2012 The Author
Online versus in-person comparison of Microscale Audit of Pedestrian Streetscapes (MAPS) assessments: reliability of alternate methods
No full textabstract: Background
An online version of the Microscale Audit of Pedestrian Streetscapes (Abbreviated) tool was adapted to virtually audit built environment features supportive of physical activity. The current study assessed inter-rater reliability of MAPS Online between in-person raters and online raters unfamiliar with the regions.
Methods
In-person and online audits were conducted for a total of 120 quarter-mile routes (60 per site) in Phoenix, AZ and San Diego, CA. Routes in each city included 40 residential origins stratified by walkability and SES, and 20 commercial centers. In-person audits were conducted by raters residing in their region. Online audits were conducted by raters in the alternate location using Google Maps (Aerial and Street View) images. The MAPS Abbreviated Online tool consisted of four sections: overall route, street segments, crossings and cul-de-sacs. Items within each section were grouped into subscales, and inter-rater reliability (ICCs) was assessed for subscales at multiple levels of aggregation.
Results
Online and in-person audits showed excellent agreement for overall positive microscale (ICC = 0.86, 95% CI [0.80, 0.90]) and grand scores (ICC = 0.93, 95% CI [0.89, 0.95]). Substantial to near-perfect agreement was found for 21 of 30 (70%) subscales, valence, and subsection scores, with ICCs ranging from 0.62, 95% CI [0.50, 0.72] to 0.95, 95% CI [0.93, 0.97]. Lowest agreement was found for the aesthetics and social characteristics scores, with ICCs ranging from 0.07, 95% CI [−0.12, 0.24] to 0.27, 95% CI [0.10, 0.43].
Conclusions
Results support use of the MAPS Abbreviated Online tool to reliably assess microscale neighborhood features that support physical activity and may be used by raters residing in different geographic regions and unfamiliar with the audit areas.The electronic version of this article is the complete one and can be found online at: https://ij-healthgeographics.biomedcentral.com/articles/10.1186/s12942-017-0101-
Avaliações em larga escala com itens de respostas construídas no contexto do modelo multifacetas de Rasch
No full textTese (doutorado) - Universidade Federal de Santa Catarina, Centro Tecnológico, Programa de Pós-Graduação em Engenharia de Produção, Florianópolis, 2015.Esta tese apresenta um estudo sobre as avaliações com itens de respostas construídas em larga escala no contexto do modelo multifacetas de Rasch (LINACRE, 1989 apud LINACRE, 1994). Essas avaliações necessitam de avaliadores para julgar o desempenho das pessoas quanto à habilidade que está sendo medida por meio do teste. Entre as avaliações com itens de respostas construídas mais utilizadas no âmbito educacional e de seleção estão as provas das diversas disciplinas do Ensino Médio, as provas de redação do ENEM e dos concursos vestibulares e as provas com itens abertos de concursos para provimento de vagas de trabalho. Os resultados das avaliações com itens de respostas construídas não dependem apenas do nível de habilidade dos examinandos quanto ao construto avaliado e da dificuldade das tarefas, dependem também da severidade dos avaliadores que julgam os desempenhos e da estrutura da escala de classificação. Um dos principais problemas nessas avaliações é a pontuação de um mesmo desempenho com graus diferentes de severidade. Quando existem vários avaliadores, o ideal é que todos atribuam exatamente a mesma pontuação para os mesmos desempenhos observados, essa é a condição principal para se ter confiabilidade de pontuação. Entretanto, são muitos os fatores que podem causar variabilidade nessas pontuações. O modelo multifacetas de Rasch vem sendo cada vez mais utilizado para aferir a qualidade das avaliações com itens de respostas construídas, por permitir a inclusão de outras variáveis aos sistemas avaliativos, além da capacidade dos indivíduos e da dificuldade das tarefas. Algumas dessas variáveis consistem em importantes fontes geradoras de vieses nos processos avaliativos. Como exemplos têm-se as características pessoais dos avaliadores, as diferenças entre a severidade dos avaliadores, as tendências dos avaliadores em julgamentos sistemáticos, as diferenças entre as dificuldades de tarefas distintas e a variação quanto ao entendimento e utilização das categorias da escala de classificação por parte dos avaliadores. O modelo multifacetas de Rasch permite a inclusão de cada variável que pode interferir na avaliação, além de possibilitar análises para os efeitos causados por cada elemento que faz parte da avaliação individualmente, o que torna a utilização desse modelo muito vantajosa. O objetivo deste estudo é estabelecer como o modelo multifacetas de Rasch pode contribuir para a determinação da qualidade das avaliações com itens de respostas construídas. A abordagem utilizada pelo modelo multifacetas de Rasch proporciona análises sobre a qualidade das medidas relacionadas aos examinandos, aos avaliadores, às tarefas, aos itens e às escalas de classificação utilizadas para a pontuação das tarefas. Este trabalho também apresenta uma aplicação do modelo multifacetas de Rasch aos dados provenientes de uma avaliação real, na qual estabelece as principais análises sobre a qualidade dessa avaliação.Abstract : This thesis presents a study about the large-scale construct-response item evaluations in the context of the many-facet Rasch model (LINACRE, 1989 apud LINACRE 1994). These evaluations require raters in order to judge the performance of the people regarding the ability that is being measured through test. Among the evaluations with constructed-responses items most frequently used in the educational and hiring ambit are those with open questions of the disciplines of the High School, the writing test of the Brazilian High School National Exam and of the university entrance exams and the tests with open questions of contests. The results of the construct-response item evaluations do not depend only on the ability level of the examinants regarding the evaluated construct and the difficulty of the tasks; they depend also on the severity of the raters that judge the performance and the structure of the classification scale. One of the main problems of these evaluations is the rating of a same performance with different severity degrees. When there are many raters, it would be the ideal if all would give exactly the same rating for the same performances observed, this is the main condition in order to have reliability of rating. However, many are the factors that can cause variability in these ratings. The many-facet Rach model have been even more used to check the quality of the construct-response item evaluations, since it allows the inclusion of other variables to the evaluating systems, besides the capabilities of the individuals and the difficulty of the tasks. Some of these variables consists of important sources generator of biases in the evaluating processes. As examples are the personal characteristics of the raters, the differences between the severity of the raters, the tendencies of the raters in systematic judgements, the differences between the difficulties of the distinct tasks and the variation regarding the understanding and use of the categories of the classification scale by the raters. The many-facet Rach model allows the inclusion of each variable that can interfere in the evaluation besides allowing analyzes for the effects caused by each element that is individually part of the evaluation, which makes the use of the many-facet Rach model very advantageous. The objective of this study is to establish how the many-facet Rach model can contribute to the determination of the quality of the evaluations with construct-response items. The approach used by the many-facet Rach model provides analyzes on the quality of the measure related to the examinees, to the the raters, to the tasks, to the questions and to the classification scales used for the rating of the tasks. This work also presents an application of multi-faceted Rasch model to data from a real assessment, which establishes the main analyzes of the quality of the evaluation
- …
