Browsing by Subject "Validity"

Now showing 1 - 7 of 7
  • Thumbnail Image
    Item
    Exploring evidence of validity for the construct of work values.
    (2010-08) Leuty, Melanie E.
    Only three vocational theories explicitly address the role of work values in career choice and development (e.g. Super's Life-Span, Life-Space Theory (Super, 1953); The Theory of Work Adjustment (Dawis & Lofquist, 1984); and Duane Brown's Values-Based, Holistic theory (Brown, 1996). However, many researchers have developed measures of work values for specific studies, causing confusion about what domains represent the construct of work values. This confusion makes it difficult to fully understand the range of relevant work values. Furthermore, limited effort has been made to explore the nomological net for the construct of work values or the relation between work values and other constructs such as vocational interests, personality, and personal values. The present study sought to examine the domain of work values, the nomological net for the construct of work values, and evidence of validity for different measures of work values by comparing multiple measures of work values. Using a sample of (N = 374) undergraduate students, results suggested that evidence of convergent and discriminant validity existed for scores on four of the five work values measures examined. Principal Components Analysis identified six components that captured the domains represented within the construct of work values. Further analyses found that work values were related to, but distinct from, interests and personality, but were most similar to personal values. Information from open-ended responses suggested that additional values may be relevant to a work situation for younger workers.
  • Thumbnail Image
    Item
    Human Cognitive Abilities: The Structure and Predictive Power of Group Factors
    (2019-04) Kostal, Jack
    General mental ability is one of the most powerful and venerable individual differences in I-O psychology. This project consists of two studies that provide comprehensive meta-analytic summaries of inter-correlations between cognitive abilities, and cognitive abilities’ validity for predicting a wide range of job performance criteria. The meta-analytic database created to address these questions consists of 2,356 independent samples from 1,030 separate studies (total N = 2,978,554). Results provide support for a newly developed compendium for classifying cognitive tests, which use would reduce idiosyncratic test classifications that are endemic to the I-O literature. Exploratory factor analyses produced solutions similar to the CHC model, albeit with important exceptions around visual processing, long-term retrieval, and quantitative knowledge. Results did not support age differentiation of cognitive abilities. Turning to validity against job performance criteria, this study found somewhat lower validities than previous work by Hunter and Schmidt. Contrary to previous work, no major differences in validity were observed between fluid and crystallized abilities.
  • Thumbnail Image
    Item
    A Mixed Methods Evaluation of the Reliability and Validity of an Entrustable Professional Activities-Based System for Assessing the Clinical Competence of Medical Students
    (2024) Gauer, Jacqueline
    Introduction: The question of how to assess the clinical competence of medical students has posed a challenge throughout the history of the field. Entrustable Professional Activities (EPAs) describe discrete activities that can be assessed in the workplace by a preceptor, who indicates the degree to which they “entrust” the student to perform the activity. Recently, the University of Minnesota Medical School (UMMS) implemented a system based on the Core Entrustable Professional Activities for Entering Residency (Core-EPAs) to assess the clinical skills of third-year medical students during their clinical clerkships. The purpose of this study was to evaluate the validity and reliability of this system, using quantitative data, qualitative data, and the integration of the two. Method: This study employed a two-phase, sequential explanatory mixed methods design to obtain evidence regarding the reliability, predictive validity, construct validity, and face validity of the Core-EPA system. Quantitative EPA assessment data from AY 22-23 were analyzed via interrater agreement analysis, linear regression modeling to predict scores on an Objective Structured Clinical Examination (OSCE), and growth curve modeling. A purposive sample of eight students was selected from the quantitative data to describe their experiences via semi-structured interviews. The qualitative data were analyzed using a Reflexive Thematic Analysis framework. Results: The interrater reliability analysis found that levels of interrater agreement were acceptable given the complexity of the clinical context (47.35% or 69.40% depending on the definition of interrater agreement), but with room for improvement. The linear regression analysis did not find convincing evidence that EPA ratings are predictive of OSCE scores. The growth curve analysis found that growth curves aligned with those expected by learning curve theory. The qualitative analysis generated five themes and two subthemes describing students’ experiences with the Core-EPA system, their perspectives on its validity and reliability including factors that contribute to those dimensions, and comparisons between the Core-EPA system and OSCEs. Discussion: The findings of this study indicate that the Core-EPA system holds promise as a tool for assessing the clinical competence of medical students. Recommendations, limitations of the study, and ideas for next steps are described.
  • Thumbnail Image
    Item
    A Novel Method for Assessing Leg Compartmental Body Composition Using Dual Energy X-Ray Absorptiometry
    (2016-05) Raymond, Christiana
    PURPOSE: Investigate the validity and reliability of a novel lateral dual energy X-ray absorptiometry (DXA) scanning method for total, fat, and lean tissue mass quantification of the anterior and posterior thigh compartments. METHODS: Twenty-one (11 female; X̅age=20.3±1.3 yrs) college athletes participated, with segmentation of anterior/posterior thigh compartments completed via laterally-positioned DXA scans. Three technicians created custom regions of interest (ROIs) using bony landmarks for each scan via enCoreTM software. Paired t-tests (lateral vs. standard frontal position) evaluated the validity of this novel method while intra-class correlations (ICCs) and coefficients of variation (CV) examined intra-/inter-rater reliability. RESULTS: Total, fat, and lean mass comparisons between frontal and lateral DXA scans were non-significant (p-values: 0.15-0.74). High ICCs were observed between-/within-raters (0.983-0.999 and 0.954-0.999, respectively), with low variation across all measures (CVs: <5%). CONCLUSION: DXA measures using lateral positioning and custom ROIs for tissue mass quantification are valid and reliable versus standard frontal positioning.
  • Thumbnail Image
    Item
    Relation between 1- minute CBM reading aloud measure and reading comprehension tests: a multilevel meta-analysis.
    (2008-12) Yeo, Seungsoo
    The primary purpose of this research was to conduct a quantitative review of studies reporting information on the relation between CBM reading aloud and reading comprehension tests. Furthermore, this study examined moderators affecting the variability in the relation between CBM reading aloud and reading comprehension tests. After computer-based information searches, an extensive hand search, an ancestral search were conducted, and after exclusion criteria were applied, the present meta-analysis contained a total of 250 correlations from 55 studies. This study showed that the estimated average correlation between CBM reading aloud and reading comprehension tests was .75 (SE = .02), with a range of .71 to .79. Compared with Cohen's (1992) threshold, the magnitude of the overall estimated average correlation between CBM reading aloud and reading comprehension tests was large. However, this study also showed that there was statistically significant variability between and within the studies used in the meta-analysis. The findings of significant variability in the unconditional model led to a series of conditional level-three meta-analyses that included potential moderators. Proportion of students with disabilities, retelling as a response format, CBM maze as a reading comprehension test, and the time difference between CBM reading aloud and reading comprehension tests were statistically significant components affecting the strength of the relation between CBM and state achievement tests. Educational implications for teachers and researchers, limitation, and directions for future research are presented.
  • Thumbnail Image
    Item
    Relations between CBM (Oral Reading and Maze) and Reading Comprehension on State Achievement Tests: A Meta-Analysis
    (2017-09) Shin, Jaehyun
    The purpose of this study was to examine the validity of two widely used Curriculum-Based Measurement (CBM) in reading – oral reading and maze task – in relation to reading comprehension on state tests using a meta-analysis. A total of 61 studies (132 correlations) were identified across Grades 1 to 10. A random-effects meta-analysis was conducted to estimate the average correlations between the two CBMs and reading comprehension on state tests, and to analyze the effects of potential moderating variables (characteristics of study, students, CBM, and state tests). Results revealed that the average correlation for oral reading was significantly larger than that for maze when all grade levels were included together in the analysis. When grade levels were separated, the difference between average correlations was only at the higher grades (Grades 4-10), favoring oral reading. In terms of correlations by grade level, oral reading and maze showed a similar pattern; that is, correlations were comparable across elementary grades, but decreased for secondary grades. In addition to the type of CBM and grade level differences, type of publication, development type of state tests (commercial versus state-developed), and time interval between CBM and state tests were significant sources of variance in correlations. Implications for research and educational practice are discussed highlighting the somewhat different conclusions from previous literature, especially regarding the use of CBM for older students.
  • Thumbnail Image
    Item
    The validity of judgment : can the assessor learn to outperform the equation?
    (2010-09) Klieger, David M.
    When judgments (i.e., predictions of outcomes) are incorrect, the negative consequences for individuals, organizations, and society can be serious. For various kinds of outcomes, meta-analyses and literature reviews reveal, time and again, that the predictive validity of information combined in the mind of the assessor ("clinical data combination") is smaller than the predictive validity when the information is combined using an equation or actuarial table ("mechanical data combination"). Therefore, using mechanical approaches instead of clinical ones would seem prudent. However, judgment validity encompasses consequential validity as well as predictive accuracy. Furthermore, even some of the scholars who have emphasized the superior accuracy of mechanical methods admit that it may be possible for a judge to systematically out-predict a mechanical method. One such possible approach is configural reasoning, an assessor's use of a functional form (e.g. an interaction) absent from the mechanical method and yet predictive of the outcome. As indicated by the aforementioned studies indicating the superior accuracy of mechanical combination, judges do not productively employ such techniques in general. Nevertheless, it remains an empirical question whether assessors can be taught to utilize configural reasoning to outperform an equation. In addition, it is important to determine the traits of those individuals who predict and learn to predict most accurately, because identifying such people can minimize the costs of error and training. This dissertation tries to be comprehensive in scope. It employs experimental designs and methods of assessing individual differences to answer questions about the degree (if any) to which people can be taught to outperform a mechanical equation, the degree (if any) to which assessors can learn to improve the accuracy of their judgments, the degree (if any) to which judges can be made less overconfident in their judgment strategies, the relationship of any changes in accuracy to any changes in confidence, the individual differences that define those who predict and learn to predict most accurately, and the timing of and extent to which (if any) assessors gain insight about the most accurate predictive approach. Prior to addressing these issues, this dissertation lays certain groundwork. It clarifies the nomological networks for clinical and mechanical combination. It enumerates much of the vast research that reveals the human cognitive limitations and informational barriers that are thought to contribute to a vicious cycle of lesser clinical accuracy and overconfidence in judgment strategies. Furthermore, it discusses why one should even care about clinical combination if mechanical procedures are generally more accurate. The most extensive background information provided prior to discussion of the studies conducted by this author concerns the Lens Model as a toolkit for measuring accuracy as well as the determinants of accuracy. Although this portion of the dissertation is somewhat detailed and intricate, it is necessary. First, understanding the Lens Model leads to understanding the determinants of judgment accuracy. Second, understanding the Lens Model leads to understanding how the judge can and cannot outperform the mechanical approach. Third, understanding the Lens Model leads to understanding the limitations of prior research. Fourth, understanding the Lens Model is essential if the reader is to fully understand results, discussion, and conclusions of the author's experiments. Also reviewed are the "skill score" as an alternative to the Lens Model for measuring accuracy as well as the major considerations involved when teaching people to improve their accuracy and lessen in confidence. The "skill score" provides information about elevation and scatter that is not available from the Lens Model. Final preliminaries focus on experimental design, namely how and why use of a disordinal interaction is central to the experiments conducted by the author, as well as issues concerning the number of experimental cues (predictors) employed, cue redundancy (intercorrelation), the importance of representative design in the experiments, the conduciveness of various types of experimental feedback to learning, and the impact of incentives on judgment accuracy in the experiments. The author conducted two studies - one in Fall 2009 and another in Spring 2010. Although some of the experimental design details of the studies differed in important ways, their general blueprints were quite similar. Using mostly undergraduate subjects at the University of Minnesota, both studies collected information about individual differences (cognitive ability, gender, personality, interests, and experience). In the experimental portions of the studies, subjects were asked to make predictions of job performance for hypothetical job candidates based on the cognitive ability test score for each candidate as well as how interesting or boring the candidate was expected to find the job. The most accurate clinical prediction strategy would involve applying knowledge that the correlation between cognitive ability and job performance was positive when the applicant was expected to find the job interesting but negative when the applicant was expected to find the job boring (i.e. a disordinal interaction). The competing mechanical model was a linear version of a model that incorporated the disordinal interaction. Subjects were asked about their confidence in how accurately they were making predictions, and in order to assess insight, subjects were asked to narratively self-report the nature of their judgment strategies. Data were analyzed using longitudinal hierarchical linear modeling (for within-person change over time in accuracy, the determinants of accuracy, and confidence), correlation (for between-person differences), and frequencies (mainly for evaluating insight). Results were fascinating, although many were inconclusive (often due to lack of statistical significance). Although subjects could outperform the mechanical model under certain experimental conditions, this superiority was not statistically significant. Some of the individuals, experimental groups, and/or subject pool means increased or declined in accuracy, the determinants of accuracy, and confidence over time as expected, but often these results were not statistically significant. Nevertheless, there was some evidence that criterion-related feedback about the disordinal interaction led to improved accuracy and decreased confidence while lack of it had the opposite effects. Several individual differences were significantly associated with accuracy, with cognitive ability being the difference most pervasively related to accuracy to a statistically significant degree. Findings for insight were complicated by the inconsistent nature of subjects' narratives. Nevertheless, there was relatively high agreement between raters of subjects' insight, and ratings of insight often had statistically significant correlations with objective measures of accuracy. Moreover, insight as variously measured was often achieved, and if achieved was usually achieved early.