Browsing by Subject "Assessment"

Now showing 1 - 20 of 39

Acoustic and Perceptual Measurements of 3- and 4-Year Old Children's Productions of Word-Initial /ɹ/ and /w/
(2023-05-18) Ancel, Elizabeth E; Smith, Michael L; Rao, V. N. Vimal; Munson, Benjamin; ancel014@umn.edu; Ancel, Elizabeth E; University of Minnesota Studies in the Applied Sociolinguistics of Speech and Language Lab
This data contains acoustic and perceptual measurements of 3- and 4-year-old children's productions of /ɹ/- and /w/- initial words. Young children's production of the /ɹ/ sound is highly variable and often inaccurate, with [w] as the most common substitution error. One acoustic indicator of the goodness of children’s /ɹ/ productions is the difference between the frequency of the second formant (F2) and third formant (F3), with a smaller F3-F2 difference being associated with a perceptually more adult-like /ɹ/. This data contains both automatically extracted F3-F2 differences for the full set of 117 children, as well as manual measurements of a subset of the children (n=14). Additionally, this data contains untrained listeners' perceptual measurements of the children's productions for comparison.
Addressing the inclusion of English language learners the educational accountability system:lessons learned from peer review.
(2010-01) Christensen, Laurene L.
This study investigated the inclusion of English language learners (ELLs) in state standards and assessments, as measured by comments made by peer reviewers in the federal evaluation of states' standards and assessments. As required by the Elementary and Secondary Education Act (ESEA), reauthorized in 2004 as No Child Left Behind (NCLB), states are required to develop challenging content standards in English language arts and mathematics, and to assess students on these standards. Nationwide, English language learners have been performing below proficient on these state tests. The goal of this study was to determine the elements of the standards and assessment system that peer reviewers thought could be addressed in order to improve the accessibility of these assessments for ELLs.This dissertation employed two research methods: a frequency and distribution analysis of the peer comments relevant to ELLs across the seven Critical Elements of the peer review, and a thematic analysis of the comments, using inductive methods to establish themes. The study found that comments relevant to assessing English language learners were present in all seven critical areas and in all states. All states had two or more critical elements with comments relevant to ELLs. Key themes from the qualitative analysis included accommodations, diverse stakeholders, reporting accuracy, inclusion, reporting for parents, migrant students, comparability of test forms, bias review, DIF analysis, and standards. States have begun to address the inclusion of ELLs in statewide standards and assessments; however, state agency personnel have more work to do in these areas before assessments can be seen to be fully accessible to ELLs.
Assessing Christian Discipleship in Catholic Youth Ministry
(2008-10) Kaster, Jeffrey Joseph
The purpose of this study was to evaluate the intermediate outcomes of a youth ministry curriculum seeking to foster Christian discipleship. The study developed and tested a preliminary Christian Discipleship Scale seeking to operationally define Christian discipleship for assessment purposes. Telephone surveys were conducted with a representative sample (n=76) of young adult Catholics (population=113) who participated in an intensive university/seminary sponsored youth ministry program between 2000 and 2004 as high school youth. Results indicate the curriculum of theological study, justice education, prayer, and vocational discernment was influential in fostering the Christian discipleship practices of theological education, volunteer service, and prayer in these young adults. Further study is warranted on the Christian Discipleship Scale as a tool for assessing Christian discipleship formation in adolescent religious education curricula as results showed it correlated with five variables including frequencies of Mass attendance, frequency of prayer, frequency of reading scripture, involvement in religious groups, and leadership in religious groups.
Assessment Of Cognitive Transfer Outcomes For Students Of Introductory Statistics
(2015-10) Beckman, Matthew
This study chronicles the creation of an assessment tool that quantifies cognitive transfer outcomes for introductory statistics students. Literature suggested that outcomes associated with cognitive transfer are closely aligned with statistical thinking and are indicative of students’ ability to apply learning to novel scenarios beyond the classroom. No assessment tool had been developed and published for the purpose of measuring cognitive transfer outcomes among statistics students. The results of this study suggest that the Introductory Statistics Understanding and Discernment Outcomes (I-STUDIO) assessment tool may effectively serve this purpose. The assessment tool was developed according to a rigorous protocol of expert feedback and iterative piloting. Data were collected and analyzed from a nationwide sample of nearly 2,000 students attending a wide variety of post-secondary institutions, and the I-STUDIO instrument was found to measure both forward-reaching and backward-reaching high road transfer outcomes with good psychometric properties. Data analysis indicated high reliability and diverse validity evidence. This evidence included confirmatory factor analysis models with compelling alignment to the theoretical model and analysis of qualitative themes among expert feedback. Analysis of scoring consistency also showed strong inter-rater agreement. Although the sample size of the scored responses is somewhat small by convention for item response theory, a graded response model generally showed good item functioning. Furthermore, the data suggested that the I-STUDIO assessment estimated student ability with consistent precision across a wide range of above-average and below-average students. Teachers and researchers can use I-STUDIO for comparing outcomes of alternative curricula. Additionally, the I-STUDIO instrument can be used to measure the effect of curriculum changes designed to improve transfer outcomes. Furthermore, the instrument and scoring rubric were designed to accommodate diverse curricula for the purpose of refining course outcomes.
Assessment of Stormwater Best Management Practices
(University of Minnesota, 2008-04) Anderson, James L.; Asleson, Brooke C.; Baker, Lawrence A.; Erickson, Andrew J.; Gulliver, John S.; Hozalski, Raymond M.; Mohseni, Omid; Nieber, John L.; Riter, Trent; Weiss, Peter; Wilson, Bruce N.; Wilson, Matt A.; Gulliver, John S.; Anderson, James L.
An Assessment of the Writing of Undergraduate Computer Science Students
(University of Minnesota, 2002) Nurkkala, Tom; Gini, Maria
Choosing a “Source of Truth”: The Implications of using Self versus Interviewer Ratings of Interviewee Personality as Training Data for Language-Based Personality Assessments
(2022-12) Auer, Elena
Advancement in research and practice in the application of machine learning (ML) and natural language processing (NLP) in psychological measurement has primarily focused on the implementation of new NLP techniques, new data sources (e.g., social media), or cutting-edge ML models. However, research attention, particularly in psychology, has lacked a major focus on the importance of criterion choice when training ML and NLP models. Core to almost all models designed to predict psychological constructs or attributes is the choice of a “source of truth.” Models are typically optimally trained to predict something, meaning the choice of scores the models are attempting to predict (e.g., self-reported personality) is critical to understanding the constructs reflected by the ML or NLP-based measures. The goal of this study was to begin to understand the nuances of selecting a “source of truth” by identifying and exploring the impact of the methodological effects attributable to choosing a “source of truth” when generating language-based personality scores. There were four primary findings that emerged. First, in the context of scoring interview transcripts, there was a clear performance difference between language-based models predicting self-reported scores and interviewer ratings such that language-based models could predict interviewer ratings much better than self-reported ratings of conscientiousness. Second, this is some of the first explicit empirical evidence of the method effects that can occur in the context of language-based scores. Third, there are clear differences between the psychometric properties of language-based self-report and language-based interviewer rating scores and these patterns seemed to be the result of a proxy effect, where the psychometric properties of the language-based ratings mimicked the psychometric properties of the human ratings they were derived from. Fourth, while there was evidence of a proxy effect, language-based scores had slightly different psychometric properties compared to the scores they were trained on, suggesting that it would not be appropriate to fully assume the psychometric properties of language-based assessments based on the ratings the models were trained on. Ultimately, this study is one of the first attempts towards better isolating and understanding the modular effects of language-based assessment methods and future research should continue the application of psychometric theory and research to advances in language-based psychological assessment tools.
Developing staff skills in e-resource troubleshooting: training, assessment, and continuous progress
(ALA Editions, 2018-05) Carter, Sunshine J; Traill, Stacie
Electronic resource troubleshooting is complex, sophisticated work that often falls to a very small number library staff, even in large institutions. Seeing a need to expand the group of staff capable of diagnosing and resolving e-resource access issues, librarians at the University of Minnesota Libraries developed a training program for E-Resource Management staff. The training program comprised a ten-part workshop, a post-workshop troubleshooting project using real-world examples drawn from user activity logs, ongoing meetings for continuous skill development, and assessment of participant knowledge levels at various stages of the training program. As a result, staff participants demonstrated an increased familiarity in troubleshooting skills and knowledge. This chapter describes the planning, design, and implementation of the training program and offers suggestions for how others might create their own training programs.
Development and validation of a physics problem-solving assessment rubric.
(2009-09) Docktor, Jennifer Lynn
Problem solving is a complex process that is important for everyday life and crucial for learning physics. Although there is a great deal of effort to improve student problem solving throughout the educational system, there is no standard way to evaluate written problem solving that is valid, reliable, and easy to use. Most tests of problem solving performance given in the classroom focus on the correctness of the end result or partial results rather than the quality of the procedures and reasoning leading to the result, which gives an inadequate description of a student's skills. A more detailed and meaningful measure is necessary if different curricular materials or pedagogies are to be compared. This measurement tool could also allow instructors to diagnose student difficulties and focus their coaching. It is important that the instrument be applicable to any problem solving format used by a student and to a range of problem types and topics typically used by instructors. Typically complex processes such as problem solving are assessed by using a rubric, which divides a skill into multiple quasi-independent categories and defines criteria to attain a score in each. This dissertation describes the development of a problem solving rubric for the purpose of assessing written solutions to physics problems and presents evidence for the validity, reliability, and utility of score interpretations on the instrument.
The development of the Minnesota Visual Autism Symptom Scale (MN-VASS)
(2010-05) Halpin, Diane Estelle
The development and psychometric characteristics of the Minnesota Visual Autism Symptom Scale (MN-VASS) was described. The relationship between the literature surrounding autism symptoms, the diagnostic criteria for autism, and the resulting content of subscales was presented. Item analyses were conducted using item total correlations. All of the item total correlations were above .30, suggesting adequate item functioning. Two internal consistency estimates of reliability were computed for the MN-VASS: a split-half coefficient expressed as a Spearman-Brown corrected correlation and coefficient alpha. For the split-half coefficient, the scale was split between whole subscales so that the traits of autism measured on the scale would be equally divided between the halves. The value was .89 for split half and .90 for coefficient alpha, suggesting satisfactory reliability. Test Retest reliability was reported as a correlation coefficient of .90 for the entire scale (n=22), with subscale correlations ranging from .63 to .93. Inter rater agreement was between 60% and 87%. Convergent validity was investigated between the Childhood Autism Rating Scale and the MN-VASS. A correlation of .89 was reported between the two scales. Teacher/therapist evaluations of the scale suggest that the MN-VASS is a useful and valid measure of the important and teachable behaviors which can be addressed through a program of instruction for children on the autism spectrum.
Direct vs. Translated Writing: What Students Do and the Strategies They Use
(University of Minnesota, 2000) Cohen, Andrew D
This study explored an alternative approach to short essay writing on language assessment tasks. Thirty-nine intermediate learners of French performed two essay-writing tasks: writing directly into French as well as writing in L1 and then translating into French. Two-thirds of the students did better on the direct writing task across all rating scales; one-third, better on the translated task. While raters found no significant differences in the grammatical scales across the two types of writing, differences did emerge in the scales for expression, transitions, and clauses. Retrospective verbal report data from the students indicated that they were often thinking through English when writing in French, suggesting that the writing tasks were not necessarily distinct in nature. Since the study was intended to simulate writing situations that students encounter in typical classroom assessments, the findings suggest that direct writing may be the most effective choice for some learners when under time pressure.
Driving change: Exploring the adoption of multimodal local traffic impact assessment practices
(Journal of Transport and Land Use, 2021) Combs, Tabitha S.; McDonald, Noreen C.
Local governments are increasingly recognizing the need to improve multimodal mobility. Using interviews with officials in 36 eastern US communities, we explore factors influencing the adoption of new traffic-impact-assessment (TIA) practices that support multimodal objectives. We focus on communities' motivations and supports for multimodal-oriented practices, as well as challenges they face in adopting these practices. By examining both the prevalence of multimodal-supportive TIA practices adopted by our study communities and the motivations for and challenges to adopting these practices, we identify pathways through which transportation researchers can better support communities' efforts to more closely align their TIA practices with desired multimodal-supportive outcomes.
The effects of causal relations and propositional density in Texts on item difficulty in reading comprehension assessment
(2013-04) Swinburne Romine, Russell E.
This research study examines the relationship between reader-level variables and text-level variables in a large-scale assessment of reading comprehension for Grade 10 students in Minnesota, administered in 2006. Six narrative passages and the associated multiple-choice reading comprehension items were examined and coded for variation in propositional density, causal connectedness, cognitive demand, item type, and distracter features. A series of Rasch models were used to calculate individual item difficulty for different groups of students, divided on proficiency level. The relationship between student proficiency, item characteristics and item difficulty was investigated. A significant effect of whether multiple choice items demanded an inference from readers on item difficulty was observed for low-ability readers. Significant effects were also observed for the presence of highly causally connected, true but irrelevant information in item distracters for students of each of four categories of ability level. Results are interpreted in light of a causal model of reading comprehension, the Construction-Integration Model and the Simple View of Reading.
Evaluating Students' Ability to Integrate Written and Visual Communication
(University of Minnesota, 1996) Gersmehl, Philip J.; Lockwood, Catherine M.
We are developing criteria for assessing writing skills of students and the applicability of written assignments in an introductory level geography course (US & Canada). This course usually has an enrollment of 175 to 250 students per quarter. Our proposal has two elements of research: 1) to determine ways to present instructions so that students clearly understand how to meld spatial ideas and graphical methods into their written arguments; and 2) to measure the degree to which students are able to integrate written and graphic text. Graphics are a powerful means of communicating ideas that text alone may not adequately convey. Graphics, in the context of geography, are more than simple illustrations. Geography combines written text with graphic text to explain spatial relationships. One of the most effective ways to portray spatial data is with maps (a graphic language with its own conventional symbols, grammatical rules, and semantic overtones). Several alternative instructions that build upon three previous projects were prepared and tested winter quarter 1993. These course projects are intended to develop an understanding of graphic text, along with the ability to read, analyze, and then explain map patterns through clear, concise written language. The instructions for the first project were a two-page handout. No formal explanation of the instructions was given, but examples and references to project elements were given in several lectures. The second project also included a two-page handout, supplemented by a poster exhibit. The display showed examples of effective integration of text and writing, proper use of color and cartographic techniques, and acceptable ways to calculate and adjust data sets. Students were given a two-page set of written instructions plus a thirty-minute formal classroom explanation for the third project. We developed forms to assess student performance in five specific areas: research, writing style, selection and description of analytical tools, integration of maps and other graphic texts, and bibliographic skills. Because of class size, a set of criteria and standards for uniform grading by teaching assistants was developed and tested. We are now evaluating the effectiveness of these criteria and student response to these projects. Preliminary evaluation of student work and method of instructions suggest that written instructions should be reinforced with some formal classroom explanation. The visual presentation enhanced student performance. Additional graphic displays would benefit students as well as aid the professor and teaching assistants with visual examples of project components. Based on the original objectives and preliminary findings of our research, we expect two outcomes: 1) students will develop a graphic vocabulary and a set of skills that can be used in other courses or applied fields; and 2) students will gain an alternative perspective on writing techniques (i.e. integration of graphic text and written text).
Examining The Relationship Between Statistical Literacy And Statistical Reasoning
(2016-06) Sabbag, Anelise
Statistical literacy and statistical reasoning have been considered by the statistics education community as important learning goals to be developed in introductory statistics courses (Garfield & Ben-Zvi, 2008). Many statistics educators and scholars have tried to define these learning goals (e.g., Gal, 2002; Watson & Callingham, 2003; Garfield, 2002; Garfield & Chance, 2000; Wild & Pfannkuch, 1999). However, there is a lack of agreement regarding these definitions and the relationship between statistical literacy and statistical reasoning. In addition, there are assumptions in the statistics education literature of an overlap between statistical literacy and statistical reasoning (e.g., Rumsey, 2002; Budgett & Pfannkuch, 2007, and delMas, 2002) and of a hierarchy between these learning goals (Garfield & Ben-Zvi, 2008 and delMas, 2002). Empirical evidence is needed to support these assumptions. The purpose of this study was to investigate how statistical literacy and statistical reasoning are related. Specifically, this research aimed to verify if these two learning goals are distinct or if they overlap. The three research questions addressed by this study were (1) what measurement model best represents the construct of statistical literacy and the construct of statistical reasoning given the criteria of fit and parsimony? (2) what measurement model best represents the construct of statistical literacy and the construct of statistical reasoning given the criteria of reliability and distinction? (3) what measurement model is most useful for understanding the constructs of statistical literacy and statistical reasoning? To answer the three research questions, the REALI instrument was developed to concurrently measure statistical literacy and statistical reasoning. This instrument is composed of 40 items with 20 items measuring statistical literacy and 20 item measuring statistical reasoning. The items in this instrument assess eight areas of learning: (1) representations of data, (2) measures of center, (3) measures of variability, (4) study design, (5) hypothesis testing and p-values, (6) confidence intervals, (7) bivariate data, and (8) probability. During the development process, several types of validity evidence were gathered to support the intended inferences and uses of the REALI’s scores (and subscores): expert reviews, response process interviews with students, a pilot test, a field test, reliability, and a psychometric analysis. Data from the field test were analyzed under the classical test theory (CTT) and item response theory frameworks (IRT). Five IRT models were fitted to the data: A Unidimensional Model, three bi-dimensional models (Uncorrelated Model, Correlated Model, and Cross-loading Model), and a bi-factor model (Bi-factor Model A). The main difference between these models was whether or not the model allowed statistical literacy and statistical reasoning dimensions to correlate and if a third dimension (statistical knowledge) was included in the model. These models were compared at the item- and model-level and the best fitting models were used to evaluate the relationships between the statistical literacy subscore and the statistical reasoning subscore. Evidence was found that the statistical literacy and reasoning subscores from the Cross-loading Model could be measured reliably and distinctly. In addition, statistical evidence also supported that reporting both statistical literacy and statistical reasoning subscores provided more distinct information than only reporting a unique score for each student. Such findings bring valuable information to the field of statistics education and can be used to guide instruction in introductory statistics courses. In addition, the REALI instrument is a tool that can be used by researchers and instructors to investigate students’ understanding of statistical concepts or to evaluate new curricula.
A Guide to Neighborhood Assessment: A Student Workbook
(2003) Crump, Jeff; Students in DHA 2453: Housing and Community Development, University of Minnesota
Identifying profiles of reading strengths and weaknesses at the secondary level.
(2012-06) Trentman, Allison M. McCarthy
The purpose of this study was to evaluate the feasibility and potential utility of reading profiles to identify common patterns of reading strengths and weaknesses among students in high school with deficit reading skills. A total of 55 students from three Midwestern high schools were administered a battery of assessments that targeted specific reading skills, as well as a self-report survey that assessed motivation. A cluster analysis revealed that four distinct profiles were present within the sample and that instruction may be differentiated among a subset of the profiles on some subskills. Profile characteristics accurately and adequately represented individual student characteristics, which may indicate that small group interventions could be devised based on the collective group deficits, which places this study within the problem analysis context. The current study extended the link between assessment and intervention for reading at the secondary level and illuminated the need for further research. Implications for the research and practice of school psychologists are discussed.
IT Competence for all: Propel your staff to new heights
(Haworth, 2008) Eels, Linda L.; Jaguszewski, Janice M.
In 2005, the University of Minnesota Libraries charged a task force with the development of a list of core information technology (IT) skills that could be expected of all 300 staff, from technical services to reference services to stacks maintenance. Once this list was developed, the task force designed and administered an online self-assessment survey to identify gaps and patterns in staff computer skills. Both the development of the core competencies and the administration of the assessment are discussed. Also provided are recommendations for next steps, including using assessment reports and data gathered in the process to develop a training and professional development curriculum focused on the specific identified training needs of staff.
Longitudinal Change in Cognition and White Matter Integrity in Young Adult Cannabis Users
(2015-09) Becker, Mary
Cross-sectional research indicates that cannabis use is associated with cognitive and neuroanatomical damage, particularly when used regularly during development. The timing of use-related impacts on cognition and brain structure remains unclear. This dissertation includes two studies to characterize the longitudinal (1) neurocognitive profile and (2) white matter microstructure of young adult cannabis users who initiated use during adolescence. Cannabis users were assessed on a comprehensive neurocognitive battery and Diffusion Tensor Imaging (DTI) protocol at baseline and at a 2-year follow-up. In Study 1, cannabis users had stable deficits in verbal learning and memory as well as planning ability, and a stable relative strength in processing speed at baseline and follow-up. Deficits in spatial working memory and motivated decision-making observed at baseline recovered to control-level performance at follow-up. Heavier and earlier use of cannabis during adolescence was associated with decline in verbal learning and memory performance over time. In Study 2, change in white matter microstructure between time points was observed. Cannabis users exhibited reduced white matter microstructure organization in the central and parietal regions of the superior longitudinal fasciculus, left superior frontal gyrus, corticospinal tract, right anterior thalamic radiation, and in the posterior cingulum; cannabis users demonstrated increased white matter microstructure in the left anterior corpus callosum and left thalamic white matter. The findings suggest that continued heavy cannabis use during adolescence and young adulthood disrupts ongoing development of white matter microstructure. White matter microstructure changes were generally unrelated to cognitive performance, and future research is needed to clarify their functional significance. Potential mechanisms and implications of the findings are discussed.
Making sense of testing: English language learners and statewide assessment
(2008-07) Stone, Karla Rae
The federal No Child Left Behind (NCLB) Act of 2001 has dramatically changed the educational landscape for all students by increasing the number of standardized tests used for accountability purposes. The impact is profound on students for whom English is not their first language, in part because of the sheer number of tests they are required to take. The challenges increase when tests become high stakes for students in the form of high school exit exams, an increasingly prevalent requirement in states, though not required under NCLB. The policies for tests are complex and can easily be viewed as devoid of any attention to the human side of testing--the ground level, nuanced, and sometimes complicated application of policy at the student level. Yet it is at this level where a policy becomes truly validated. The literature on accountability testing rarely considers student perceptions. This research is an attempt to contribute to that void by combining a student voice paradigm with a sense-making framework. Case study methodology was used and incorporated methods that moved from breadth towards depth as the study evolved. In the case of this research, that evolution was from participant observation (Phase I) to document review, inquiry groups and interviews (Phase II). Participants' pre-existing knowledge and experiences with testing are explored alongside the new knowledge gained as participants in this research study. Findings clearly demonstrate that the stakes connected to testing influence the ways in which students make sense of testing, as do the policy signals. Beyond contributing to the fields of educational policy and student voice initiatives, the intent of this research is to spur advocacy on behalf of and in conjunction with English language learners. Whether for systems accountability (the Test of Emerging Academic English), or for individual accountability as a requirement for a diploma in Minnesota (the Basic Skills Tests), the findings reveal both the complexities of and urgency for exploring testing policies from students points of view. Additionally, findings validate the need for students' voices in policy research, and suggest several implications based on reported experiences with statewide testing.

University Digital Conservancy

Browse by Subject

Browsing by Subject "Assessment"