Browsing by Subject "evaluation"

Now showing 1 - 20 of 51
  • Thumbnail Image
    Item
    Analysis And Control Of Temporal Biases In Surgical Skill Evaluation
    (2020-05) Kelly, Jason
    Objectively and accurately assessing the technical skill of a surgeon is critically important. The current gold standard relies on a panel of expert surgeons evaluating surgical video footage using structured survey instruments. This is a prohibitively time-consuming process, thus leaving the majority of procedures unevaluated. Previous methods of evaluating skill remain prone to bias towards a surgeons' speed or task times, fueling the need to investigate the mechanisms underlying human motion in favor of techniques impervious to biases. The research objective of this work is to investigate the effects of time and speed on the relative accuracy of both human and computational methods of measuring surgical technical skill. Human methods consist of both expert and non-expert raters (faculty surgeons and Amazon Mechanical Turk crowd workers respectively). Computational methods consist of both neurophysiologically-derived measures from other disciplines and recent model-free machine learning methods. This research objective is pursued by the following four specific aims: Specific Aim 1: Determine whether surgical motion segments are directly correlated to tangential velocity prediction models, and if the result is impervious to surgeon speed. The objective was to test the null hypothesis that there is no relationship between the minimum jerk trajectory velocity prediction model and increases in technical skills proficiency. Prior work in human reaching suggests that adherence to the minimum jerk model should increase as technical skill increases, for proficiency of reaching motions in stroke rehabilitation. This thesis investigates whether this phenomenon holds true in surgical technical skill during simulated dry lab tasks. Specific Aim 2: Implement a classification algorithm which uses recorded data to classify surgical skill in a manner which is impervious to task time. It was hypothesized that recent machine learning algorithms which exploit temporal duration, used on kinematic data from dry lab laparoscopic training tasks, would increase the performance of the current state-of-the-art computational methods of classifying surgical skill. HASH(0x3de9178) Specific Aim 3: Determine how human ratings of surgical tasks are affected by video playback speed and duration. It was hypothesized that the perceived skill of a surgeon followed a unimodal function, in which human raters would experience an increase in perceived surgical skill as the speed of a surgical task video approaches the function's maximum, then immediately decreasing once being aware of the video's playback manipulation. Specific Aim 4: Measure the effect that pre-operative warm-up using validated virtual-reality simulator tasks has on practicing surgeons in real robotic surgeries as measured by the most accurate, least-biased methods detailed in the previous specific aims and prior art. This tested the hypothesis that pre-operative warm-up results in a measurable improvement in surgical technical skill among practicing surgeons (no novices) using surgical robots from live patients. This research concluded firstly that neurophsyiologically-derived models of skill, specifically the minimum jerk model, do not necessarily extend to surgical settings. Surprisingly this research found the opposite, that surgeon experts exhibit movements which deviate further from the minimum jerk model. Second, a classification algorithm was created, using a bidirectional long short-term memory network which controls for task time, and is capable of classifying experts and novices with over 95% accuracy for tasks most resembling real surgery. This research brought about questions of label noise and accuracy, and emphasizes the importance of properly labeled data for machine learning algorithms. It was found that humans appear to have a speed bias in rating surgeons both for laparoscopic surgical training tasks as well as real robotic surgery procedures. Unexpectedly, this effect was more substantial for more expert performances and negligible for novice performers. Counter to our original hypothesis, expectations derived from biological motion models -- that skill discrimination capability would unimodally decrease when video playback was obviously artificially sped up -- were not met. Observer ability to discriminate skill continues well after people are cognizant of a video being played at quicker speeds and no discernible difference between biological-motion-relevant question groups (e.g. motion fluidity) and other questions appeared. Finally, a new dataset of robotic surgeries was introduced, with 343 videos of robotic surgeries including tooltip kinematic data. Evidence obtained from motion metrics, crowd ratings, and faculty surgeon ratings suggest that no measurable warm-up effect was present in our population of 41 practicing surgeons; no evidence supported the use of warm-up.
  • Thumbnail Image
    Item
    Application of Multi-Attribute Utility Analysis to Values-Based Prioritization of Academic-Clinical Partnerships in Physical Therapist Education
    (2022-06) North, Sara
    Strong partnerships between academic health professions programs and clinical healthcare practice settings, termed academic-clinical partnerships, are essential in providing quality full-time clinical training experiences to health professions learners. However, the precise conditions and factors that promote strong academic-clinical partnerships are unknown. There is a need to investigate the application of systematic, scholarly methods to the problem of understanding and weighting clinical education program values and priorities regarding academic-clinical partnerships. The purpose of this study was to examine the extent to which one or more existing, evidence-based evaluative frameworks may be used to identify and quantify prioritized stakeholder values in order to render judgment in the evaluation of academic-clinical partnerships. Multi-Attribute Utility Analysis (MAUA) was identified as a decision-making framework that facilitates comparative analysis of multiple alternatives with unique complex attributes, offering a methodological process for identification and quantification of relevant options. In this study, Directors and Co-Directors of Clinical Education (DCEs) in U.S. accredited academic Doctor of Physical Therapist programs were invited to complete a survey regarding perceptions of academic-clinical partnership attribute priorities and relative importance weights in the context of their institution. Analysis of survey results offered insights into partnership preferences, differences between long-term vs. newer DCEs, and a disconnect between philosophical vs. practical priorities. A simulated application of this proposed framework was performed for two example academic programs and three example clinical partners. Findings demonstrated objective differentiation between partnerships dependent upon academic program priorities. Results indicate that MAUA served as an effective methodology to address gaps in partnership evaluation. Overall, the products of the study analysis and applied simulation offer a methodology and flexible framework that may be utilized by academic clinical education faculty to perform a values-based partnership evaluation and make decisions within the context of their institution.
  • Thumbnail Image
    Item
    Assessing Food Shopping and Preparation as a Mediating Factor Associated with Healthy Outcomes
    (University of Minnesota Extension, 2013-10) Lovett, Kathleen; Hurtado, G Ali; Bain, Jamie; Sherman, Shelley; Katras, Mary Jo
    To understand the relationship of food shopping and preparation of USDA Supplemental Nutrition Assistance Program Education (SNAP-Ed) key nutrition messages have on other key nutrition messages, a quasi-experimental design was developed using data from Minnesota’s 2012 SNAP-Ed evaluation system. SNAP-Ed participants attended a series of nutrition education classes delivered by Extension staff along with schools and community-based organizations and completed a retrospective evaluation survey. Correlations were used to assess self-reported changes for the food shopping and preparation key message. Results found that food shopping and preparation key message was significantly correlated with other key messages and mediation association of food shopping and preparation with low-fat calcium-rich foods, limiting added sugar, food safety and increasing physical activity were also significant. Food shopping and preparation key message is relevant when discussing other nutrition concepts and in future program development and research.
  • Thumbnail Image
    Item
    Best Practices for Field Days: 2008 Children’s Water Festival Evaluation: Presentation Skills for 29 Learning Stations
    (St. Paul, MN: University of Minnesota Extension Service, 2009) Carlson, Stephan; Wang, Hui-Hui
    Twenty-nine stations were observed by sixteen observers. Because the research studies could not control how many times that a station was observed, some of the stations were observed only once, while other stations were observed more than once. The station, “Water! Science Museum” was observed the most frequently, a total of sixteen times by sixteen different observers. The stations observed once by the sixteen observers are: “Well,Well,Well”, “DisappearingWaterfall Mystery”, “Streams Creatures”, “Lakes & Rivers & Oceans-Ohmy”, “BackyardWater Recycling”, and “Groundwater on the Move.” The following stations were not observed at all: “Water!Water! From the River to the River”and“Water Arcade.”
  • Thumbnail Image
    Item
    Best Practices for Field Days: Modified Delphi used for Observation Tool Development
    (St. Paul, MN: University of Minnesota Extension Service, 2009) Heimlich, Joe; Carlson, Stephan; Tanner, Dawn; Storksdieck, Martin
    A team of 40 people from across the country were invited to the Best Practices for Field Days (BPFD) Delphi panel to develop an effective observation instrument for determining the quality of field day components that represent best practices. Thirty nine people accepted the invitation and 27 people participated.
  • Thumbnail Image
    Item
  • Thumbnail Image
    Item
    Carver County Water Management Organization Aquatic Invasive Species Program Description and Evaluation Plan
    (Resilient Communities Project (RCP), University of Minnesota, 2015) Errecaborde, Kaylee Myhre; Loots, Natalie; Mino, Rebecca; Villagra Mostaceros, Diego
    This project was completed as part of the 2015-2016 Resilient Communities Project (rcp.umn.edu) partnership with Carver County. To combat the spread of aquatic invasive species (AIS), the Carver County Water Management Organization developed a program focused on inspection, monitoring, and education. The goal of this project was to develop evaluation approaches and methodologies that could be used to assess the success of the program. Carver County project lead Madeline Seveland collaborated with students in PA 5311: Program Evaluation to develop an evaluation plan, survey, focus group questions, and a program logic model. The students' final report, poster, and logic model are available.
  • Thumbnail Image
    Item
    City of Minnetonka Home Loan Program Evaluation
    (Resilient Communities Project (RCP), University of Minnesota, 2013) Moon, Doug
    This project was completed as part of the 2012-2013 Resilient Communities Project (rcp.umn.edu) partnership with the City of Minnetonka. The Minnetonka Home Enhancement Program (MHEP) is a housing rehabilitation program that was established in 2011 to assist low- to medium-income residents to make improvements to their homes. Since the beginning of MHEP, only one housing rehabilitation loan has been executed. Minnetonka project lead and community development supervisor Elise Durbin worked with a Ph.D. student in OLPD 8595: Evaluation Problems, to conduct an evaluation of the program using surveys and phone interviews with MHEP applicants to determine why the program was not being used. Based on this analysis, the student provided recommendations related to program promotion, administration, and logistics. The student's final report is available.
  • Thumbnail Image
    Item
  • Thumbnail Image
    Item
  • Thumbnail Image
    Item
  • Thumbnail Image
    Item
    Competing and contesting constructions of ‘modern’ womanhood: A vertical case study examining the effects of international development discourse on marriage and education in rural Upper Egypt
    (2015-05) Sallam, Mohamed
    In the Middle East and North Africa (MENA) education is widely understood to play a key role in promoting gender equality and economic empowerment. In the MENA region generally, and Egypt in particular, "early-marriage" is implicated as one of the main barriers to educational access for girls living in rural areas. In 2001 inspired by the Egyptian Government's commitment to the principles of the United Nations Girl's Education Initiative (UNGEI), Population Council in Egypt developed Ishraq, a literacy and life-skills program targeting rural and adolescent out-of-school girls in Upper Egypt. This dissertation examines how conceptions of womanhood are framed at varying levels of the international development landscape, and the extent to which they affect and are affected by national policy considerations (represented by the UNGEI and the Ishraq Program) and local understandings around education and marriage in rural Upper Egypt. This research is guided by the assumption that education policy formation is grounded in particular values regarding the role and purpose of education for girls. Through utilizing a vertically-oriented design, this dissertation explores how international and national policy discussions come to shape the construction and implementation of development programs targeting girls at local levels. Emerging from my conversations, interviews, and many observations with former Ishraq participants, program stakeholders, and other young women in rural Upper Egypt - are varied experiences and understandings that participants related regarding what it means to be a "modern" woman in rural Upper Egypt during this current revolutionary moment. What is revealed is an interplay between transnational development discourse and how particular women in rural Upper Egypt women engage in the social contests concerning marriage and education. The experiences and understandings of participants situated at the most local levels suggest a dynamism and complexity around these social contests that is all but left out of the prevailing policy documents, program materials, and among the views of those responsible of the funding and design of the Ishraq program. Moreover, participants experiences with safety and security in rural Upper Egypt during this most recent period of political transition appears to be contributing to the further isolation of rural communities.
  • Thumbnail Image
    Item
    CURA Self-Study Report
    (Minneapolis: Center for Urban and Regional Affairs, University of Minnesota., 2007) Center for Urban and Regional Affairs
  • Thumbnail Image
    Item
    CYFC Scholars Program: Results of Ripple Effect Mapping and Implications for Extension Program Design Practices
    (University of Minnesota Extension, 2013-10) Jordon, Cathy; Chazdon, Scott; Alviz, Kit
    The Children, Youth and Family Consortium (CYFC) leads a multidisciplinary, multi-year, cohort-based, professional development program for five Twin Cities campus faculty and research staff. The program includes two features: 1) an intensive and sustained “Learning Community” to increase capacity of scholars to conduct genuine community-engaged research, explore professional identity as a community-engaged scholar, provide feedback on ideas, discuss interdisciplinary research opportunities, share findings, and build relationships and potential collaborations; and 2) four years of funding ($17k-$25k/year) to support the scholar in proposing, conducting and disseminating a research project to generate new knowledge about CYFC’s focus on the intersection of educational and health disparities and to apply that knowledge to the work of practitioners and policy makers. This poster explores the potential effective ingredients of the Scholars Program, as elucidated through ripple effect mapping, that resulted in individual, institutional and community impact. Implications for Extension education and program design are also highlighted.
  • Thumbnail Image
    Item
    Data for Evaluation of Subcategorization Frames
    (dataset self-published online, 2006) Chesley, Paula; Salmon-Alt, Suzanne
    Data for evaluation of subcategorization frames as detailed in the paper "Automatic extraction of subcategorization frames for French", available at http://pages.cs.brandeis.edu/~marc/misc/proceedings/lrec-2006/pdf/101_pdf.pdf
  • Thumbnail Image
    Item
    DCN Workshop Application Rubric
    (2023) Wright, Sarah
    This rubric is used by the Data Curation Network to evaluate workshop applications. This was initially created by Sarah Wright, Cornell University.
  • Thumbnail Image
    Item
    Development and Validation of a Survey to Measure Perceived Team Communication Skills in Middle and High School STEM Out-of-School Time Programs
    (2017-12) Grack Nelson, Amy
    Twenty-first century skills are vital for preparing youth for careers in science, technology, engineering, and math (STEM) fields. STEM out-of-school time (OST) programs play an important role in helping youth develop these skills, particularly the teamwork skills necessary for the growing collaborative nature of STEM jobs. However, there is a lack of appropriate measures to evaluate this key programmatic outcome in STEM OST settings. This dissertation research addresses the lack of measures through the development of an instrument to assess team communication skills in middle and high school STEM OST programs. The instrument was developed and validity evidence was gathered though a rigorous four-phase process. Phase 1 focused on identifying and operationalizing the teamwork skill area to be measured by the instrument. The skill area of team communication was most common among STEM OST programs and was defined as information exchange, closed-loop communication, and listening. In Phase 2, the survey scenario and items were developed and then reviewed by experts in STEM OST, youth development evaluation, teamwork, and measurement. Phase 3 involved think-aloud interviews and a national pilot test. Revisions to the survey occurred throughout each phase, leading to the final phase: a national field test of the instrument with 959 youth from 40 STEM OST programs across the country. Through confirmatory factor analysis, a five-factor model of team communication skills was found to be a good fit. The model included two factors for closed-loop communication, two factors for information exchange, and a listening factor. Responses for each of the five factors were reliable with coefficient alphas ranging from  = .70 to  = .79. The final instrument is a 28-item scenario-based, self-report measure of middle and high school youths’ perceptions of their team communication skills. The survey instrument and operationalization of team communication skills in STEM OST programs will be valuable for both the evaluation and STEM OST fields.
  • Thumbnail Image
    Item
    English Language Learners and Career and Technical Education: Focus Group Report
    (2021) Chazdon, Scott; Diaz, Alejandra; Spanier, Tobias; Hernandez-Swanson, Jocelyn
  • Thumbnail Image
    Item
    Evaluating Conservation Improvement Programs in North St Paul
    (Resilient Communities Project (RCP), University of Minnesota, 2013) Erdmann, Julie; Guerrero, Humberto; Hanzlik, Timothy; Rodrigo, Sashan
    This project was completed as part of the 2013-2014 Resilient Communities Project (rcp.umn.edu) partnership with the City of North St. Paul. North St. Paul has its own municipal electric utility, and offers energy conservation programs and rebates to customers. Concerned that residents were not taking full advantage of the programs, the City wanted to evaluate current energy conservation efforts. Project lead Brian Frandle partnered with students in OLPD 5501: Principles and Methods of Evaluation, to design an evaluation plan. The students outlined questions, types of data needed, data sources, and data-collection methods. The students' final report, presentation, and handout are available.
  • Thumbnail Image
    Item
    Evaluation of Lifeworks’ Progression Tool
    (2020-05-03) Ambuel, David; Wilson-King, Rachel
    This evaluation seeks to analyze the effectiveness of the progression tool Lifeworks uses to develop person centered curriculum in their Community Skill Building (CSB) groups. Further, this evaluation seeks to provide Lifeworks guidance on how to revise the progression tool to improve the user experience and better meet the needs of program clients and staff. It consists of four interviews with Lifeworks Community Skill Building group facilitators, specialists, and managers at four sites: three in the Twin Cities Metro region and one in Mankato. Though this was a smaller sample size than initially intended due to the COVID-19 pandemic, the evaluators feel that the recommendations outlined in this report are still worthwhile for Lifeworks to pursue, given the results we gathered. These recommendations are aimed at improving both the content and user experience of the tool: 1. Hold bi-monthly meetings with CSB staff members specifically about how they utilize the progression tool. 2. Create standardized policies surrounding the progression tool. 3. Encourage innovation and adaptation when administering the tool.Based on our findings, we believe that these recommendations will allow for Lifeworks’ to continue providing the tool to their CSB participants in a person-centered way, while alleviating some of the administrative burden currently on service specialists and facilitators. By creating a culture around the progression tool that incentivizes idea sharing and innovation, staff members will be able to seek organizational support and will have a more positive experience with the tool overall.