Large-Scale Needfinding Methods, Quality Metrics, and Need Prioritization in User-Centered Design

Abstract

This thesis describes front-end, user-centered design methods to generate and prioritize unmet needs of large, diverse groups. Understanding user needs and preferences is increasingly recognized as a critical component of early stage product development. The large-scale needfinding methods in this series of studies attempt to overcome shortcomings with existing methods, particularly in environments with limited user access. The thesis is presented in four main parts, each with differing objectives. Part 1 focuses on need quantity and includes three studies to evaluate three specific types of stimuli to help users describe higher quantities of needs. Part 2 focuses on uniqueness and describes an automated method to effectively process large quantities of content commonly generated in open innovation practices, including the needs-based data produced in Part 1. Part 3 focuses on quality and describes methods to rapidly prioritize a large set of needs to identify a small subset for further consideration. Previous analytic methods have been used for small quantities (often fewer than 75 statements). Part 4 includes a case study relating to a target application area of medical technology. Study participants in part 1 were trained on need statements and then asked to enter as many need statements and optional background stories as possible. One or more stimulus types were presented, including prompts (a type of thought exercise), shared needs, and shared context images. The topics used were general household areas including cooking, cleaning, and trip planning. In part 2, a series of studies explored automated duplication detection using state-of-the art natural language processing (NLP) algorithms. The Semantic Textual Similarity (STS) algorithms had been specifically developed to compare sentence-length text passages and were used to rate the semantic similarity of pairs of text sentences describing unmet needs. Additional participants were recruited in part 3 to rate need statements using an online interface and a simplified quality metric appropriate to initially screen and prioritize lists exceeding 500 statements for a single topic or product area. In part 4, the methods described in parts 1 and 2 were adapted for use as an email accessible web application delivered to a group of professionals in the medical education field. The topic for the study was a needs assessment for a next generation of medical simulation manikins. Across the series of studies, a number of hypotheses relating to need quantity and quality were tested and secondary research questions were explored. The novel methods were demonstrated as effective to rapidly general lists of unmet needs from large groups. A final quantity study collected 1735 needs statements and 1246 stories from 402 individuals in 24 hours. The Part 1 (Quantity) results show that users can articulate a large number of needs unaided, and users consistently increased need quantity after viewing a stimulus. Part 2 (Uniqueness) results identify top modern STS algorithms for needfinding. These predicted similarity with Pearson correlations of up to .85 when trained using need-based training data. Part 3 (Quality) results and individual hypothesis tests provide additional key contributions. Increasing the number of participants contributing needs can increase the quantity of unique needs as well as the number of high-quality needs. Increasing the number of needs contributed per person increases the number of high-quality needs. Increasing levels of self-rated expertise will not significantly increase the number of high-quality needs per person. Needs submitted first are not lower quality than needs submitted after a sustained period of time. Part 4 demonstrates feasibility of applying online needfinding methods to professional users and suggests that these methods can result in a set of overlapping needs compared to focus group data and can also identify unique needs not identified in focus groups. The results contribute baseline studies to describe a systematic quantity focus as applied to finding needs and demonstrate how users can articulate quality needs given appropriate training and tools. Quality study results provide evidence of a benefit to balancing widespread short user interactions with longer, in-depth interactions. If the objective of a user research effort is to maximize the number of high-quality needs identified, the results in aggregate support the use of multiple approaches including 1) increase the user group size, 2) increase the quantity of needs suggested per person, 3) increase or maintain a diversity in levels of expertise in the user group.