Deng, Zhihang2020-08-252020-08-252020-03https://hdl.handle.net/11299/215140University of Minnesota Ph.D. dissertation. March 2020. Major: Computer Science. Advisor: Victorial Interrante. 1 computer file (PDF); xi, 155 pages.Traditionally, most Virtual Reality (VR) applications use a generic avatar model as a digital representation of one's own body that can provide a reference as a means of localization and an interactive tool in the immersive virtual environment to users. With the development of depth-sensing technology, RGB-D data can be used to perform 3D reconstruction, thereby providing a personalized self-representation to the user in VR. However, this approach is not without its limitations; these include the following: expensive equipment, setup time, and difficulty providing stimuli that are naturally available in the real-world. The latter limitation, we believe this may diminish the VR experience. One sense that is lost in VR is haptic feedback. While research is ongoing to improve the ability to incorporate this sense in VR, many challenges remain to be addressed to support such functionality robustly. This thesis looks to address some of these limitations. First, we present a video-based approach that can provide a realistic personalized self-representation that allows users to see their own body easily and quickly. After this method was developed, we conducted three user case studies to understand which approach, model-based hand and video-based hand, is inherently more promising in VR. In the first user case study, we used visual/haptic cue conflict paradigm to quantitatively compare participants' inherent sense of trust in the visual representation of their hands when a video-based vs model-based approach was used for the hand representation during passive haptic interactions in a VR setting. This experiment found equivalently stronger visual dominance in each of the hand representation conditions versus a control condition of no visible hands suggesting equivalent potential promise in each of these representational approaches. Furthermore, we also found a surprisingly small range of imperceptibility of visual displacements of the hand locations coincident with bi-lateral haptic feedback. The second experiment focused on the use of model-based hands and assessed the impact of the self-likeness of the model-based self-representation (using one's own scanned hand versus the scanned hand of someone else) on multiple measures of user satisfaction and task performance in three different types of tasks. We found that ratings of perceived visual realism were significantly higher, overall, when participants used their own scanned hand vs. the scanned hand of another participant, but we did not find significant differences in ratings of agency, ownership, presence, or functional realism. In our last experiment, we focused on studying video-based hands and examined the importance of applying realistic shading and shadows to the hand representation when using the RGB-D data approach. We first investigated the influence of perceived-correct vs incorrect hand brightness on the accuracy with which participants could identify hand/surface contact at different hand/surface distances. The results of this study showed decrease accuracy of contact detection both under lower levels of scene illumination and when there was a mismatch between the hand and scene brightness. We then compared contact detection accuracy under three different shadow conditions: when the video-based hand did not cast a shadow, when the 2.5D mesh of the video hand was used to cast a shadow, and when the video hand's shadow was defined by a concurrently tracked (but not visible) generic 3D hand model. Our result found a significant main effect of the distance from the hand to the surface on the likelihood of contact perception, but no significant effect of the shadow type. In summary, this thesis examines the effects of multiple aspects of model-based and video-based hand representations on multiple measures of user satisfaction and performance in various VR tasks. The results that are presented in this thesis have the potential to inform future applications that use a video-based or model-based approach to represent users' hands in VR.enHuman Computer InteractionPersonalized self-avatarUser case studyVirtual RealityThesis or Dissertation