Reliability of Heuristic Evaluation during Usability Analysis

Abstract

Scientific reliability --- the degree to which a research method produced stable & consistent results --- represents a major linchpin of tenable scientific research. A number of different studies support the conclusion that usability testing of software interfaces lacks scientific reliability. The present report augments this conclusion with findings that heuristic evaluation, the second major pillar of usability analysis, also lacks reliability. We conclude that key priority of "usability science" should be to initiate a systematic program of inquiry to investigate the degree to which this term has meaning.