Browsing by Subject "Software testing"

Now showing 1 - 5 of 5
  • Thumbnail Image
    Item
    Manifold-based Testing of Machine Learning Systems
    (2022-08) Byun, Tae Joon
    With the remarkable advancement of deep learning in many domains, such as in computer vision, learning-enabled systems are rapidly being adopted in safety-critical domains where it is crucial to verify and validate the system rigorously. However, due to the unique characteristics of the learning-enabled components compared to traditional systems, existing verification techniques do not work in many cases, which calls for new approaches to address this problem. In the literature, we identified that a practical and scalable testing technique is lacking for computer-vision deep neural networks (DNNs) that deal with high-dimensional and unstructured input data. Moreover, most of the existing approaches for addressing this problem are white-box solutions that are dependent on the DNN under test, solutions that may be inappropriate given the highly iterative model development workflow. To address this problem, we propose systematic testing techniques for DNNs that resolve the dependency on the model under test, since the dependency comes with several critical shortcomings. In doing so, we investigated the following three concrete ideas. First, we propose a test prioritization technique that can identify failure-revealing test inputs to help reduce the test construction cost. Second, we propose a DNN-independent test adequacy measurement technique that can measure the adequacy of testing, and also help construct a representative test suite. Third, we propose a DNN-independent test case generation technique that can synthesize realistic test cases that are effective at finding failures in the DNN under test. The last two approaches are black-box solutions in that the test adequacy measurement and the test case generation are performed independently of the DNN under test, a unique direction compared to existing approaches. The experiments showed that (1) test prioritization can effectively prioritize failure-revealing test cases, (2) the black-box coverage criterion can help construct representative test cases that achieve effectiveness comparable to those constructed with white-box criteria, but with much lower measurement cost, and (3) the black-box test generation can synthesize realistic test cases that are also effective at finding failures in the model under test. We believe that the black-box approaches bring complementary benefits to white-box approaches and that they deserve further investigation.
  • Thumbnail Image
    Item
    Oral history interview with Gayle Spiess
    (Charles Babbage Institute, 2015-12-14) Spiess, Gayle
    Gayle Spiess grew up in Minneapolis and attended Valparaiso University (in Indiana), graduating in 1977 with a Bachelor of Science in mathematics. She had worked for Sperry Univac one summer during college, then after graduating took a full-time professional position at Plant 8 (Eagan, MN) doing programming for a Navy ship project. She notes support from her direct supervisor as well as self-study in 16-bit assembler code, which she used for more than a dozen years. Her working group was stable for 3-4 years, even when she physically worked on a top secret project in Building 6 near the original Engineering Research Associates (ERA) plant in St. Paul. Back in Eagan, she worked on a Navy communication system (NAVMACS) and assisted with warship installations in Virginia, Japan, and Australia. Later she did programming with the high-level language ADA as well as C, which became the dominant programming language. A major responsibility was software for the air traffic control (ATC) group from 1993 to 2002 (eventually part of Lockheed Martin), then first-line management and project engineering for ATC (2002-7). She discusses recruitment and characteristics of successful project teams and managerial strategies for them. She also relates observations about changes in corporate culture with the Unisys merger, Loral acquisition, and Lockheed Martin purchase. This material is based on work funded by the Alfred P. Sloan Foundation award B2014-07 “Tripling Women’s Participation in Computing (1965-1985).”
  • Thumbnail Image
    Item
    Oral history interview with Joyce Malleck
    (Charles Babbage Institute, 2008-08-01) Malleck, Joyce
    Joyce Malleck graduated from Mundelein College with a major in math and a minor in physics, and then received a master’s degree in math from the University of Wisconsin–Madison. She accepted a job at Western Electric and went to work at the Bell Labs facility in Naperville, Illinois. She did programming courses at the corporate training center in Princeton NJ, learning COBOL, assembler, PL/1, and a proprietary Bell database management language. (She later did a MBA at the University of Chicago, completed in 1980.) An early assignment was programming to direct an automatic wiring machine for the ESS manufacturing. She was promoted to department chief, initially maintaining a data center’s operating system then doing software and database development for the customer side of ESS. In the 1970s she started a software quality department, which involved greater attention to written formal specifications, code reviews, and structured developmental processes — software engineering. Leaving Bell in 1989, she worked for Motorola for ten years as a product manager and consultant to industry. She compares Bell’s and Motorola’s treatment of and attitudes to women, relating insightful personal anecdotes. This material is based on work funded by the Alfred P. Sloan Foundation award B2014-07 “Tripling Women’s Participation in Computing (1965-1985).”
  • Thumbnail Image
    Item
    Oral history interview with Judith Kinsey
    (Charles Babbage Institute, 2015-12-03) Kinsey, Judith
    Judith Kinsey grew up in southern Minnesota and graduated from Wellesley College in 1962. She applied to graduate school at the University of Wisconsin–Madison, but also took the IBM Programmer Aptitude Test (PAT) and received a job offer from the Minneapolis branch office. She received extensive corporate training especially in the first years of her work. As a System Engineer she supported IBM sales in the manufacturing area, working out of the Minneapolis and St. Paul branch offices. With the coming of the System/360 she helped install these at customers’ locations by doing assembly-language and other programming. While raising children she was out of the workforce during 1970-76 then returned to IBM as Staff Programmer at Rochester, Minnesota, and then moved into management in 1980. She describes programming assignments, college recruiting, gender relations, and Rochester’s distinctive work culture. During development of the AS/400, she was Technical Assistant to the Directory of the Programming Lab at Rochester. In 1995 she took a position at IBM corporate (in Somers NY) and experienced the re-engineering of IBM under Louis Gerstner. She adds descriptions of efforts to encourage Girl Scouts in computing. This material is based on work funded by the Alfred P. Sloan Foundation award B2014-07 “Tripling Women’s Participation in Computing (1965-1985).”
  • Thumbnail Image
    Item
    Oral history interview with Patricia B. Myhre
    (Charles Babbage Institute, 2015-11-23) Myhre, Patricia B.
    Patricia Myhre graduated from Creighton University with a mathematics degree, and then went to work for Sperry Rand Univac in St. Paul in 1976. She did software testing for several U.S. Navy programs, starting with destroyer warships for Iran and later the P3 aircraft. Myhre eventually moved from software testing to system testing, involving complex operational interfaces between Univac and other companies’ equipment. The interview discusses work culture and environments in several different Univac office complexes in the Twin Cities metro as well as with the corporate reorganizations (first, the merger with Burroughs and later the purchase by Lockheed Martin). This material is based on work funded by the Alfred P. Sloan Foundation award B2014-07 “Tripling Women’s Participation in Computing (1965-1985).”