Updated NIST software uses combination testing to catch bugs fast and easy

Researchers at the National Institute of Standards and Technology (NIST) have released an updated version of a computer system testing tool that can cut costs by more efficiently finding flaws. A tutorial on using the tool accompanies the new release.

NIST's software for testing computer systems -- ACTS -- takes advantage of research that shows that virtually all software failures appear to be caused by six or fewer interactions.

Catching software "bugs" before a program is released enhances computer security because hackers often exploit these flaws to introduce malware, including viruses, to disrupt or take control of computer systems. But it's difficult.

A widely cited 2002 study prepared for NIST* reported that even though 50 per cent of software development budgets go to testing, flaws in software still cost the U.S. economy $59.5 billion annually.(* Research Triangle Institute, The Economic Impacts of Inadequate Infrastructure for Software Testing, NIST Planning Report 02-3, May 2002.)

Exhaustive checking of all possible combinations of input actions that could cause software failure is not practical, explained NIST's Raghu Kacker, because of the huge number of possibilities, but it's also not necessary. Based on studies of software crashes in applications, including medical devices and Web browsers, NIST's Rick Kuhn and other researchers determined that between 70 and 95 percent of software failures are triggered by only two variables interacting and practically 100 percent of software failures are triggered by no more than six. "Testing every combination up to six variables can be as good as exhaustive testing," said Kacker.

Working with researcher Jeff Yu Lei and his students from the University of Texas at Arlington, NIST designed Advanced Combinatorial Testing System (ACTS), a freely distributed software tool to generate plans for efficiently testing combinations of two to six interacting variables. The method goes beyond the commonly used "pairwise" approach to software testing, which tests combinations of two variables, so it can detect more obscure flaws. (See: 'Combinatorial' Approach Squashes Software Bugs Faster, Cheaper" in NIST Tech Beat, Dec. 12, 2007,