SoylentNews

AnonTechie [soylentnews.org] writes:

In tests, they missed 98 percent of the vulnerabilities in researchers' code.
Researchers at New York University's Tandon School of Engineering in collaboration with the MIT Lincoln Laboratory and Northeastern University decided to find out how much they are missing. LAVA, or Large-Scale Automated Vulnerability Addition, is a technique created by the researchers to test the limits of bug-finding tools in order to help developers improve them. It does that by intentionally adding vulnerabilities to a program’s source code.

LAVA makes targeted edits in real programs’ source code to create hundreds of thousands of unstudied, highly realistic vulnerabilities that span the execution lifetime of a program, are embedded in normal control and data flow, and manifest only for a small fraction of inputs so as to avoid shutting the entire program down.

When tested with existing bug-finding software representing both the "fuzzing" and symbolic-execution approaches commonly used today, just two percent of the bugs created by LAVA were detected.

http://www.pcworld.com/article/3093420/the-truth-about-bug-finders-theyre-essentially-useless.html [pcworld.com]

[Paper]: LAVA: Large-scale Automated Vulnerability Addition [ieee-security.org] [PDF]

Original Submission