I’ve read plenty of extensive blog posts about how to evaluate development tools, but I want to take a quick moment to attack a common fallacy in the approach to tool evaluation. Namely, when presented with a wide variety of very different options, some people get lazy and reduce the problem to “apples to apples.”
Superficially, this seems bright. It’s a big problem, and there’s only so much time, so let’s reduce the problem to something manageable. Yeah, well, “apples to apples” is the wrong method of reduction. What you are really doing in this case is saying “let’s spend our time and effort pretending that things are not nearly as complex as they actually are.” Presented with a smartphone, a tablet, and a calculator, who in their right mind would only compare the calculator functionality across all three to determine the winner? Apples to apples, right? This would be comical if it was not true of the evaluation processes some organizations use for static analysis tools. Especially in safety-critical industries, it crosses a line from carelessness into reckless endangerment.
Instead, let me propose a different reduction of your evaluation problem that will save you from arguing over which product chose the best font for the number “2.” Stop evaluating features and start evaluating solutions. State your problem in such a way that it allows (or even forces) creativity on the part of the vendor to get to the end goal. (Do you really care about false positive rates of 7% versus 9%? What if the 7% false tool takes 1 hour per task to resolve whereas the 9% false tool takes 5 minutes per task?) When you take a step back from the marketing and focus on the reason you want a tool — reduce risks, reduce costs, secure revenue, improve productivity, maximize customer satisfaction — you can spot the slippery metrics about product capability and refocus on your driving needs. You are in a better position to capitalize on unexpected opportunities. You can even remember that this is for the long haul, so you would rather work with a company that cares about your success and will challenge you to make the right decisions.
[editors note: there is a good article about this as it specifically applies to static analysis tools on the Parasoft blog.]