What Went Wrong with Static Analysis?

Let me say up front that I’m a big fan of static code analysis. I’m one of the people that believe that when static analysis is used properly the benefits are tremendous. That being said, I’m constantly surprised at how many people aren’t getting the value they expected, leading me to wonder what went wrong.

There are a few things that seems to cause people to stumble. They range from tools that are too difficult or impractical to use in a real production environment, too many false positives, too much “noise”, no developer buy-in, no management buy-in, and improper expectations.

Difficult / Impractical
I could go on for days talking about the various ways in which a tool can be impractical. Having seen numerous company attempts to adopt new technologies and processes over the years, I can attest to the fact that employees can successfully kill any such projects… and rightly so. Frequently management becomes enamored over the latest idea and tries to implement it without thinking about the real cost and possible disruption to existing business practices. If the new process/tool doesn’t provide more value than the existing process, it will be killed by the users, who at best will perform willful non-compliance, and at worst will actively work to make sure the project/tool/process fails.

Any disruptive changes need to be carefully weighed for value and cost. If it’s believed that they’re worth the disruption, then management must be prepared to fully back them up. This means mandates for compliance, time allowed and scheduled for training, changeover, etc. Without this, most attempt to improve tooling and process company-wide end up being short-lived diversions to the status quo. In the near future I’ll go into more detail on how to successfully select tools and processes.

False positives / noise
False positives and noise are among the biggest reasons why developers start using static analysis and then don’t continue. With false positives there is initially a perception problem. Developers often think of any static analysis message they don’t agree with as a false positive. I’d have to say that a false positive is more properly defined as a message that is actually incorrect. For example, it says I have an unclosed resource such as a JDBC connection, when in reality the connection is closed. The difference between tools in the area of false positives is extremely important. Some tools do a poor job of getting it right, and some rules the some vendors choose to create are simply poor choices. Sometimes there are patterns that static analysis looks for that are particularly prone to false positives – such tools and rules should be avoided like the plague.

Frequently I see developers say things like “I don’t agree with that” or “I don’t understand why it’s saying that” etc. These are not necessarily false positives, but more likely a training or configuration issue. This is why I mention both noise and false positives together, even though they are not necessarily the same thing. In my mind, false positives are one particular kind of noise.

Noise I like to define much more broadly, as anything that a developers doesn’t want to see. This is frequently the claim of false positives, whether correct or not. It can also be a rule problem, in that some rules are very context sensitive and only apply in certain areas. It can be a poorly configured tool, for example one that doesn’t understand the difference between legacy code and current code. It’s not uncommon for teams to have a policy that you don’t touch legacy code simply to fix static analysis issues. If your tool doesn’t accommodate this policy, it’s going to be a problem.

Rule choice is another area that produces unwanted noise. As mentioned above, some rules are poorly implemented in some tools, or even simply a bad idea. This is way more common than it should be. Any rule that produces a constant amount of noise should not be used. By constant amount I don’t mean I’d accept a rule that gave me the “right” answer 51% of the time, I mean one that gives me the right answer 90+% of the time. It’s all too common for people to turn on rules because it sounds like a good idea, but in actual production, the rule requires way to much manual evaluation.

In some sense it’s similar to a spell checker. If you have an untrained spell checker working against a document, it will likely show real spelling errors buried in a mountain of noise based on words unique to your company, industry, or product. If you choose to scan through to manually find the real misspelled words, you’ll very likely miss some. If instead you take the time to add new words to your dictionary, then the spell checker will be right a very high percentage of the time. This means you’re less likely to miss errors, and more likely to use the tool.

Again, tools that are noisy, bad rules, and tools that cannot be configured for real production code should be avoided. It’s extremely unlikely that you can sustain their use in the long-term.

Buy-in seems like such a simple concept but it’s really much more. I’ve worked with organizations that felt like static analysis should be like sugar – the developers will just want to use it. It’s important to understand that static analysis is a quality process which will cause extra work up front for developers. In some cases it may even impact deadlines. The theory is that the up front costs pay off in higher quality, less debugging, shortened testing cycles, etc.

It’s unrealistic to think that you can add a step to your development process that will not have some kind of impact on work effort and schedule. Yes, you should expect a long-term return on investment for the cost and effort of using static analysis, but not on day one, or week one, or even month one.

In order for static analysis to “stick” as a process, you need to make sure that management understands what it’s for, and is willing and able to enforce it’s use. This of course presupposes that the tool is a good one, with low noise and well configured. This minimizes the negative impact and improves developer buy-in. If developers get messages that aren’t useful, they’re likely to ignore the results. If on the other hand static analysis saves their neck by telling them something important, then you’re on the right path.

Proper expectations
So what should you expect from static analysis? Is it a golden bullet? Will simply applying any static analysis tool with any rule set do the job? Is just buying the tool enough – developers will be happy to use it? The answer to all the above is obviously no.

The best possible way to make sure that you get ROI on your static analysis is to tie the configuration to problems you’re actually having. Surprisingly most teams I see are using SA because its the “right thing to do” without any regard to problems they’re actually experiencing.

It is much more effective to do some kind of postmortem on your existing code and projects. Then based on those results configure your rules based on problems you’re actually having. For example, if you have a server that’s crashing or hanging randomly, turning on rules that check for unhandled exceptions and resources that aren’t closed are likely to be effective. Rules that check for the placement of curly brackets are not.

Static analysis can be a huge benefit with minimal impact when deployed properly. Organizations that look at it as a long-term investment in quality and configure it for actual problems are likely to get the best benefit and most likely to see it’s use continue over time. Groups that simply do it because it right, without an understanding of risk, investment, and proper configuration end up losing. At best, they produce unmeasurable results that they cannot justify, at worst they can actually decrease quality by wasting time and focus on unimportant items, while taking resources from real important issues. Make sure this doesn’t happen to you.

As a reminder, I work for Parasoft, a company that among other things make static analysis tools. This is however my personal blog, and everything said here is my personal opinion and in no way the view or opinion of Parasoft or possibly anyone else.


Google and Motorola – What Will Googorola Do?

In case you haven’t followed any tech industry news in the last several days – Google (GOOG) and Motorola Mobility (MMI) have agreed on an acquisition to the tune of $12.5 billion. Some are calling them Motoroogle, but I prefer Googorola, especially since Google is the one doing the buying – their name should come first.

This is a pretty big step for a company like Google that is essentially a software play. Naturally people are asking what’s it all about. In order to figure out what may come out of this we need to try to understand why Google would buy a mobile hardware company. Unfortunately, the truth is there is no way to really know what Google was thinking. There are a couple of prevailing theories.

Patent Protection

One is that Google sees this as a way to protect Android intellectual property against patent suits, especially from Apple (AAPL) and Microsoft (MSFT). Larry Page mentioned this in the Official Google Blog on Monday. Many pundits and analysts have lined up behind this theory. In fact it was mentioned by almost every single android manufacturer.

End-to-end Production

The other popular theory is that Google wants to be able to reproduce the capability of full ecosystem offerings like phones from Apple, Rim, and HP. This gives Google the chance to control the hardware, software, user-interface, and in short the complete user experience. This is a straight-on attempt to duplicate the popularity of the iPhone.


While I have to agree that the patent protection side is at least part of the reasoning, I am doubtful that it’s the only reason. Recall that during the recent hissy fit between Google and Microsoft re the Nortel patents, it came out that Google had the chance to be part of the winning consortium, but choose against it. Google claimed that it was smoke and mirrors on the part of Microsoft, since if it was part of the group it couldn’t use the patents to protect against a lawsuit by the co-owners of the patents. However, it’s undeniable that they also couldn’t be sued by the co-owners of the patents over the same said patents. Clearly they were after something other than just protecting Android.

In addition, Apple has been moving forward in their suit against Motorola. Florian Mueller has discussed this in depth over at Foss Patents. Obviously if Motorola can’t protect themselves from suits by Apple, they can’t protect Google and Android either.

This leads us to the other alternative – Google wants to build phones. This will be clear to all soon enough. Either Google will divest themselves of the phone-building part of Motorola, while keeping control of the patents, or… they will start building phones. If they divest, then what they claimed was all they’re after, and you can quit reading now. But the interesting part is what happens if they do start building phones.

One can’t deny that Google hasn’t gotten a lot of traction with their Nexus phones. On the other hand, the iPhone is the single most-popular smartphone out there. Anyone who wishes to seriously compete with them needs to take a serious look and try to understand how they’re succeeding. Leaving conspiracy theories about fan-boys etc aside, the conventional wisdom is that Apple’s ability to control the software and hardware gives them a serious edge in end-user experience. Let’s presume for a moment that this is at least part of why Google would spend such a premium on Motorola.

The question then is how does it all work? Google can isolate Motorola, run them as an independent company, and treat them on a level playing field with all other phone vendors. This is what Google is claiming so far. I don’t see it happening – if that’s the plan, why spend so much? This would essentially make it a pure patent play, and in that case the price was simply too high. More likely Google will feed Motorola first, possibly even giving them exclusive access to extra functionality. Given Motorola’s past success with popular phones from the RAZR to the DROID, I expect this could be a pretty big hit. It sounds very interesting to me.

End result then is the next step. What does it mean for android and android customers? Well, if you’re a Samsung or HTC phone fan, you might want to start checking out what Windows has to offer, because I expect both of them will be adding Windows to their stable. I don’t expect they’ll drop Android, but I won’t be surprised to see them pay less attention to it.

If you’re a Droid fan or Nexus fan, lookout. This could be a truly amazing set of phones coming from Google and Motorola working together in an unfragmented way. I’d be pretty happy right now if I was a Droid user.

As for Android itself, well it depends on how it all plays out. If Google starts competing with their hardware partners, then they will look for alternatives. Microsoft has a chance here to really make a great competitive offering at a great price. It might even end up being cheaper to license Windows than to pay royalties on Android. This could be the thing that they need to really finally get Windows Mobile off the ground. Android will survive as long as Google needs it and uses it, but it’s day’s of dominance in the mobile space may be numbered. Ironically, in an attempt to compete, Google may be strengthening a major competitor in Microsoft. Only time will tell.

[Update 2011-08-17]
One other possibility is that Googorola is interested in set-top boxes. It makes sense.

[Update 2011-09-29]
The Justice department is is asking for more info about the Motorola deal. This may or may not mean anything, depending on who you ask. Me, I’m not sure, but I don’t think it’s overly serious.

[Update 2011-10-06]
So Intellectual Ventures has filed a patent infringement suit against Motorola Mobility. Apparently Motorola’s patents can’t even defend themselves. So much for the “Google only wanted their patents” theories.

[Update 2016-03-12 – Google has long since spun Motorola back out – so their IP play didn’t really work and they never really leveraged the hardware capabilities of Motorola]

Anonymous Collateral Damage

Once again Anonymous is releasing the personal data of a host of random private individuals, in the name of fighting for freedom. Over the weekend they hacked various Bay Area Rapid Transit (BART) systems in retaliation for BART shutting down cellular services at several stations last Thursday to avoid a potential protest. It seems like the “regular people” are the proxies in this battle. In both cases they were the ones to suffer, both at the hands of BART and Anonymous. In neither case did they have anything to do with the issue.

BART claims they had information related to a potential protest the was being coordinated on Thursday. Further, they claim that a protest creates a safety issue for BART passengers, therefore they are justified in suspending freedom of speech (vis-a-vis cellular service) because they believe safety trumps civil liberties. As you may have guessed, I feel that’s a pretty specious argument. There are various groups looking into a variety of remedies and legal action related to this issue, and the courts are the right place to tackle this problem.

Like petulant children, Anonymous feels that it’s better to work outside the system. Innocent bystanders are of no concern – “If I can’t have it my way, no one gets to play”. In all likelihood the outcome of legal action will result in a change or clarification of policy, and may even cause a few heads to roll at BART, as they should. Hacking on the other hand will do nothing to stem the problem. The funny thing is, it’s easy to sympathize with Anonymous on the surface, since they claim to be taking on the evil government for the little people. Why then are their target consistently users of the organizations they attack, and not the organizations themselves? This is the same tactic as when they went after Sony, but instead released credentials for thousands of customers, cause little problem for Sony, but a huge problem for many innocent bystanders. Some Anonymous members say that it wasn’t them, but LulzSec is a spin-off, so this is like saying the US Army didn’t attack, it was just Delta Force. The two organizations are intertwined.

The truth is if you look back at the attacks, the ones that cause the most problems are consistently related to publishing the private information of individuals. Organizations can easily weather a web outage for a couple of hours, or repair their home page from silly graffiti. But end-users can end up as victims of identity theft, have bank accounts ravaged, and more. Problems difficult to deal with and unrelated to their personal behavior, other than the bad luck to patronize an organization that Anonymous doesn’t like. In the case of Sony, maybe users could choose a different gaming system, but is that really the goal? Maybe Anonymous is pushing X-box? Ridiculous of course. In the case of BART – what choice do commuters have? Does Anonymous prefer they use their cars to go to work, increasing traffic congestion and pollution? If not, why attack people who use BART.

Anonymous talks a good talk but until they start behaving like responsible individuals, it’s difficult to see how the world would be a better place if they got their way.

[Update 2011-08-15]
As expected, the FCC is reviewing the cellular shutdown.

[Update 2011-08-17]
Anonymous has hacked BART police officer accounts once again avoid those really responsible and harming the little guy.

[Update 2011-08-19]
A member of Anonymous has quit the group and says that it’s members are hypocrites. He says “Does anon have the right to remove the anonymity of innocent people?” and “Truth Is Anonymous Hasn’t Brought Down Governments. The People Have.” Worth a read.

[Update 2011-08-20]
A member of anonymous spoke with the Cisco Security blog and says “Getting files and giving them to WikiLeaks, that sort of thing, that does hurt governments. But putting user names and passwords on a pastebin doesn’t [impact governments], and posting the info of the people you fight for is just wrong.

[Update 2011-09-21]
Apparently the FBI have arrested a few more LulzSec and Anonymous members. The release says that one of them is the person responsible for the Sony attack.

Agile 10 years later: Dogma vs Doctrine

It seems like every few years a new software development methodology comes along. Frequently it’s a rehash of an old methodology, like object oriented programming’s rise (again) in the 90’s. Sometimes it’s a formalization of existing ideas and techniques, and sometimes it’s something completely new. Invariably the new toys excite a group of people who will promote them ad nauseam. Many of them become a flavor-of-the-month (or year) and then are never heard of again. Despite that, some of them end up being interesting and manage to survive their 15 minutes of fame.

The funny thing is that you can usually separate such ideas into two key components. One is the actual technology (ideas, tools, techniques, etc) and the other is the philosophy or religion. In other words, the “why” behind the whole thing. I prefer to think of it as religion because it tends to provoke the same kind of arguments and behavior as religious discussions and disagreements.

Agile development is a clear-cut case of such a problem. There are some wonderful, useful ideas embodied in the Agile Manifesto. If you apply them judiciously with an eye toward constant improvement and productivity, and where necessary take into account the evils of financial viability and mandated compliance overhead (such as aerospace, FDA, government, etc), you can start producing better software more quickly, and with less cost.

However when a group of people start doing Agile for Agile’s sake, you can have a disaster on your hands. I have witnessed this personally at some large companies where an Agile team had an interpretation of agile development that was at odds with the company’s needs.

I say interpretation of Agile, because different people have widely different opinions about the degree to which Agile alters their world. In one case a company building very specific hardware/software products has a very strong need to meet particular deadlines, and to deliver very specific features because they release a matched blend of hardware and software. In this company, the Agile team claimed that asking them to definitively schedule feature delivery was a ridiculous request. The team felt their company’s request was “ignorant” and beneath them. Not surprisingly, such an attitude doesn’t work well at companies who simply cannot have an agile release schedule.

For web-based organizations like Google or Amazon, the timing of feature delivery can be more or less whimsical. They can produce what they want, when they want.

On the other hand, companies building smartphones or heart-monitors face a very different reality. There are deadlines, requirements, scheduling, documentation, etc., all of which have to be met and coordinated.

The sad thing is that the dogma fueling the religious war (We can’t be expected to produce a schedule, we can’t write comments, etc) is counter-productive, which leads people to avoid Agile altogether, thereby depriving themselves of potential and probable productivity improvements.

In my experience Agile, like other methodologies before it, works best when followed in a contextual manner, mindful of product needs, organization (corporate) needs, and other issues surrounding the software in question. If you pick and choose the doctrines that can help your team and your product, you’ll be an Agile fan. If on the other hand you dogmatically adhere to “Full Agile,” without regard to the environment you’re in, you can cause project failure, put people out of work, and potentially kill a project or even a small company.

So Agile is 10 years old now. Should you use it? Will it make a difference? In my opinion if you separate the dogma from the doctrine you can definitely benefit. But beware the pitfalls. If you’re careless, you’ll waste time at best, and money, people and perhaps even more at worst. If you’re careful, the benefits from increase quality and productivity can be enormous.


Ranting about Software, Security and Tech