As I talk to people about static analysis, I get a lot of questions and it seems that static analysis means different things to different people. The definitions people use for static analysis can be any or all of the things in this list:
- Peer Review / Manual Code Review / Code Inspection
- Pattern-based code scanners
- Flow-based code scanners
- Metrics-based code scanners
- Compiler / build output
A working definition of static code analysis is the analysis of computer software that is performed without actually executing the software being tested. I’d like to talk briefly about each of these techniques, when/where to use it, and why it’s helpful.
Perhaps the oldest version of static analysis is Metrics-based code scanners where we look at things like complexity or even simply number of lines or methods in a file. Pattern based code scanners are what some think of as the traditional static analysis technique. A more modern offshoot of this is Flow-based code scanners where they look at paths through the code and say “Oh this could happen to you or that could happen to you”. The last is, which most people don’t think about, it output from your compiler or your build process which is a very valuable thing.
Peer Code Review
Let’s start with what we would call peer code review or code review or manual code review or code inspection. The idea is that humans are looking over each other’s shoulders, the idea being to check to see if the code is doing what you are trying to do. And there’s some really cool stuff out there to help you do this more efficiently. What you don’t want code review to be is checking syntax, or in fact anything that could be checked by an automated tool.
What we want to do is in some point, get some other eyeballs into some other’s code beyond the people who write for themselves, so we don’t get some kind of self-sustained mechanism that someone decides to do something in certain way.
Peer Code Review will help you finding problems early and functional problems. Most important part of Peer Code Review is have an access to mentoring process where other people look over your shoulder and can give you feedback like “you know… I would do that differently for this case. Here is better way to do it”.
You learn from code review because you benefit from the experience of others. In addition it helps you learn the code base because you are looking at other pieces of an application rather than just your own.
Pattern based analysis
Pattern based static analysis is finding a specific pattern from your code. This could be a good pattern meaning something you want in you code, or a bad pattern meaning something you don’t want to be in your code (bugs).
For example, I may have a branding issue. I want to make sure when my code prints out copyright statement. In this case the pattern is that certain code exists where I expect it to be, such as the footer of a web page.
Or the pattern might be a bad one, such as code that doesn’t free up resources when it’s done, causing memory leaks.
It may also be formatting issues that curly is here and under bar used there and case sensitive names used here and there. But we have to remember that it’s not just syntax problems but really things could cause a bug, for example when we try to internationalize our product or it may cause performance issue things like that.
Additionally the really cool thing about pattern based static analysis is that it improves the developers themselves. For example, a developer writes code for accessing a database. The code includes a try/catch block but he forgot to free up resources with a finally block.
The pattern based static analysis tool catches it and lets the developer know. After a few times with this warning, the developer will learn from it and start right code with a finally block to avoid the violation (and the nagging from the tool).
In other words, the tool is actually teaching the developer by suggesting a best practice over and over again. Ideally, you encapsulate intelligence of your best developers into pattern based rules.
Pattern based static analysis is not just to check syntax and code formatting. It’s designed to save you time, not to “take time”. Some people say they don’t have enough time to perform static analysis, but generally, you don’t have enough time to skip it.
Flow Analysis
Flow analysis is the idea that instead of looking for a specific pattern in a particular file or class, we are going to look for a pattern based on trying to follow particular path through the application. But rather than run the application, it simulates the application execution.
Flow analysis looks possible paths of the logic and then manipulates the data to see if the bad pattern appears. For instance, it might try to inject bad data to see if it causes a problem, such as a SQL injection.
The paths are hypothetical in that they may or may not actually occur when you use the application, but they are at least possible. The cool thing is that it find real bugs in your application.
One of the things that flow analysis can find is uncaught exceptions. This may always be a problem because sometimes you handle the exception another way. For example, web application servers commonly have a wrapper to catch all exceptions. This is important because system uncaught exception acts basically same way as system exit.
Sometimes you have application stability issues, and very frequently it is related to unhandled exceptions. In such a case, flow analysis is a big help in improving your application.
API misuse is another common source of problems that are handled with flow analysis. Where the API not well understood or poorly documented it can lead to memory leaks or corruption.
With security, it finds the some potential types of problems for you that you can start to work on. It’s a great first pass, but it’s not as powerful as pattern based analysis for preventing issues and for giving thorough coverage, being limited to the hypothetical paths that the testing tool can figure out.
Metrics
Metrics falls into two goals: you want to understand what’s going on in the code and the other is find possible problems. They do this by measuring something in the code. Sometimes when they people talk about metrics, they mean KLOC, cyclomatic complexity, number of methods or classes, things like that.
Metrics can point you to potentially dangerous design issue which is very helpful. They are generally more useful more at the design level than the debugging level.
When tools started doing static analysis about 20 years ago, there were a lot of metrics in place. When people had a bug in field and couldn’t reproduce it they tried to use metrics to suggest where the problem might be, then use a debugger to check the area suggested by the metrics.
The problem is that sometimes it gives you a good idea and sometimes it doesn’t. It really depends on what metrics you are looking for. If you’re using metrics to try and find bugs, it can be difficult and time-consuming. But if you’re trying to use them to understand your application, they actually end up telling you things.
So let’s assume you have a metric that measures the number of lines in files in your application and you start notice giant files. It probably means that design is not as good as it should be, because components should be very discrete and they should have known inputs and they should produce known outputs. When files get large they probably have a lot of complicated logic in the middle of them. Typically it is a good time to look at and refactor and build them down.
Compiler / build output
You should think of compiler warnings as a useful form of static analysis. Internally we set a policy many years ago that our products must compile without compiler warnings. It turns out that many of the compiler warnings are traceable to real problems in the field. At best, they mask real problems buried within your code. If you think that you can ignore compiler warnings, you’re assuming you know as much about the language as a compiler programmer. They put compiler warnings in place because they think about the code in terms of how the language is supposed to be used. If they give you a warning, it means they’re concerned that the code won’t operate properly. It’s best to pay attention to such warnings.
All of these types of static analysis can be valuable in improving your code and your development process, and even your developers as I discussed. I’ll go more in depth on these techniques in future posts.
[Disclaimer]
As a reminder, I work for Parasoft, a company that among other things make static analysis tools. This is however my personal blog, and everything said here is my personal opinion and in no way the view or opinion of Parasoft or possibly anyone else at all.
[/Disclaimer]
Resources
