I delivered a webinar yesterday as part of my day job at Parasoft . The topic was “Development Testing – Is It Worth It?”.
I talked about the reasons why Development Testing is useful, how it relates to process, policy, and how you can move from a reactive process of finding bugs to a proactive process of writing code that is resistant to bugs. As the old saying goes, an ounce of prevention is worth a pound of cure, or a week of debugging in this case.
The presentation is general, not designed to push any specific tools, so you should find it helpful. I’ve made both the powerpoint slides and audio available below. You can usually access most of the past webinars and other video content on the Parasoft site.
Take a look for yourself. If you have any comments or suggestions, feel free to mention it in the comments, or email me or reach me on twitter.
Development Testing is a lot like exercising and eating well: pretty much everyone agrees that it’s beneficial and should be done, but few actually achieve it in practice.
A rising number of organizations are flirting with Development Testing by giving developers a static analysis tool and hoping that they use it to prevent defects. This is not unlike packing some raw broccoli and spinach in your son’s lunch box and expecting his health to improve as a result. This approach to Development Testing inevitably fails to deliver the results that organizations have been able to achieve with a comprehensive, policy-driven Development Testing process: reduced risks while cutting development time and costs.
If you can’t bear the business risk associated with defects surfacing in your organization’s software, join our webinar—Development Testing: Is it Worth It?—to learn how to get the maximum risk reduction from your investment in Development Testing. After exploring the dangers of relying on static analysis alone and the top barriers to comprehensive Development Testing, you’ll learn how Parasoft’s comprehensive Development Testing platform can help you:
Consistently apply a broad set of complementary Development Testing practices—static analysis, unit testing, peer code review, coverage analysis, runtime error detection, etc.
Accurately and objectively measure productivity and application quality
Drive the development process in the context of business expectations—for what needs to be developed as well as how it should be developed
Gain real-time visibility into how the software is being developed and whether it is satisfying expectations
I’m kind of surprised, or at least disappointed that we are still talking about SQL injection breaches. About a year ago I wrote about SQL Injection and yet it’s still hitting major web sites. For example Hackmageddon has an interesting chart of cyber attacks in June (not just SQL injection). But for me writing about SQL injection feels a bit like writing about planking, it’s so 2000-and-late.
SQL injection? Not sexy, but it sure is effective.
The latest entry in the SQL Injection Hall of Shame is Yahoo (YHOO). I’m sure you’ve heard by now about the hundreds of thousands of user email addresses and plain-text password that were stolen from Yahoo using SQL injection. There is much to be learned from this event.
SQL injection attacks are completely preventable
Using static analysis flow-analysis tools will not catch all security errors
The only place we should be seeing SQL injection attacks today is in the classroom, as IT professionals are being trained to prevent such attacks
If SQL injection is getting through your system, you have a process/infrastructure problem. Or maybe a testing problem, but really that’s secondary. I’d go so far as to say if any testing or penetration tools find a SQL injection, see answer number one, you have a process/infrastructure problem. Seriously, if you’re running flow analysis for example and it finds a SQL injection error, don’t just fix that one, address the underlying root problem, IE you’re using unvalidated user input, and fix it everywhere, not just the instances that static flow analysis found.
As long as organizations play hunting games to find SQL injection they will continue to be vulnerable and pay the price. If nothing else learn this, SQL injection as well as other input validation errors are totally preventable, if you’re having problems then you’re doing it wrong.
Catching ALL the errors
Really this concept is just an extension of the previous one. Namely, you need to look at the root cause for SQL injection and put in place a development and testing methodology that requires 100% compliance. In this case as in many other security issues, the root cause is the use of user input that hasn’t been validated. You need to code in such a way that you don’t leave any room for unvalidated input to be used, ever.
Now the trendy thing to do these days is use flow-analysis tools to find “bugs” in software. Disclaimer for those who haven’t figure it out yet, my day job is at Parasoft who is a maker of static and flow analysis tools among other things. The problem is that people frequently misunderstand what flow analysiscan do, and then either don’t follow through with what it’s told them, or have a false sense of security that they’ve done all they can.
Both behaviors are incorrect and will lead to vulnerabilities in your software. It’s important to understand that flow analysis isn’t a thorough methodology. It’s somewhat random in that it’s attempting to determine possible paths through your code without human intervention. By it’s nature it will miss things. On the upside it will tell you something you didn’t know about your code without any human effort.
The right response to finding security vulnerabilities in flow analysis isn’t to simply fix the ones found, but rather to ask yourself what the underlying cause is, then create a policy drive programming approach that will negate the possibility of such conditions surviving your development and test efforts.
STOP storing plain text passwords – immediately! If people forget their passwords, do a reset. They don’t need the old one back, they couldn’t remember it anyway. And make sure you’re salting and putting strong encryption on the passwords. Perhaps the biggest mistake that Yahoo made was storing usernames and passwords in plain text.
When doing security, it’s not only safest to deal with the root case, it can save your bacon later on. Always promote best practices and you should be able to limit liability to actual loss at worst.
I’d like to say that hopefully this will be the last time I write about SQL injection, but I’m willing to bet that it’ll happen again to a big company before the end of the year. In fact, I wouldn’t even be surprised if it happened again before the end of this summer. Sometimes we never learn. Feel free to sound off with your observations in the comments section or you can reach my on Twitter, Facebook, Google+, etc.
Enable empirical research based on large test sets
Encourage improvement of tools
Speed adoption of tools by objectively demonstrating their use on real software
I find SATE interesting because it takes a couple of different approaches that are pretty useful to people trying to understand what static analysiscan and cannot due. One approach is to have several full-fledged applications with known bugs, and versions of the application with those bugs fixed. These have the effect of showing what static analysis tools can do in the real world. Unfortunately, they don’t help much when trying to find out what kinds of issues static analysiscan handle overall.
To do that, NIST has developed a test suite that has thousands of test cases with specific issues in them. Part of SATE is running various tools on the test applications and test suites, and then trying to analyze what they can find, how much noise they produce, etc. It’s an interesting exercise. You should check it out.
This year I was privileged give a presentation myself. I wanted to talk about some of the pragmatic aspects of actually trying to use static analysis in the real world. To that end, I created a slide show around the top 10 user mistakes, meaning things that prevent organizations from realizing the value they expected or needed from static analysis. These range from improper configuration to poor policy to dealing with noise.
Take a look for yourself. If you love or hate any of them, let me know. If you have others I missed, feel free to mention it in the comments, or email me or reach me on twitter.
I’ll be speaking this Thursday at the SATE IV software security conference in McLean, VA. This is a free event open to the public and a great chance to learn more about static analysis at a day-long event. You can register at http://samate.nist.gov/SATE4Workshop.html
My talk is title “Top 10 User Mistakes with Static Analysis” and I think you’ll enjoy it. If possible I’ll post the slides here after the conference.