Building An Automated Quality Assurance System with Python
27 Mar 2009, Pycon 2009 USA
Are you leveraging automation in your project's quality assurance process? This talk is about Automated QA systems: a software system that tests everything, all the time. The normal unit, integration, and other automated tests, plus other tools doing things like code coverage and static source code analysis, are used as building blocks. The recommended features and functionality of the QA system are discussed, and an overview is given of how to set up one using the BuildBot framework.
Resources
Given as Building an Automated Quality Assurance System Using Open-Source Python Software at Pycon USA 2009.
About eighty people attended the talk. I asked the audience if they rigorously used version control, and then asked if they wrote lots of unit tests and other automated tests. Almost everyone in the room raised their hand on both counts. (I was proud of them for this, and told them so!) In contrast, when I asked how many had measured code coverage in the past week, fewer than one in ten claimed they had done so.
This demonstrates the value of an automated QA system. If you need to manually execute the code coverage tool, then in practice you just won't do it as often as if it is run for you. If your QA system automatically runs code coverage each night (for example), you and your team are freed up from bothering to do it manually - or even remembering to do so. It's just done silently, and a fresh coverage report is available when you are ready to see it.
This talk referenced the The Buildbot QA/CI Framework. There are many such frameworks with different plusses and minuses. BuildBot's weakness is its brief but steep learning curve, which makes it harder than anyone would like to set up for simple projects. Its plusses are its generality, range, and extensibility: it can be made to do almost anything you need your QA system to do, even for tremendously large projects with complex test metrics. Overall, I recommend BuildBot be used for building your QA framework, unless you have some particular reason to use one of the others that are out there.