Continuous Integration: A Mindset, part II February 16, 2012at20:00
I received a couple of excellent questions/concerns about CI in a comment from Roger that I will try to address.
“How about human testing/QA? If team or feature branches are used, the QA department has a chance testing PBIs isolated before integrating with mainline which, hopefully, leads to more stable mainline.”
Ideally the QA department should not concern itself with testing in the traditional sense. The roll of the QA department, in my point of view, is to support the development teams by improving on the testing tools, maintain resources required to do automatic testing, help to improve strategies, etc. QA should concern itself with making sure that the process works, that the tests get run, that the tests are sound. They should do very little testing themselves.
All tests should be automated. Not only unit tests, but integration tests, load tests and acceptance tests as well.
That was an intentional lie. There are two types of tests humans should do and that can be performed by the QA-department:
- Exploratory testing
- Verification of aesthetics
However, nothing specifies that these activities must be done in isolation. I would prefer that the test-specialists worked in the design-teams and performed the testing together with the design team.
CI requires another thing: Every designer should be able to run most of the tests themselves from their local copy.
If you do work that likely has a big impact on the system, run some smoke-integration-tests first, before you commit to main.
Along-side or prior to a new feature is being developed, automated function tests and acceptance tests should be written. Obviously they will fail until the feature is complete. You don’t want these new tests from halting the build/deployment. So you will need some kind of mechanism that allows the developers to test against these acceptance tests but preventing someone turning the feature on. If your testing framework and CI-engine doesn’t allow you to easily exclude tests that are expected to fail, I could agree to put the test code on a “feature branch”. The reason I would allow this, is because tests, if correctly written are completely isolated and require no integration.
If you have commits that keep breaking main in a show stopping fashion, then you have the wrong set of tests and the wrong mindset. CI requires discipline; more so in the daily work than waterfall. Because you will be called on it every time you slip and break something. In waterfall, you get away with doing the wrong things for much longer, until it comes to crunch time and all the ugliness becomes apparent – or worse, does not show it self until it is running in the customers’ systems, causing havoc.
“Also, if new stuff is too unstable or there has been misunderstanding about functionality, it doesn’t have to be integrated at all.”
If there are uncertainties: discus, do pretotypes (pretotyping.org/), do internal demos. If a commit breaks the build, undo it. Also, new features should have a feature lock preventing them from showing up in the delivered product until the full feature is completed and ready for market. Large features may take months to develop to completion. Should you hold of the integration for that long? Of course not! Even if there are concerns with CI, the benefits exceed the drawbacks many times over.
You will have issues with CI and it will have drawbacks, but the alternative is far worse. So even if you don’t feel that I have addressed the stated concerns in a successful way, I would still argue that these concerns are considerably small compared to the issues that CI do solve.