For internal builds is it acceptable to have some failing tests?
Posted by Mike Longin on December 19, 2008
So the typical answer to this question is no. Usually with some harshly worded questions such as:
- Why would you have a test in there, if you dont mind if failing?
- Why would you even consider releasing a build with failing code?
But let me add some context. When you have 20+ teams using the build and the test that fails is specific to one team AND the code is for a more “minor” feature. Does that change the equation? Or how about it affects 5 teams, but not the other 25. Is that ok?
I am curious what people think.
To me its worth starting to think of multiple tiers. Tier one tests must pass every time. Tier 2 tests are tests that affect 10% of the teams and while an internal build can be deployed, these teams must code freeze and fix the issue. Tier 3 tests are limited to an individual team and that team would need to freeze and fix but everyone else can continue. I am sure someone will tell me I am wrong, but to me its worth at least thinking about.
