“How many bugs have your unit tests found? And why they didn’t find the one that’s currently killing our production? See? This proves that unit testing doesn’t work. It’s just a waste of money. My money.” said the boss. Of course not my boss.
That’s actually a pretty valid point. How to prove that unit tests that I have written have avoided a lot of problems? Unexistence is hard to see. Management has to be a little bit religious here. Defects found by testers are measurable, because they are officially reported. Everyone can see the issues chart, you hear about them during meetings.
But who has ever reported how many bugs he has avoided thanks to unit tests?
But who has ever reported how many bugs he has avoided thanks to unit tests?
I am not a very religious type. Quite the opposite. That’s why I’m not feeling comfortable when advocating unit tests. I just can’t find any measures, numbers, graphs to show that would clearly visualize the benefits. The more I think of it, the more it gives me the impression that we should start a movement against unit tests.
Let all bugs rise and ruin down the production. We will count them and put them into glass jars with a little help of unit tests. Add salt, oil, sergeant pepper and serve it to the management with colorful defect burndown chart. Their oak tables full of canned bugs are the best evidence they can imagine. When you tell them that it will never happen again if we first write unit tests and then go to production, they will make you the employee of the week. Maybe even of the month.
Don’t worry. They will forget and it will come back again. Decreasing budgets, missed deadlines and always-more-important tasks will keep unit tests in the waiting line. Then you know what to do. Corkboard misses your photo. Let them out again! Get high with a little help from our friends.
4 responses to “With a little help from our friends”
The greatest benefit of unit test in not catching bugs for me. They improve the architecture, ensure loose coupling and are always up to date documentation (if we run them regularly)
How do unit tests help, and can you prove it with real numbers? That’s easy to answer – if you care to track it:
1. How many regression errors did you catch this release during development?
2. How many bugs were found after going to production in new/changed code that had unit tests?
3. How many bugs were found after going to production in new/changed code that did not have unit tests?
Repeat these questions for “last release”, “Two releases ago”, “N releases ago”.
Most notably, what is the difference between questions #2 and #3?
You’re right Erin, but that requires some time and effort to gather relevant numbers. Even then somebody can say that it doesn’t prove anything.
As a reminder of the woodcutter’s parable, Are you too busy to sharpen your saw?
And yes, anybody can say it doesn’t prove anything, but then you have bigger fish to fry if they actually balk at empirical evidence. Of course your numbers also have to be relative to something else; bugs/LOCs, bugs/StoryPoint, etc. Just saying 2 bugs in unit-tested code vs 2 bugs in non-unit-tested code doesn’t say much, especially if you have 100k unit-tested LOCs vs 1k non-unit-tested LOCs. I recommend the book, The Game of Work by Charles Coonradt, for advice on tracking performance and generating meaningful numbers, regardless of your profession.