Motivation: Increase the quality and reliability of our codebase - the NetBeans modules.
When developing - writing new code, refactoring or just bug fixing - people always make mistakes and produce
new bugs. This is an unavoidable fact and there is almost nothing we can do to prevent ourselves from making
mistakes when coding (perhaps writing less code). That's why we should carefully learn how to find our own
mistakes reliably and quickly. The sooner the bug is found and fixed the better, because once the bug is fixed
can no longer negatively influence further work. The best solution would be to find and fix all bugs before the
code containing them is committed into the version control system and the bugs can disrupt others who use the code.
Every developer is excited when designing and writing new features and every developer hates fixing and
code maintenance. In reality developers spend significantly more time on bug fixing than developing
new features, don't you think? Why? They haven't found their own mistakes before putting the code into the user's hands! Finding
bugs immediately after writing new code will give you more time for new features and for fun. Besides that, it will
make your users happy and your boss satisfied. Yes, so much good can come just from having no bugs.
How to find bugs?
In NetBeans there are several instruments which we use to find bugs. They are focused on different
levels of testing, use different methodology and have different complexity, but in the end they all
test the code and are capable of detecting bugs. Hopefully the list isn't final and will soon grow.
The NetBeans QE team releases Milestones to assure a basic level of quality during the release cycle.
The Milestone Process is well-established among the developer community. Milestones reports are published on the NetBeans site and announced on the mailing list. There are strict rules followed by all developers about when and how to fix Milestone stopper bugs.
Commit Validation Suite
The Commit Validation Suite should help all developers to verify that the changes they are going to push
to the NetBeans Mercurial repository don't break any vital functionality provided by the NetBeans modules. This is the suite
of tests gathered among NetBeans modules which developers should run before pushing their changes to the Mercurial
repository. Remember, your bugs may disturb other developers around the globe.
The Commit Validation Suite can be extended by any module whose functionality the developers consider
vital and should be tested before every commit. You can read more about
commit guidelines here.
The NetBeans developers and QA team invested a lot of time in implementing an easy-to-use framework allowing
to write and run unit tests. Nowadays I doubt there are any developers who have never heard of
unit testing or think that writing unit tests is a bad thing.
Unfortunately, the reality is that even those developers still do NOT write
any tests. Why? They haven't discovered the benefits.
Why write unit tests?
Tests represent the most practical design spec possible -
Unit tests constitute design documentation that evolves naturally with a system. This is the Holy Grail of software
development, documentation that evolves naturally with a system. What better way to document a class than to provide
a coded set of use cases. That's what these unit tests are: a set of coded use cases that document what a class does,
given a controlled set of inputs. As such, this design document is always up-to-date because the unit tests always
have to pass.
Tests give a developer confidence in the code - you can always check whether your change breaks something
or not by running tests. Well-written tests should catch all potential regressions.
Tests are the lab - when testing your code this is usually done in laboratory environment which eliminates
dependencies and side-effects caused by other parts of code. You can test each functionality separately which will
allows you to quickly track the problem. You can't expect your code to work in the production environment if
it doesn't in the lab.
Tests speed up refactoring - by coding a set of use cases into your tests you can easily verify that you
are not compromising any functionality or causing incompatibilities when refactoring your code. Well written tests
should catch all potential incompatibilities.
Tests help to fight memory leaks - it's usually hard to find memory leaks in a program, and it's harder when
the program is big and complex. Running tools like OptimizeIt doesn't necessarily help in all cases and it's usually
slow. It's better to write tests for isolated pieces of code (subsystems) which check whether your objects
are really getting garbage collected when they aren't used anymore.
Tests will make your boss more happy - by writing tests your code will be more stable attracting more users,
making the product more successful. That's what all bosses like!
How many tests do I need to write?
All of them, and it still will not be enough!
Seriously, you should write as many tests as possible, ideally the tests
should cover all the code in your module. Practically this is hard to achieve and software projects
usually define some lower level of code coverage as sufficient. However, the code coverage by unit tests
should be at least 70% for production quality code. This is a high number!
The problem with code coverage by tests is that even 100% coverage doesn't mean that there are
no bugs in the tested code. This is because your code behaves according to the state of memory and thus
you shouldn't only exercise each line of your code but exercise it for each possible state of memory.
Obviously threading issues bring even more complexity to the problem.
When to write Unit tests?
In NetBeans, currently we have a lot of code which isn't covered by tests. Trying to write tests for all
this code sounds like a never-ending task and in fact it's not really practical to stop all work and write
tests for all the code now! Instead we should establish few rules about how to continuously improve the situation and
increase the number of code lines covered by tests. And here it goes:
All newly written code should be covered by unit tests
When fixing a bug, a test should be written to simulate the bug and then verify that the fix really solved
Whenever you feel tired by your current work try to write some tests for existing code which isn't tested
enough yet (there is a lot of untested code overall in the NetBeans sources).