Every piece of software should be able to test itself!

Automated testing is a magnificent help in finding programming errors and thus delivering more reliable, successful software. Several reports show that automated testing considerably increases software quality and reduces total development costs. No wonder that the subject has become more and more popular during the last years. Surprisingly, nearly all programming languages lack any integrated support for automated testing.

Because automated testing is so much important, it has been integrated into Obix in an easy-to-use way. Combined with Program with checks they are an invaluable help for creating reliable code.

Ideally, automated testing should be able to prove that a program is correct. Unfortunately, this is only possible in some very rare cases. The reason is that, to prove the correctness by testing, a test case would have to be defined for every set of possible input values. But in most cases, the total number of possible test cases reaches astronomical values, easily surpassing the number of atoms in our universe. Let's take, for example, a command with one integer input and one string input with a fixed length of 20 characters, each character representing an uppercase letter. This would yield into 2^32 x 26^20 = 8.55907E+37 different test cases. If each test case took only 1 microsecond to execute then 8.55907E+37 / 1000000 / 60 / 60 / 24 / 365 = 2.71406E+24
= about 3000000000000000000000000 years would be needed to prove the correctness of our command.

A much better way to be sure of the correctness would be a mathematical proof. But, unless we are a genius in math and programming (like Donald Knuth: 'I proved that the program is correct, but I didn't test it.') and ready to invent new methods for proving 0 errors in source code, we simply have to forget this option.

The only choice we have is to define our test cases in such a way that the chances of finding every error are maximized. That's the art of testing: finding a maximum of errors with a minimum of test cases. However, even if we forget some cases: It is much better to provide some tests than to provide none.

Although automated tests will never be perfect error reporters, they contribute to the following advantages:

• They undoubtedly reduce programming errors
  
  
• They are an unbeatable help for finding new errors introduced when code is modified. The scenario is well known: Shortly after delivering a new version to our customer, he calls us: 'Yes you added function C I asked for, but now function A and B don't work anymore'. Depending on the quality of the tests we can be confident that in the case of adding a new functionality or in the case of refactorings, simply re-executing all tests will report errors we did not think of.
  
  
• Test cases integrated in the source code have the pleasant side effect of providing technical documentation by example, at no cost. Looking at the test cases will help a reader to better understand the script's behavior, and shows the results produced in the case of boundary values.
  
  
• Once again: They reduce development costs and considerably reduce maintenance costs
  

Here are some guidelines that should be helpful for testing software:

• Resist the temptation of writing no test. Even if it takes some time, we can nearly always be sure to win at the end. And we feel better, knowing that we did what we should have done.
  
  
• Have a humble attitude toward testing. Every programmer makes errors. Instead of thinking 'I write this test to prove that my script is correct' a much better attitude is: 'I write this test in order to detect all possible errors'
  
  
• Write the test script before writing the script. The benefit will be that we will have a much clearer idea of what the script should do, and the chances to consider all exceptional cases in the script's first version are higher.
  
  
• Test boundary values systematically, because they typically are the sources of many errors. For example, in the case of an integer input, we should write test cases for the values 0, 1, -1, 2, -2, max_value, max_value - 1, min_value, min_value + 1 and void.
  
  
• Consider writing table-driven tests. Instead of hard-coding test cases into the source code, input values and expected output values could be defined in an external source, like an ASCII, XML or Excel file that has been prepared and is easily modifiable by the people who defined the specifications, e.g. the customer.