Chapter 21. Testing

Chapter 21. Testing
Prev	Part III. Advanced concepts	Next

Description

Obix has an integrated framework for testing software components. The basic idea is to embed test scripts everywhere in the source code, so that source code can repeatedly be tested in an easy way. The goal of these tests is to quickly find a maximum number of coding errors (bugs) in an early stage of the project.

Obix supports the bottom-up approach for testing software. Small software units are first created and then tested immediately. After checking these lowest level units, higher level units that use lower level units (and/or units at the same level) are created and again tested immediately. This cycle continues until the highest level of abstraction is reached, which is typically the application object.

	Note
	Testing in Obix has a number of commonalities with unit testing which exists as optional third-party additions in other languages (e.g. JUnit for Java).

The rules for writing tests in Obix are explained in the next section.

Rules

The smallest unit that can be tested in Obix is a script. Each script can optionally have an associated test script used to test the functionality of the script.
The test script is appended to the script to be tested between a test and end test instruction.
A script with no test script has the following syntax:
```
script
   // script instructions
end script
```
And the syntax for a script with an associated test script is:
```
script
   // script instructions
end script
test
   script
      // test script instructions
   end script
end test
```
Each kind of script, except test script themselves, can have an associated test script. See the section called “Kinds of scripts” for a complete list of kinds of scripts.
The following terminology is used:
The script to be tested is called the tested script.
The test script is called the script test script.
A script test script is used to test the correctness of the tested script it belongs to. It does this by calling the tested script with different input conditions and checking the results for each call. If a result is different from the expected result, a test fail error is created and appended to a list of all encountered test fails.
For more information on script test scripts see the section called “test script”.
A test script can also be associated with a Root Software Element (RSE) (i.e. a type, factory or service).
A test script that tests a RSE is called a RSE test script.
A RSE test script always appears at the end of a RSE's source code, after the definition of the RSE's features (e.g. attributes, commands and events).
For example, a factory with no RSE test script has the following syntax:
```
factory foo type:foo

   // attributes
   // commands
   // creators

end factory
```
And the syntax for the same factory with an associated RSE test script is:
```
factory foo type:foo

   // attributes
   // commands
   // creators

   test
      script
         // RSE test script instructions
      end script
   end test

end factory
```
A RSE test script tests a RSE by using the RSE's features (attributes, commands and events) and comparing the real results with expected results. If a result doesn't match the expected result, a test fail error is created and appended to a list of all encountered test fails.
For more information on RSE test scripts see the section called “test script”.
Test scripts can contain all script instructions explained in Chapter 8, Script instructions.
As test scripts can contain all kinds of instructions, the full power of the language can be used to create and manage test cases. For example, instead of hard-coding input values and their corresponding expected results in the source code, they could be read from an external source, such as an XML or Excel file fed by people who are not necessarily programmers (e.g. the users of the software).
The verify instruction is used to detect test fails by comparing real results to expected results.
For more information on the verify instruction see the section called “verify instruction”.
The verify error instruction is used to ensure that a runtime error is generated in a given situation.
For more information on the verify error instruction see the section called “verify error instruction”.
The test instruction is used to launch a test case in a script test script.
For more information on the test instruction see the section called “test instruction”.
Note
The test instruction cannot be used in RSE test scripts, it can only be used in script test scripts.
All test scripts in a RSE are executed by calling the RSE's test_ command, which is implicitly defined whenever at least one test script is defined in the RSE.
The test_ command executes the RSE test script as well as all script test scripts defined in the RSE.
Command test_ has one input argument that holds the list of all test fails encountered. This input argument's id is i_test_fail_list_ and its type is ty_test_fail_list. Each time a test fail is detected through a verify or a verify error instruction in anyone of the test scripts, an object describing the test fail is automatically appended to i_test_fail_list_.
At the time of writing type test_fail_list is defined as follows:
```
type test_fail_list

   command is_empty
      out result type:yes_no end
   end command

   command item_count
      out result type:zero_positive32 end
   end command

   command item_iterator
      out result type:iterator end
   end command

   command append
      in test_fail type:test_fail end
   end command

   command display_result
   end

end type
```
The following code shows how to execute all test scripts of an RSE (fa_bank_account in this example):
```
         // create list to hold all test fails encountered during testing
         var ty_test_fail_list v_test_fail_list = fa_test_fail_list.co_create

         // execute all test scripts in fa_bank_account
         fa_bank_account.co_test_ ( v_test_fail_list )

         // display the result of testing
         v_test_fail_list.co_display_result
```
Note
A future version of Obix will provide an easier way to execute test scripts and display their results.

Rationale

It is a proven fact that one of the most important and efficient methods for finding coding errors is testing the code by running it under a representative number of normal and exceptional conditions. The better the tests, the more likely are the chances to find remaining errors.

Smart compilers and intelligent code analyzing tools are able to detect some errors, but they can't eliminate the need for testing software before delivery. Hence, testing and debugging software is undoubtedly one of the most important tasks during software development.

However, if testing is not supported by the programming language itself, then testing can be cumbersome and risks resulting in just a few and sometimes sloppy executed manual tests. Most importantly, because of the lack for easily saving and automatically re-executing test cases, there is a high risk of not detecting new errors introduced after changing or extending existing code.

Therefore, support for testing has been integrated in the core of the Obix programming language. It should always be easy for the programmer to write tests, preferably without the need for choosing and installing an optional third-party testing framework.

Moreover, test scripts obviously have a very pleasant side effect. They provide technical documentation about software components in a precise, up-to-date and reliable manner. This reduces (or even eliminates) time spent with the boring task of writing technical documentation as well as the much more boring and error-prone task of updating and maintaining that documentation. Test scripts help to quickly understand the exact behavior of software components. Moreover, because boundary conditions and error generating situations are included in well written test cases, every programmer is well informed about exceptional situations to consider.

Testing examples

For a first simple example of testing a service command, see Example 7.3, “A simple test script example”

To see how a factory can be tested, suppose a tiny application that manages bank accounts.

A bank customer is defined as follows:

type bank_customer default_factory:yes
  
   attribute identifier type:positive32 end
   attribute name type:string end
   attribute city type:string end
  
end type

A bank account is associated with one bank customer. There are two operations: pay_in and withdraw. The source code for type bank_account looks like this:

type bank_account

   attribute customer type:bank_customer end 

   attribute balance type:zero_positive32 default:0 kind:variable setable:factory end 

   command pay_in
      in amount type:positive32 end
   end command 

   command withdraw
      in amount type:positive32 end
   end command 

end type

Suppose the factory to be tested is the following one:

factory bank_account type:bank_account

   command pay_in
      script
         a_balance = a_balance + i_amount
      end script
   end command 

   command withdraw
      script
         a_balance = a_balance - i_amount
      end script
   end command 

   creator create
      in customer type:bank_customer end

      out result type:bank_account end

      script
         o_result.a_customer = i_customer
         o_result.a_balance = 0
      end script
   end creator 

end factory

There are 4 different test scripts that can be written for this factory:

2 script test scripts for testing the scripts of commands pay_in and withdraw
one script test script for testing the creator
one RSE test script to test the whole factory

Tests for command pay_in can be written like this:

...

   command pay_in

      script
         a_balance = a_balance + i_amount
      end script

      test
         script
            // create an object for testing purposes
            v_test_object_ = create ( fa_bank_customer.create ( &
               identifier = 10 &
               name = "Foo" &
               city = "Bar" ) )

            // first test case. pay in 100 and verify new balance is 100
            test 100
            verify v_test_object_.balance =v 100

            // second test case. pay in 200 and verify new balance is now 300
            test 200
            verify v_test_object_.balance =v 300

            // fourth test case. pay in too much and produce an arithmetic overflow error
            test se_positive32.max_value
            verify error
         end script
      end test

   end command 

...

A we can see:

The script to be tested is immediately followed by a test script.
Variable v_test_object_ (which the compiler implicitly declares in a test script) holds the bank_account object that will be used by subsequent test instructions.
Note
Implicitly defined variables are, by convention, always followed by an underscore (_) in order to distinguish them from variables that are explicitly declared in the source code.
A first test is launched by instruction test 100. This instruction is similar to writing v_test_object_.co_pay_in ( 100 ), but it is shorter to write and the behavior at runtime is adapted to testing purposes. test 100 executes the tested script and provides values for input arguments (i_amount = 100 in our case).
The verify instruction is used to check the result of a test case. verify is always followed by a yes_no expression which has to evaluate to yes if the test passes validation. If the expression evaluates to no, or if a program error occurs then a test fail object is automatically created and appended to the list containing all test fails encountered. In any case program execution continues.
Besides testing for correct results, the verify error instruction is used to ensure that a program error actually occurs in a given situation. In our case, a runtime error must occur in case of an amount paid in that is too big and produces an arithmetic overflow error.

Tests for command withdraw can be written in a similar manner.

To test the features of factory bank_account altogether, the following RSE test script can be inserted at the end of the factory's source code, just before the end factory instruction:

factory bank_account type:bank_account

...

   test
      script
         // create a customer
         var bank_customer bc = fa_bank_customer.create ( &
            identifier = 10 &
            name = "Foo" &
            city = "Bar" )

         // create an account
         var bank_account account = create ( bc )

         // check customer attribute of account
         verify account.customer =r bc
         verify account.customer.name =v "Foo"

         // balance must be 0 after creation
         verify account.balance =v 0

         // add 100 to the account
         account.pay_in ( 100 )
         // verify balance
         verify account.balance =v 100
   
         // withdraw 70
         account.withdraw ( 70 )
         // verify balance
         verify account.balance =v 30

         // withdraw 10
         account.withdraw ( 10 )
         // verify balance
         verify account.balance =v 20

         // ...
      end script
   end test

end factory

To execute all test scripts in fa_bank_account, the following code can now be executed. Any test fails will be displayed on the system console.

         // create list to hold all test fails encountered during testing
         var ty_test_fail_list v_test_fail_list = fa_test_fail_list.co_create

         // execute all test scripts in fa_bank_account
         fa_bank_account.co_test_ ( v_test_fail_list )

         // display the result of testing
         v_test_fail_list.co_display_result

For other examples, see also:

Prev	Up	Next
Chapter 20. Source code templates	Home	Obix's Extended Backus-Naur Form (EBNF)

	Note
	The `test` instruction cannot be used in RSE test scripts, it can only be used in script test scripts.

	Note
	A future version of Obix will provide an easier way to execute test scripts and display their results.

	Note
	Implicitly defined variables are, by convention, always followed by an underscore (_) in order to distinguish them from variables that are explicitly declared in the source code.