What is software testing
Software testing is the act of examining the artifacts and the behavior of the software under test by validation and verification. Although software testing can determine the correctness of software under the assumption of some specific hypotheses, testing cannot identify all the failures within the software.
The process of testing (software) can be manual or automated. It is obvious that while the automated testing process takes more time for the it’s development, it is easier to run it again and again in the future compared to the manual one. For the context of this page, when we refer to test we will mean an automated software test.
Types of testing
There are lots of software testing approaches, each one used for different reasons and has its own benefits or drawbacks. Hare is a short list presenting how are tests grouped based on different aspects:
- White Box Testing : verifies the internal structures or workings of a program, as opposed to the functionality exposed to the end-user (also known as clear box testing, glass box testing, transparent box testing, and structural testing).
- Black Box Testing : treats the software as a “black box,” examining functionality without any knowledge of internal implementation, without seeing the source code (also known as functional testing).
- Component Interface Testing :a variation of black-box testing, with the focus on the data values beyond the related actions of a subsystem component.
- Unit Testing : verify the functionality of a specific section of code, usually at the function level.
- Integration Testing : seeks to verify the interfaces between components against a software design.
- System Testing : tests a completely integrated system to verify that the system meets its requirements.
- Acceptance Testing : conducted to determine if the requirements of a specification or contract are met. It may involve chemical tests, physical tests, or performance tests.
- Performance Testing : is executed to determine how a system or sub-system performs in terms of responsiveness and stability under a particular workload.
We can see that some types of testing do not assist on validating the correctness of a software. They assist us to validate some requirements we have set. A great example of such a test group is the Performance testing where we have to reassure that the software produces complies with some performance requirements that we have set.
Test pyramid
It makes sense that each different testing types and technique exists to serve a different need or a different approach. Some of them are use to test small code snippets while other are designed to test a system as a whole. The smaller the unit that a test examines for defects it will probably be easier to write and will probably require less time to run compared to a test that checks a larger portion of a system or codebase.
The metaphor of the test pyramid is a way to represent the grouping of tests based on their granularity. Also it provides a way to give an idea of how many tests we should have in each of these groups.
The main message that the test pyramid gives us is that we have to write more tests that check smaller parts of our system or code and less tests that check our system in a more integrated with any 3rd party system or with other subsystems. Also, the tests that are closer to the top of the pyramid are generally more expensive in terms of development and also slower on their execution.
While test pyramid gives us an example of how many tests of each type we should write for our system, we should always adopt this to our own system and its needs.
The AAA
- Arrange
- Act
- Assert
This is the triple A (AAA) principle that each test should follow. A test is mainly consisted of these three distinct parts.
The Arrange part is the part of each test where we have to prepare all the required and appropriate prerequisites for our test scenario to run. For instance, we have to prepare any data on the database, establish a connection to a 3rd party system or even prepare a mock that will be latter used. This part of the test should use as many primitive functionality of our software so that any defect on the higher level functionality wont affect our test.
The ACT part is the part of each test where the functionality under test is invoked or triggered in the predetermined way. The ACT part is usually smaller compared to the other parts at it’s main role is just to trigger the functionality under test.
The ASSERT part is the part of the test where we have to check that the functionality under test produced the expected results. Such results could be the return value(s) of a function, the data in a Database, a message on a message broker system and more. Assertion are the way we endure that the functionality under test had the expected behavior.
In many cases within each test we write down a comment indicating each different part of the test:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
//Arrange
db.CreateUser("name", "[email protected]")
//Act
user, err := users.GetByName("name")
//Assert
if err!=nil{
t.Errorf("did not expect error, but got one: %v", err)
}
if user.Name!="name"{
t.Errorf("expected user name %v, but got: %v", "name", user.Name)
}
if user.Email!="[email protected]"{
t.Errorf("expected user email %v, but got: %v", "[email protected]", user.Email)
}
In some case a multiple AAA approach is followed. At multi AAA a test follows multiple Act and Assert phases like: Arrange -> Act -> Assert -> Act -> Assert and so on. While this seems many time a bit tempting to use we should avoid using it especially in Unit testing where we have to test just a specific functionality.
Arrange and Assert parts of the test are pretty crucial. At the arrange part we have to carefully prepare all the prerequisites in order to be able to reproduce the scenario we want to run and test. The assert part is where we verify that everything went as expected, any data produced are valid and no unwanted data altered or created.
Cleanup
As we usually run a bunch of tests every time we must reassure that a test does not affect somehow any other test execution - even the exact same test. For this reason it is essential to clean things up as soon as the are no longer required. Cleanup should include database records, disk files, 3rd party integrations and even cleanup anything created by the functionality under test.
In go, the defer statement is suitable for such cases:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
//Arrange
dbUser := db.CreateUser("name", "[email protected]")
//cleanup
defer dbUser.Delete()
//Act
user, err := users.GetByName("name")
//Assert
if err!=nil{
t.Errorf("did not expect error, but got one: %v", err)
}
if user.Name!="name"{
t.Errorf("expected user name %v, but got: %v", "name", user.Name)
}
if user.Email!="[email protected]"{
t.Errorf("expected user email %v, but got: %v", "[email protected]", user.Email)
}
Test coverage
Test coverage is a percentage measure of the degree to which the source code of a program is executed when a particular test suite is run. Each line of the source code executed during a test execution is considered as test covered. The more the lines we cover with our test the higher the test coverage of our tests.
Text coverage is measured in percentage and usually a bare minimum it pursued to be achieved. Ensuring a minimum value to this measurement during the lifecycle of the development process, indicates that we value the necessity of the tests.
Trusting the tests
Test coverage is a great tool when it come for gaining trust to a test. When we execute a test with test coverage details we ψαν review each line covered by that test. Using this approach we can identify if the lines covered by the test are the expected ones. It’s a common case that a test could expect some error but not checking the specific error might cause a successful test but not checking the expected behavior.
Higher test coverage does not necessarily mean that a program has fewer bugs. While test coverage is measured by the degree to which the source code of a program is executed when a particular test suite is run this does not implies that we are have the right assertions for each part of our code. This means that we can simply achieve a hight test coverage (even 100%) and we could change our source code without having any test to fail! More details on this can be found on a previous post about Testing the Tests where a technique called Mutation testing is presented which verifies the existing tests with an automated way.
Wrap up
As we saw earlier, there are lots of different testing approaches and techniques. Each one has its own benefits and drawbacks. As each project has its own features and particular requirements we must carefully select which types of tests and up to what a degree are required. Testing is a difficult path, sometime more difficult than the software development. While doing it right can be very beneficial, doing it in a not do appropriate way might make our work harder.