Manual testing is typically how most companies begin their journey into the jungle of software testing. Only companies where the founders are experienced test automation people would start automated testing from day one, and even then it is highly unlikely. So how does a company know when it is time to make the jump to automated testing?
It is time to introduce automated testing when you can no longer run the entire regression test suite before the development team produces the next build version for testing. From that point onwards, every regression test not executed introduces additional risk to the company. These risks always return and cost the company in higher defect rectification costs. There are several studies that document the increasing costs to development projects as time passes between the defect entering the software and the defect being rectified. Defect rectification costs are reported to be up to 80% of the total cost of a development project. The sooner a defect is fixed after being reported, the lower the cost to the company, and the faster a defect is reported, the lower the cost again. Defects that are not rectified can also result in a loss of consumer confidence.
Automating test cases provides some excellent benefits:
- Faster defect feedback – Automated tests provide the development team with feedback about the code base within minutes, not weeks. This reduces the time, and therefore the costs to fix the defect.
- Faster release cycles – By automating regression tests, testing efforts can concentrate on new software features and less time in spent on testing older code resulting in improved code coverage.
- Higher quality software – with a net of automated tests capturing regression defects the risk they will make it through to production is reduced, and the resulting software will be of higher quality.
- Customer confidence – solid software will impress customers, generating good word of mouth and increasing return business.
Automated Testing Frameworks
An Automated Testing Framework is a set of concepts, and practices that provide support for automated testing. A good framework will allow re-use of test cases, produce robust test cases that require less maintenance, are maintainable if they require updating, and most importantly should allow for tests to be created efficiently.
Characteristics of an efficient automated testing framework
Choosing the right testing framework for your development project is important. The wrong tool can cost a lot of time and effort. Moving from manual testing to hand coding test scripts is almost certain to produce poor results and take a lot of time to get results at all. The temptation then is to ask developers to concentrate on scripting automated functional tests, but this is a poor use of their time and skills as it takes them away from product delivery.
A good automated testing framework will have the following characteristics:
Simple Test Creation – Record and Replay allows for tests to be created by recording the actions of the user during manual tests. If implemented well in the testing framework this is the fastest way to create test scripts. Test scripts can be created in almost the same amount of time as required to perform the test manually. A good Record and Replay tool will produce scripts that can handle dynamic content, and will usually require no additional manual editing for the script to be able to execute. This means the investment of time is returned within two or three release cycles.
Data Driven – Manual testers will often re-run the same test for regression using different data each new release. The same practice should be allowed in automated tests. By allowing the use of data files a test can be performed multiple times quickly (i.e. performed once per row of data), and the data file can easily be updated quickly keeping the test fresh.
Parallel Execution – A good testing framework will allow for execution to be performed in parallel, either on a single server, or across multiple servers at once. This allows for your test scripts to be executed more efficiently and allows for your test lab to grow over time.
Scheduled Execution and Continuous Integration – automated test scripts are most valuable when they can be executed frequently and without recurring manual intervention. Execution of automated test scripts should be scheduled at the least, triggered by the build script to execute once a build is completed successfully, or ultimately, triggered to execute as part of the build process on a continuous integration server (e.g. Hudson).
Reporting Test Results – both the development and testing teams need to be able to see the results of the tests executed. Developers should see the results of tests as soon as possible after committing new code. By executing automated functional test scripts along with unit tests on a build or continuous integration server, developers get feedback on code quality very quickly. Ultimately the developers should be able to execute the tests on their development machines, so they can check their code against automated functional tests and only submit code that they can be sure will pass (i.e. quality code). In a perfect world the development team would not allow the new code to be released to the test team while automated tests are failing.
LiquidTest is an example of a testing framework that allows your company to grow beyond the time restraints of manual regression. LiquidTest reduces the investment of time needed to code test scripts and reduces test script maintenance. It meets each of the characteristics of an efficient automated testing framework:
- It takes minutes to create tests scripts that take hours to build with other automated testing tools
- The tests are robust, requiring maintenance less frequently
- It allows for tests to execute multiple times based on data rows in a data file
- It can be scheduled, or triggered on a build or continuous integration server along with the unit tests
- Developers can execute the tests on their workstations before even committing code
- It can execute tests in parallel on a single server or on multiple servers
- It reports tests results using the build or integration servers test reporting facility to provide the developers with defect feedback in minutes
A trial of LiquidTest and relevant documentation can be found here – www.jadeliquid.com