Is manual testing crippling your development project?


Dave Furlani






Manual testing is typically how most companies begin their journey into the jungle of software testing. Only companies where the founders are experienced test automation people would start automated testing from day one, and even then it is highly unlikely. So how does a company know when it is time to make the jump to automated testing?

It is time to introduce automated testing when you can no longer run the entire regression test suite before the development team produces the next build version for testing. From that point onwards, every regression test not executed introduces additional risk to the company. These risks always return and cost the company in higher defect rectification costs. There are several studies that document the increasing costs to development projects as time passes between the defect entering the software and the defect being rectified. Defect rectification costs are reported to be up to 80% of the total cost of a development project. The sooner a defect is fixed after being reported, the lower the cost to the company, and the faster a defect is reported, the lower the cost again. Defects that are not rectified can also result in a loss of consumer confidence.

Automating test cases provides some excellent benefits:

  • Faster defect feedback – Automated tests provide the development team with feedback about the code base within minutes, not weeks. This reduces the time, and therefore the costs to fix the defect.
  • Faster release cycles – By automating regression tests, testing efforts can concentrate on new software features and less time in spent on testing older code resulting in improved code coverage.
  • Higher quality software – with a net of automated tests capturing regression defects the risk they will make it through to production is reduced, and the resulting software will be of higher quality.
  • Customer confidence – solid software will impress customers, generating good word of mouth and increasing return business.

Automated Testing Frameworks

An Automated Testing Framework is a set of concepts, and practices that provide support for automated testing. A good framework will allow re-use of test cases, produce robust test cases that require less maintenance, are maintainable if they require updating, and most importantly should allow for tests to be created efficiently.

Many testing frameworks require a moderate to high level of understanding of scripting languages in order to create the test cases – javascript, VB script, or a proprietary scripting language. These tools require a large investment of time to create and maintain the test cases. In some cases you could expect to spend over 120 times the length of time to script the test, as to perform it manually. This is hardly efficient. You would need over 120 release cycles without needing to update or maintain the test script in order to recover the costs of the time invested. Don’t panic, not all tools are that bad and one is a stand-out, but I’ll come to that.

Characteristics of an efficient automated testing framework

Choosing the right testing framework for your development project is important. The wrong tool can cost a lot of time and effort. Moving from manual testing to hand coding test scripts is almost certain to produce poor results and take a lot of time to get results at all. The temptation then is to ask developers to concentrate on scripting automated functional tests, but this is a poor use of their time and skills as it takes them away from product delivery.

A good automated testing framework will have the following characteristics:

Simple Test Creation – Record and Replay allows for tests to be created by recording the actions of the user during manual tests. If implemented well in the testing framework this is the fastest way to create test scripts. Test scripts can be created in almost the same amount of time as required to perform the test manually. A good Record and Replay tool will produce scripts that can handle dynamic content, and will usually require no additional manual editing for the script to be able to execute. This means the investment of time is returned within two or three release cycles.

Data Driven – Manual testers will often re-run the same test for regression using different data each new release. The same practice should be allowed in automated tests. By allowing the use of data files a test can be performed multiple times quickly (i.e. performed once per row of data), and the data file can easily be updated quickly keeping the test fresh.

Parallel Execution – A good testing framework will allow for execution to be performed in parallel, either on a single server, or across multiple servers at once. This allows for your test scripts to be executed more efficiently and allows for your test lab to grow over time.

Scheduled Execution and Continuous Integration – automated test scripts are most valuable when they can be executed frequently and without recurring manual intervention. Execution of automated test scripts should be scheduled at the least, triggered by the build script to execute once a build is completed successfully, or ultimately, triggered to execute as part of the build process on a continuous integration server (e.g. Hudson).

Reporting Test Results – both the development and testing teams need to be able to see the results of the tests executed. Developers should see the results of tests as soon as possible after committing new code. By executing automated functional test scripts along with unit tests on a build or continuous integration server, developers get feedback on code quality very quickly. Ultimately the developers should be able to execute the tests on their development machines, so they can check their code against automated functional tests and only submit code that they can be sure will pass (i.e. quality code). In a perfect world the development team would not allow the new code to be released to the test team while automated tests are failing.

LiquidTest is an example of a testing framework that allows your company to grow beyond the time restraints of manual regression. LiquidTest reduces the investment of time needed to code test scripts and reduces test script maintenance. It meets each of the characteristics of an efficient automated testing framework:

  • It takes minutes to create tests scripts that take hours to build with other automated testing tools
  • The tests are robust, requiring maintenance less frequently
  • It allows for tests to execute multiple times based on data rows in a data file
  • It can be scheduled, or triggered on a build or continuous integration server along with the unit tests
  • Developers can execute the tests on their workstations before even committing code
  • It can execute tests in parallel on a single server or on multiple servers
  • It reports tests results using the build or integration servers test reporting facility to provide the developers with defect feedback in minutes

A trial of LiquidTest and relevant documentation can be found here – www.jadeliquid.com



These icons link to social bookmarking sites where readers can share and discover new web pages.
  • Digg
  • del.icio.us
  • StumbleUpon
  • Reddit
  • DZone
  • TwitThis
  • email
 Comments (0)





No comments yet

Add Comment..



About this blog..


This is an informal place for the team at JadeLiquid to discuss software, the rotation of the earth and other things usually discussed in the JadeLiquid corridors.




Subscribe: By Feed  By Email













Software Testing








Recent Posts



Selenium 2 / WebDriver support coming soon to LiquidTest..
Running Selenium Tests from the Command Line
LiquidTest 3.0 with Selenium support ships!
Setting up Atlassian Bamboo to run automated Web 2.0 tests
888 installer builds and 7469 code checkins later LiquidTest 2.0 ships!






Popular Posts



Testing complex Ajax content
Running Selenium Tests from the Command Line
Setting up LiquidTest with Subversion and Hudson (Continuous Integration) - part 1
Creating Data-Driven Functional Tests with LiquidTest
One tool to rule them all - Reducing team division
Setting up LiquidTest with Subversion and Hudson (Continuous Integration) part 2
Is manual testing crippling your development project?
Eclipse and LiquidTest UI Introduction
Protecting investment: Tests that do not break with page changes
Testing Dynamic Elements – Having a Plan B









May 2011 (2) April 2011 (1) November 2010 (2) March 2010 (1) February 2010 (1) November 2009 (1) October 2009 (1) September 2009 (2) August 2009 (1) July 2009 (1) June 2009 (1) April 2009 (1) March 2009 (1)





   News / Events


 > LiquidTest EOL Announcement - Info
 > LiquidTest Release 3.0! - Available
 > Visual Studio Plugin Released! - Release
 > Cruise Control .NET Integration - Info
 > Automate your Dev/Test Process - Webinar
 > Is manual testing crippling your project? - Blog
 > Testing complex Ajax content - Blog
   Recently Added Content  
 > Run LiquidTest's on Selenium RC/Server - Info
 > Setting up LiquidTest with Maven - Blog
 > Officially Supported Ajax Frameworks - Info
 > Atlassian Bamboo Integration - Tech
 > Setting up LiquidTest with SVN and Hudson - Blog
 > Creating Data-Driven Functional Tests - Blog
 > Reducing Test Maintenance - Blog
  Copyright JadeLiquid Software - 2014