Automated UI Testing

This is the fourteenth blog of a series of posts about the topics Continuous Integration and Continuous Delivery. It explains how important continuous integration is, to deliver software in short iterations to the customer with a high quality standard.

After an automated deployment has been set up, we have the chance to test our application on the target platform. We can figure out problems much earlier in the development phase and we can react on it. But manually testing applications is very time-consuming. Therefore, the basic key scenarios and regression tests should be automated. That gives the possibility to execute those tests after changes have been made to the software (e.g. every night). The result is fast feedback about the state of the application. These tests can and should be even executed after every deployment in order to check the health of the software and identify configuration issues. This gives us reliable feedback weather the applications main features are working or not. This results in a higher customer satisfaction even if not all the bugs are found upfront but at least the application does not break down after the first click and the main business can be still served. If a critical error is reported by the customer, which has not been found by the automated tests, they should be of course extended.

Microsoft and HP provide great tools for automated UI tests. HP QuickTest is the market leader in this field and provides a mature and stable framework for automating UI tests. Microsoft’s Coded UI Tests are much newer and do not support as many UI technologies as HP QuickTest (e.g. Java and Flash applications are not supported out-of-the-box by the Microsoft Test Framework). But I would still consider the Coded UI Tests from Microsoft if you are working with the Team Foundation Server and Windows/Web Applications. The coded UI tests can be integrated into the CI build of the TFS (like Unit Tests) and therefore easily executed after deployments and scheduled by the TFS build system.

Short comparison between Microsoft Coded UI Tests and HP QuickTest:

	Microsoft Coded UI Tests	HP QuickTest
Supported Platforms	Windows, Web	Windows, Web, Java, Flash, SAP, etc...
Test Types	UI Tests, Functional Tests, Uni Tests, Performance Tests, Load Tests, Manual Tests	UI Tests, Functional Tests
Maintainability	Separation between object identification and test methods (complex UI Maps)	Separation between object identification and test methods (simple object repository)
TFS Integration	Highly integrated with Test, Bug and Task Management as well as build integration	Plug-in needed (see HP Quality Center Synchronizer - TFS Adapter)
Custom Extensions	Has open architecture with support to write a variety of extensions	Mainly not supported
Summary	Provides integrated environment with TFS and .NET Windows and Web Applications but does not provide many technologies out-of-the-box	Should be used when many different platforms and technologies take part

Automated UI Tests mainly fail and cause high effort in maintenance because of the following reasons:

• The UI element identification is not separated from the test steps.
  That means that the different UI elements, like textboxes and buttons, are identified in many different places in the code because the same UI elements are used from different test cases. Usually, the criteria (e.g. an ID or Text) changes quite often. And that is the reason why it is important to centralize the object identification parameters. That changes have to be applied just once.
• The test cases rely on instable data.
  They are usually designed as end-user tests which depend on the functionality and data of all the connected systems. Of course, you can try to write tests which do not depend on the data but that also means the test does not cover the most important parts of your application. Therefore it is very important to think about the data management before actually implementing UI tests.
• The test cases depend on each other or a complex test setup.
  In order to minimize the maintenance effort, test cases should be independent from each other. Otherwise many or all tests fail because of one single problem.

I have rarely seen good written UI Tests because it will still not be taken as seriously as the productive code. It is important to make planning and architecture also for UI Test code because these tests can just help if they life together with the productive code. It is very important to define upfront how to handle the mentioned problems like the identification of UI elements, data management and test case dependencies.

Unit Tests are not enough to ensure the quality of software. They have, indeed, the advantage to give immediate feedback about changes because they can be executed quickly. But it is very important to test the application from the end-user perspective using automated UI tests.
Automated end user tests have to be performed as soon as possible in the development cycle. Usually, we deploy every night the latest source on a test system and execute the automated tests afterwards. This gives us instant feedback about the quality of the check-ins. If there are any problems they can be immediately investigated and fixed, and not just right before the software has to be delivered to the customer. With automated deployments and automated UI tests delivering high-quality software on a regular basis is much easier.