Feature Roll-Out

This is the fifteenth blog of a series of posts about the topics Continuous Integration and Continuous Delivery. It explains how important continuous integration is, to deliver software in short iterations to the customer with a high quality standard.

As funny as it sounds but one of the main problems about Continuous Delivery is the permanent delivery. It can be that one feature is not completely implemented yet and spans multiple releases to get ready. The main idea of Continuous Delivery is to split big requirements into smaller junks which still gives the user new possibilities. Unfortunately, this is not always possible and therefore the feature-flagging technique gets important.

Feature Flagging simply means that every big new functionality should be built in a way that it can be easily turned on and off like in the following listing shown:

public void PlaceOrder(Order order)
{
  var orderSystem = CreateNewInstance();
  orderSystem.Place(order);
}

public IOrderSystem CreateNewInstance()
{
  if (FeatureFlagManager.IsAvailable("NewOrderSystem"))
  {
    return new OrderSystem();
  }
  else
  {
    return new LegacyOrderSystem();
  }
}

Advantages of Feature Flagging
This approach gives a lot of advantages and great flexibility during the roll-out of a new version:

• Features can be switched on and off, even for a certain group of users if the feature flag component has been implemented to support it.
• A feature can be smoothly rolled out for a small group of users (like administrators, testers, people from a country, etc...) and does not affect the other users. Therefore some people can test the feature in the real world environment before it is available for public use.
• The roll-out can be done step-by-step. It can be coordinated and monitored what effects it has on the whole system regarding performance or usability. This approach is especially in web applications with many users extremely useful where the load cannot be simulated on a staging environment anymore.
• If any problem occurs the old variant is just one click away and there is no need for a big rollback with possible data inconsistency or loss.
• Furthermore the problems can be identified by a small group of users and do not affect all users at once which might cause an extreme increase of the support tickets.

Problems with Feature Flagging
Of course, the trade-off of this approach is that the design of new components has to be thought through.
The code for implementing feature flagging (e.g. if clauses, factories or attributes) should not spread around the whole code and make it much more difficult to maintain. If a feature has been completely rolled out, it should be even removed to simplify the code afterwards again.
The applied changes (e.g. database schema change) have to be compatible for both code parts. This has to be considered anyhow to support hot deployments where the application stays online during a deployment.
Additionally, the test effort is higher because both cases have to be tested as well as the possible dependencies between these cases.

But in the end I think that Feature Flagging and step-by-step roll-out is a really important concept which is worth to use in bigger web applications. It helps to reduce the risk of deployments dramatically.

Automated UI Testing

This is the fourteenth blog of a series of posts about the topics Continuous Integration and Continuous Delivery. It explains how important continuous integration is, to deliver software in short iterations to the customer with a high quality standard.

After an automated deployment has been set up, we have the chance to test our application on the target platform. We can figure out problems much earlier in the development phase and we can react on it. But manually testing applications is very time-consuming. Therefore, the basic key scenarios and regression tests should be automated. That gives the possibility to execute those tests after changes have been made to the software (e.g. every night). The result is fast feedback about the state of the application. These tests can and should be even executed after every deployment in order to check the health of the software and identify configuration issues. This gives us reliable feedback weather the applications main features are working or not. This results in a higher customer satisfaction even if not all the bugs are found upfront but at least the application does not break down after the first click and the main business can be still served. If a critical error is reported by the customer, which has not been found by the automated tests, they should be of course extended.

Microsoft and HP provide great tools for automated UI tests. HP QuickTest is the market leader in this field and provides a mature and stable framework for automating UI tests. Microsoft’s Coded UI Tests are much newer and do not support as many UI technologies as HP QuickTest (e.g. Java and Flash applications are not supported out-of-the-box by the Microsoft Test Framework). But I would still consider the Coded UI Tests from Microsoft if you are working with the Team Foundation Server and Windows/Web Applications. The coded UI tests can be integrated into the CI build of the TFS (like Unit Tests) and therefore easily executed after deployments and scheduled by the TFS build system.

Short comparison between Microsoft Coded UI Tests and HP QuickTest:

	Microsoft Coded UI Tests	HP QuickTest
Supported Platforms	Windows, Web	Windows, Web, Java, Flash, SAP, etc...
Test Types	UI Tests, Functional Tests, Uni Tests, Performance Tests, Load Tests, Manual Tests	UI Tests, Functional Tests
Maintainability	Separation between object identification and test methods (complex UI Maps)	Separation between object identification and test methods (simple object repository)
TFS Integration	Highly integrated with Test, Bug and Task Management as well as build integration	Plug-in needed (see HP Quality Center Synchronizer - TFS Adapter)
Custom Extensions	Has open architecture with support to write a variety of extensions	Mainly not supported
Summary	Provides integrated environment with TFS and .NET Windows and Web Applications but does not provide many technologies out-of-the-box	Should be used when many different platforms and technologies take part

Automated UI Tests mainly fail and cause high effort in maintenance because of the following reasons:

• The UI element identification is not separated from the test steps.
  That means that the different UI elements, like textboxes and buttons, are identified in many different places in the code because the same UI elements are used from different test cases. Usually, the criteria (e.g. an ID or Text) changes quite often. And that is the reason why it is important to centralize the object identification parameters. That changes have to be applied just once.
• The test cases rely on instable data.
  They are usually designed as end-user tests which depend on the functionality and data of all the connected systems. Of course, you can try to write tests which do not depend on the data but that also means the test does not cover the most important parts of your application. Therefore it is very important to think about the data management before actually implementing UI tests.
• The test cases depend on each other or a complex test setup.
  In order to minimize the maintenance effort, test cases should be independent from each other. Otherwise many or all tests fail because of one single problem.

I have rarely seen good written UI Tests because it will still not be taken as seriously as the productive code. It is important to make planning and architecture also for UI Test code because these tests can just help if they life together with the productive code. It is very important to define upfront how to handle the mentioned problems like the identification of UI elements, data management and test case dependencies.

Unit Tests are not enough to ensure the quality of software. They have, indeed, the advantage to give immediate feedback about changes because they can be executed quickly. But it is very important to test the application from the end-user perspective using automated UI tests.
Automated end user tests have to be performed as soon as possible in the development cycle. Usually, we deploy every night the latest source on a test system and execute the automated tests afterwards. This gives us instant feedback about the quality of the check-ins. If there are any problems they can be immediately investigated and fixed, and not just right before the software has to be delivered to the customer. With automated deployments and automated UI tests delivering high-quality software on a regular basis is much easier.

Automated Deployments

This is the thirteenth blog of a series of posts about the topics Continuous Integration and Continuous Delivery. It explains how important continuous integration is, to deliver software in short iterations to the customer with a high quality standard.

The basis of Continuous Delivery is a completely automated deployment. It should not depend on single persons and manual clicks whether or not software will be delivered to the customer or just published on a staging server for quality tests. How often have I heard "we cannot deploy because X is on vacation" or "it is too high effort to deploy it now". In order to deliver new features to a customer on a regular basis, automated deployments are a must-have.

Of course, depending on the complexity of the software, automated deployments can be difficult to set up. But Microsoft provides, for instance, already an extensive set of tools for that. In the web environment MSDeploy and MSBuild are the most important once. Besides copying the application assemblies and files, MSDeploy can create automatically up- and downgrade scripts for the databases. Further details about the features of MSDeploy and the deployment process can be found at Enterprise Deployment Tutorial.

In most of the companies there are even organizational boundaries between the software development and operations staff. It is very important that the responsibilities are clearly defined. The operations team provides the mechanism for automated deployments but should not be needed during a regular deployment nor have any knowledge about the exact contain of the deployment package. They should be just responsible for the infrastructure and deployment process itself. The creation of the packages has to be in the hand of the software development team and, of course, is completely automated like mentioned before.

This approach has many advantages:

• The development team is responsible for the software where they have a deep knowledge.
• The development team does not have to investigate problems based on the filtered details from the operations team. They get full access to the tracing information for their application.
• The operations team has is focus on their competences, the infrastructure.
• The operations team can be easily scaled and shared because they do not have to build up the application knowledge anymore but just provide managed IT services.

Continuous Delivery is not just about delivering new features to the customer. It is also even more important to hand over the software regularly to the testing team. As faster they can test new features as sooner we know about certain problems, bugs or even architectural issues in our application. I have seen plenty of times that the software has been developed and handed over to the test team a few days or weeks before delivery. After that when bugs have been found, the panic started because they not just had to be fixed but also re-deployed and re-tested. This all could have been prevented by continuously deployment and testing from the first day of development.

This all should just explain how important automated deployments are and that they are the basis for Continuous Delivery.

Continuous Delivery

This is the twelfth blog of a series of posts about the topics Continuous Integration and Continuous Delivery. It explains how important continuous integration is, to deliver software in short iterations to the customer with a high quality standard.

After I wrote mainly posts about continuous integration, I also want to spend some words about continuous delivery.

With the success of agile development processes, the iterative creation of shippable software pieces is getting more and more important. It actually sounds great to delivery regularly new features to the customer in order to get instant feedback and to understand the solution better. It is usually pretty difficult for non-technical guys to understand complex systems based on hundreds of pages in a software specification. But these popular agile development processes come with other difficulties, like how to ensure the quality of the software in short development cycles and how to give the customer an easy possibility to accept and release features.

The previously posted concepts about continuous integration and gated check-ins are a very important basis for delivering high quality software, of course. But they do not address the issues during delivery.

In the next posts I would like to write about the following important topics to establish a working continuous delivery process:

• Automated Deployments
• Automated UI Testing
• Feature Roll-Out