Code Review Criteria

I have had the opportunity to review a lot of automation test code and to work with teams to begin peer reviewing each others code.  I felt it would be useful to codify and share some of the criteria I use.  And just to be clear, this criteria is what I use for the automated User Interface and Acceptance tests for web applications.  Many of this may apply to other test types or system, but no warrantees are expressed or implied if used outside of its intended use. Continue reading “Code Review Criteria”

In a previous article I discussed how to prioritize a test automation backlog.  An important part of prioritizing and tackling a test automation backlog is to identify or calculate the business value of the each of the tests.

Overview

Calculating the business value for an automated test is similar, but somewhat different, to calculating value for developing a product feature. The business value for building test automation for a particular feature is calculated based on four factors.  The first is the percentage of affected site visitors for the feature under test.  This initial value may then be modified for the financial impact of the feature.  Next the value may be further modified to account for the probability of failure.  The last factor to consider is the incremental impact for the feature.  These are each defined in more detail below. Continue reading “Test Automation: Calculating Business Value”

Starting a test automation strategy may seem daunting due to the very large number of features and use cases that will be identified for automation. Determining where to start and the order to tackle the backlog of hundreds or thousands of potential tests can be paralyzing.  I recommend three techniques to organize the test automation backlog.  First, should this test be automated at all?  Second, what is the business value of this test? Third, what is the difficulty or effort to implement this test?

Remove any test from the backlog that should not be automated.  Then use the business value and difficulty/effort metrics to prioritize the backlog. Start with the highest value, lowest effort tests first and schedule them to be worked on.  I would not worry about scheduling out more tests at this point.  When you have worked through the first list of scheduled tests, your should reevaluate your backlog again and schedule the highest value, lowest effort tests.  You will find that you will have added items, or that the business value or effort will have changed since your first reviewed the list, so reevaluation is important.

When you review your prioritized list and see items that have a higher effort then business value, these should not be worked on.  These would seem to be an example of tests that should not be automated at all.  After all, if the effort exceeds the value, then they would clearly be a poor choose for automation.  However I don’t suggest that you simply remove these from your unscheduled backlog.  The reason is that over time the effort to automate some of your more difficult tests will probably drop.  Your tools and frameworks, as well as your teams growing skills, will reduce the effort for many of these tests over time.  Also the effort may decrease as a side effect of implementing higher value tests.  Review these items along with the other items on your backlog periodically.

Calculating the effort to implement a test is something that most automation engineers will grasp fairly easily.  Depending on the team, this may be in story points, hours, to-shirt sizes, or some other technique.  However calculating business value and identifying tests that should not be automated (outside of business value and effort) is not so straightforward.  I will cover these in detail in future articles.

Next: Calculating Business Value