Test Automation: Calculating Business Value

In a previous article I discussed how to prioritize a test automation backlog.  An important part of prioritizing and tackling a test automation backlog is to identify or calculate the business value of the each of the tests.

Overview

Calculating the business value for an automated test is similar, but somewhat different, to calculating value for developing a product feature. The business value for building test automation for a particular feature is calculated based on four factors.  The first is the percentage of affected site visitors for the feature under test.  This initial value may then be modified for the financial impact of the feature.  Next the value may be further modified to account for the probability of failure.  The last factor to consider is the incremental impact for the feature.  These are each defined in more detail below.

Definitions

  • Feature under test: A feature of a product which can be independently tested against functional requirements.  A product will be made of many different features.  In most cases, the features of a product are similar or the same as the individual product requirements.
    • Example: The Login feature of the Gigya community product.
    • Example: The Pin It feature on a photo gallery photo.
  • Use Case: A feature will typically have a number of different use cases.  This could be the feature used on different templates, as displayed for different users, or other variations.

Business Value factors

  1. Impacted Users: Set business value based on the number or percentage of site visitors that are affected by the feature under test.  If all or most site visitors will be affected by the feature then this would be high impact.  If a very small percentage of users are affected then it would be low.
    1. Example: Recipe reviews on Food Network appear on nearly every recipe and are used by many users viewing a recipe.  Display of recipe reviews would be a high impact feature.
    2. Example: Change profile photo appears on the user account setting page.  The percentage of site visitors that would typically use this feature would be extremely small making this a low impact feature.
  2. Financial Impact: If there are unusual financial impacts related to this feature it may modify the value.  The feature could be tied to a partnership which drives extra revenue, there could be legal obligations which could have direct or indirect financial consequences, or there could be unusually high impact to brand reputation.  Any of these may increase the automated test value for this feature.  There may also be features which have no ads and are not reachable from Search Engines (no SEO value) and therefore have a lower than normal financial value which would decrease the test automation business value.
    1. Example: The user registration process includes capturing user age to comply with COPPA requirements.  A failure of this feature could expose us to legal action and financial penalties (Up to $16,000 per instance).  This would increase the value of tests for this feature.
    2. Example: The home category photo galleries have an integration with WayFair.com.   A failure of this feature may impact the revenue from this partnership.  However the actual revenue from this partnership may be very minimal at present and so it may not affect the business value at all.
    3. Example: There may be a feature that involves a page that contains no ads and is not indexable by search engines.  In which case the financial impact may be lower than normal.
  3. Failure Probability: Some features have a history of failure or are known to depend on complex or fragile code or infrastructure which would create a greater risk of failure.  Of course the feature may also be considered to have a lower than normal risk of failure.  In either case this may adjust the overall business value up or down.
  4. Incremental impact: The business value for test automation for a feature can sometimes be affected by other features which already have test automation.  This may be due to an overlap in the features, but it more commonly impacts additional use cases for a single features.  When there are multiple use cases, the tests for the first use case may cover the underlying features and code used for several use cases.  This may somewhat reduce the incremental value of adding the additional use cases.
    1. Example: The Asset Title component can be used with many different temples. The business value for automated tests of the Asset Title on an article is high.  Extending the tests to cover Asset Title on Photo Gallery are low (but not zero) due to the fact that the component logic and behavior are the same between the two templates.  However the Asset Title component has some different behavior when used on a Episode template where it has different logic (related to the underlying show asset) in handling some of the default behavior.  So in this case it would have less value than the original use case on Article but more than adding to photo gallery.
    2. Example: The user login feature supports several different social networks for authentication (Facebook, Twitter, Google, Yahoo).  A test for one of these would cover much of the underlying technology and code used by all of them, however there are some parts that are also different.  Therefore the first social network test would have a higher value and each subsequent one would have less incremental value.  

Stakeholders

Calculating business value for automated tests may involve several individuals.  First, the product owner for the product in question is in the best position to know the details used in the calculation.  Secondly, the lead engineer for the product would have additional insights, such as code fragility (used in Failure Probability) which would be valuable as well as possibly being in possession of much of the information available to the product owner.  Finally the test automation engineers and the automation architect will have details about related tests or experience with calculations for other products.

Next: Identifying tests not to automate.

2 thoughts on “Test Automation: Calculating Business Value”

Leave a Reply

Your email address will not be published. Required fields are marked *