Calculating Test Automation Complexity

In the past I have glossed over how to calculate complexity for an automated test, because in general it was not the area where people were the most confused or needed the most help. However after recently speaking on the practice of prioritizing automated tests at ConTEST NYC, I think that providing some more guidance is going to be useful for a lot of test engineers.

Continue reading “Calculating Test Automation Complexity”

Test Priority Spreadsheet

A have had a few requests for a test automation spreadsheet to use to help with assigning the business value and test complexity and calculating priority. This particular spreadsheet is very simple, but if there is any actual interest, I may provide a more complex and feature-full template.

Enjoy, but please use at your own risk .

Using Proxies in UI Testing

When testing web browser based software, I often find using a proxy to intercept  the browser calls is a powerful tool that helps me cover all my use cases.  It is also something that I sometimes get questions about. Usually I get questions about why to use proxies, and what proxy to use.

Why use a proxy?

A proxy routes all the calls from the browser (or mobile or desktop application) which gives you the ability to observe the calls, and to manipulate them. Continue reading “Using Proxies in UI Testing”

Code Review Criteria

Code Review Criteria

I have had the opportunity to review a lot of automation test code and to work with teams to begin peer reviewing each others code.  I felt it would be useful to codify and share some of the criteria I use.  And just to be clear, this criteria is what I use for the automated User Interface and Acceptance tests for web applications.  Many of this may apply to other test types or system, but no warrantees are expressed or implied if used outside of its intended use. Continue reading “Code Review Criteria”

Can’t is a 4 Letter Word

Can’t is a 4 Letter Word

What is the most offensive word that you know?  Would you use it at work with your colleagues or with your manager or team lead? Maybe you would under certain circumstances.  How do you react when you hear your colleagues using it around you?  I certainly don’t mind a certain amount of swearing, and use it myself from time to time.  But there is one word that I really can not stand.  And after having heard it used on me four times one day recently, I just about lost it.

Warning: Rant In Progress
Warning: Rant In Progress

So what offensive word am I referring to? The word is “can’t“.  That is right, can’t is one of the most offensive words that I know.  And using it with me may result in an unpleasant response.  Unfortunately I hear it way to often in a business setting. Continue reading “Can’t is a 4 Letter Word”

Test Automation: Calculating Business Value

In a previous article I discussed how to prioritize a test automation backlog.  An important part of prioritizing and tackling a test automation backlog is to identify or calculate the business value of the each of the tests.


Calculating the business value for an automated test is similar, but somewhat different, to calculating value for developing a product feature. The business value for building test automation for a particular feature is calculated based on four factors.  The first is the percentage of affected site visitors for the feature under test.  This initial value may then be modified for the financial impact of the feature.  Next the value may be further modified to account for the probability of failure.  The last factor to consider is the incremental impact for the feature.  These are each defined in more detail below. Continue reading “Test Automation: Calculating Business Value”

Prioritizing a Test Automation backlog

Starting a test automation strategy may seem daunting due to the very large number of features and use cases that will be identified for automation. Determining where to start and the order to tackle the backlog of hundreds or thousands of potential tests can be paralyzing.  I recommend three techniques to organize the test automation backlog.  First, should this test be automated at all?  Second, what is the business value of this test? Third, what is the difficulty or effort to implement this test?

Remove any test from the backlog that should not be automated.  Then use the business value and difficulty/effort metrics to prioritize the backlog. Start with the highest value, lowest effort tests first and schedule them to be worked on.  I would not worry about scheduling out more tests at this point.  When you have worked through the first list of scheduled tests, your should reevaluate your backlog again and schedule the highest value, lowest effort tests.  You will find that you will have added items, or that the business value or effort will have changed since your first reviewed the list, so reevaluation is important.

When you review your prioritized list and see items that have a higher effort then business value, these should not be worked on.  These would seem to be an example of tests that should not be automated at all.  After all, if the effort exceeds the value, then they would clearly be a poor choose for automation.  However I don’t suggest that you simply remove these from your unscheduled backlog.  The reason is that over time the effort to automate some of your more difficult tests will probably drop.  Your tools and frameworks, as well as your teams growing skills, will reduce the effort for many of these tests over time.  Also the effort may decrease as a side effect of implementing higher value tests.  Review these items along with the other items on your backlog periodically.

Calculating the effort to implement a test is something that most automation engineers will grasp fairly easily.  Depending on the team, this may be in story points, hours, to-shirt sizes, or some other technique.  However calculating business value and identifying tests that should not be automated (outside of business value and effort) is not so straightforward.  I will cover these in detail in future articles.

Next: Calculating Business Value

Controlling Sauce Connect

Programmatically controlling a Sauce Connect Tunnel

I recently decided that I wanted to create and control Sauce Connect tunnels from within my own test code and could not find any examples of how this could be done.  After some research and experimentation, I was successful and so I wanted to document the method that I am using in case anyone else finds a need to do the same.  Note that my test framework is written in Java and the method that I outline should work with any JVM language.  Modifying this to work with another setup is left as an exercise to the reader. Continue reading “Controlling Sauce Connect”

JavaScript parameter passing

How does JavaScript pass parameters into functions?  I have seen a lot of confusion and misunderstanding on this topic for several years.  Even among people that consider themselves JavaScript experts.  I have seen articles and posts that state Pass-By-Reference and I have seen others that state it sometimes uses Pass-By-Value or Pass-By-Reference depending on the argument type.  These are all wrong, so I thought I would take a stab at explaining exactly how it works and why it is so misunderstood.

To start with lets look at the several different methods used by various programming languages to handle parameter passing.  The two most common are Pass-By-Value and Pass-By-Reference.  There are some other less common methods such as Pass-By-Name that I have used, but we’ll skip over those since the are not applicable here.


In the Pass-By-Value method, parameter variables are copied onto the stack and then made available to the function.  Once the function returns, and the stack is popped, then the copies are gone.  This means that the the variables in the outer scope will never be modified, no matter what the function my do to them.  This provides a nice safety net to ensure that a function does not accidentally modify something that it shouldn’t.


In the Pass-By-Reference method, a reference (pointer) to the variable is created, placed on the stack, and made available to the function.  Once the function returns, and the stack is popped, then the reference is gone.  However  unlike in the Pass-By-Value method, any changes made to the referenced variable will be visible in the outer scope.  This can be more performant when dealing with large data structures since you do not have to create copies of the entire value.

JavaScript (aka ECMAScript)

So which method is used by JavaScript?  JavaScript always passes function arguments by value.  But I’m sure you have seen many cases where functions modify the objects in the outer scope.  This is why people often think that Pass-By-Reference is being used.  To see what is going on, lets look at a few examples.
Continue reading “JavaScript parameter passing”

Woman in Technology

I attended the CodeStock development conference in Knoxville recently.  One of the sessions that I attended was the Woman in Technology panel. Michael Neel was on the panel and he made an interesting comment.  He stated that while there were woman in technology companies, woman did not give keynote addresses at technology conferences. Continue reading “Woman in Technology”