In the past I have glossed over how to calculate complexity for an automated test, because in general it was not the area where people were the most confused or needed the most help. However after recently speaking on the practice of prioritizing automated tests at ConTEST NYC, I think that providing some more guidance is going to be useful for a lot of test engineers.
Continue reading “Calculating Test Automation Complexity”Category: Development
Code Review Criteria
Code Review Criteria
I have had the opportunity to review a lot of automation test code and to work with teams to begin peer reviewing each others code. I felt it would be useful to codify and share some of the criteria I use. And just to be clear, this criteria is what I use for the automated User Interface and Acceptance tests for web applications. Many of this may apply to other test types or system, but no warrantees are expressed or implied if used outside of its intended use. Continue reading “Code Review Criteria”
Test Automation: Calculating Business Value
In a previous article I discussed how to prioritize a test automation backlog. An important part of prioritizing and tackling a test automation backlog is to identify or calculate the business value of the each of the tests.
Overview
Calculating the business value for an automated test is similar, but somewhat different, to calculating value for developing a product feature. The business value for building test automation for a particular feature is calculated based on four factors. The first is the percentage of affected site visitors for the feature under test. This initial value may then be modified for the financial impact of the feature. Next the value may be further modified to account for the probability of failure. The last factor to consider is the incremental impact for the feature. These are each defined in more detail below. Continue reading “Test Automation: Calculating Business Value”
Prioritizing a Test Automation backlog
Starting a test automation strategy may seem daunting due to the very large number of features and use cases that will be identified for automation. Determining where to start and the order to tackle the backlog of hundreds or thousands of potential tests can be paralyzing. I recommend three techniques to organize the test automation backlog. First, should this test be automated at all? Second, what is the business value of this test? Third, what is the difficulty or effort to implement this test?
Remove any test from the backlog that should not be automated. Then use the business value and difficulty/effort metrics to prioritize the backlog. Start with the highest value, lowest effort tests first and schedule them to be worked on. I would not worry about scheduling out more tests at this point. When you have worked through the first list of scheduled tests, your should reevaluate your backlog again and schedule the highest value, lowest effort tests. You will find that you will have added items, or that the business value or effort will have changed since your first reviewed the list, so reevaluation is important.
When you review your prioritized list and see items that have a higher effort then business value, these should not be worked on. These would seem to be an example of tests that should not be automated at all. After all, if the effort exceeds the value, then they would clearly be a poor choose for automation. However I don’t suggest that you simply remove these from your unscheduled backlog. The reason is that over time the effort to automate some of your more difficult tests will probably drop. Your tools and frameworks, as well as your teams growing skills, will reduce the effort for many of these tests over time. Also the effort may decrease as a side effect of implementing higher value tests. Review these items along with the other items on your backlog periodically.
Calculating the effort to implement a test is something that most automation engineers will grasp fairly easily. Depending on the team, this may be in story points, hours, to-shirt sizes, or some other technique. However calculating business value and identifying tests that should not be automated (outside of business value and effort) is not so straightforward. I will cover these in detail in future articles.
Controlling Sauce Connect
Programmatically controlling a Sauce Connect Tunnel
I recently decided that I wanted to create and control Sauce Connect tunnels from within my own test code and could not find any examples of how this could be done. After some research and experimentation, I was successful and so I wanted to document the method that I am using in case anyone else finds a need to do the same. Note that my test framework is written in Java and the method that I outline should work with any JVM language. Modifying this to work with another setup is left as an exercise to the reader. Continue reading “Controlling Sauce Connect”
JavaScript parameter passing
How does JavaScript pass parameters into functions? I have seen a lot of confusion and misunderstanding on this topic for several years. Even among people that consider themselves JavaScript experts. I have seen articles and posts that state Pass-By-Reference and I have seen others that state it sometimes uses Pass-By-Value or Pass-By-Reference depending on the argument type. These are all wrong, so I thought I would take a stab at explaining exactly how it works and why it is so misunderstood.
To start with lets look at the several different methods used by various programming languages to handle parameter passing. The two most common are Pass-By-Value and Pass-By-Reference. There are some other less common methods such as Pass-By-Name that I have used, but we’ll skip over those since the are not applicable here.
Pass-By-Value
In the Pass-By-Value method, parameter variables are copied onto the stack and then made available to the function. Once the function returns, and the stack is popped, then the copies are gone. This means that the the variables in the outer scope will never be modified, no matter what the function my do to them. This provides a nice safety net to ensure that a function does not accidentally modify something that it shouldn’t.
Pass-By-Reference
In the Pass-By-Reference method, a reference (pointer) to the variable is created, placed on the stack, and made available to the function. Once the function returns, and the stack is popped, then the reference is gone. However unlike in the Pass-By-Value method, any changes made to the referenced variable will be visible in the outer scope. This can be more performant when dealing with large data structures since you do not have to create copies of the entire value.
JavaScript (aka ECMAScript)
So which method is used by JavaScript? JavaScript always passes function arguments by value. But I’m sure you have seen many cases where functions modify the objects in the outer scope. This is why people often think that Pass-By-Reference is being used. To see what is going on, lets look at a few examples.
Continue reading “JavaScript parameter passing”
Woman in Technology
I attended the CodeStock development conference in Knoxville recently. One of the sessions that I attended was the Woman in Technology panel. Michael Neel was on the panel and he made an interesting comment. He stated that while there were woman in technology companies, woman did not give keynote addresses at technology conferences. Continue reading “Woman in Technology”
CodeStock 2014
I have been attending CodeStock 2014 here in Knoxville the last few days with a group of my software and quality assurance engineers. Several were even speakers this year! Thanks to the conference organizers for putting together another great conference this year! Continue reading “CodeStock 2014”
Sticky Sessions are Evil
Sticky sessions are evil. Ok, I’ve said it. Actually I have said it before. In fact I am surprised at how often I have had to say it. They seem to get used with surprising frequency without properly thinking through the repercussions. I have been asked to explain the issues several times, so let me go walk through and document them. Continue reading “Sticky Sessions are Evil”
Bad Proxy
A fairly bizarre issue come up recently that I thought would be amusing to share. It all started fairly simply. One of the development teams needed to test some website changes that required overriding some of the website files without making changes to the website itself. This is not difficult for a developer to do, but they wanted to set it up more simply for a number of testers to use. I suggested setting up a proxy server and configure it to proxy most requests on to their normal location. Then send requests for the files they wanted to test on to the test location. We would only need it for a week or two during the test phase.
I had a little free time, so I decided to set it up for them. I considered setting up an apache instance using mod_proxy and mod_rewrite. I decided instead to write something custom in Node.js. It took me about an hour to write a custom proxy server and add rules to detect requests for the test files and rewrite the request before proxying it. I turned it on, tested by changing my local hosts file to direct website requests to my proxy server, and all was good. Continue reading “Bad Proxy”