In the last blog post we looked how Azure DevOps Test Plans is used to structure testing and how we can run the regression testing. In this post we are going to look how Test Plans can be used to handle different test environments and how we can improve our test cases with parameters.
Configurations (Environments)
Configurations is a Test Plan feature that can be used to define different environments that we should run our tests in. For example we can define configurations for operating systems, browsers and devices. If we are developing IoT devices we can create configuration for every device type (or generation) we are currently supporting and the run test cases against all those configurations. This is simple way to duplicate test cases for different devices.
Configurations are split into two different things: Configuration variables and Test Configurations. The Configuration variable is for example an operating system or a browser. It is variable that will have multiple different values and the test configuration is a set of multiple configuration variables. In my example I’m having browser configuration variable with following values: Firefox, Chrome and Edge. I also have operating system with Windows 10, Windows 11 and Windows 8. Now I will combine these two test variables to create a test configuration: Operating system Windows 10 and browser Firefox, operating system Windows 10 and browser Edge etc. I don’t have to create configurations for all the combinations of variables, just the ones that I’m going to need.
After creating the configurations, we can assign them into test suites or test cases by right-clicking on top of test suite/case.
When the configuration is assigned into test suite, the Azure DevOps will duplicate all the test cases for selected configurations. In this example I have three different configurations set for test suite, so all the tests are tripled into test run (one time for each configuration).
We can also filter our test progress report with configurations to monitor how tests are succeeding in different configuration sets
Parameters
Parameters is a way to share information between test cases. For example if we have username and password that we are using in multiple different use cases (registration, logins etc.). We don’t want to write down those into every single test case, because if we need to change them, it is hard to go through all the test cases. We can use the parameters to store the username and password combinations and then use them in test cases.
Here I have created parameter set called Usernames that contains usernames, passwords and groupnames. I have two different users in this parameter set. Now when I’m writing the test case I can use the parameters with @Username, @Password and @Groupname notation. The Azure DevOps will recognize the syntax and link the parameter set into test case. We can see that there is “Pameter values” at the bottom that means we have successfully linked test case into parameter set.
Now when we go to execute this test the Azure DevOps will add iteration for each parameters. We had two rows in our example, so we will get two different iterations per linked test case.
Note that you can link only one parameter set per test case. If you need two, you have to combine the parameter sets into one and just use the parameters that you need from the set.
Summary
Configurations can be used to create different kind of test environments. It can be used to track testing against different devices or different browsers. We can easily filter test results based on their run configurations so we can get good overview of test progress per configuration.
Parameters are simple way to share information between multiple test cases. Parameter row will generate iteration per test case into test run where the configuration will duplicate the test case itself per assigned configurations.