I have been recently contacted a couple of times about same thing: How to run .NET tests in release pipeline? The answer isn’t actually that simple. It requires some decision making and pipeline workout to get it running.
In this post I’m going to walk through how to run XUnit (or NUnit/MSTest) tests in release pipeline and report test results so, that they are shown at Tests tab after finished release run.
The build pipeline
In this example we are using simple ASP.NET Core project and some .NET 6.0 Xunit tests. In our build pipeline we have a build task for the API solution (app in this sample), test task to run unit tests, publish for API project and publish for test project. To avoid building the test project in release pipeline too, we need to publish the test project dll files with build artifacts. To do that we can use .NET Core task with publish command. Uncheck the “Publish web projects“, change path of projects to target our test project (SampleTestProject.csproj) and add arguments to use the given build configuration.
So in the build pipeline we are publishing two different projects as zip files. The SampleWebApi.zip contains our API application as published web project and the SampleTestProject.zip contains test project published as zip package. Test project is actually quite big when published, but it is necessary to do to avoid building the project second time at release pipeline.
Another solution could be just to add code repository as artifacts to our release pipeline and then build the tests there.
Release pipeline
The release pipeline takes artifacts from build pipeline as “input”. In normal case we would just deploy those files into Azure or something like that, but in this example we are going to use them also to run tests before deployment.
We have two different jobs in our pipeline. The “Run Tests Before Deployment” and “Run Deployment“. In this sample the Run Deployment is just simple echo hello world. In Run Tests Before Deployment agent job we need to have three different tasks:
- Extract test dll files from SampleTestProject.zip, which are created by the build pipeline publish task
- Run XUnit tests with Visual Studio test runner
- Publish Test Results to Azure DevOps for test report.
Extract test binaries
To extract the zip file we can use “Extract files” task. Task requires path to target zip file and the destination folder. To avoid absolute paths we can use **/SampleTestProject.zip syntax to command the Azure DevOps to search for the zip file from any subfolder.
For destination folder we are using $(Agent.TempDirectory), that is basically a good folder to store test binaries temporary.
Running the tests
To run the tests, we are using “Visual Studio Test” task. This task requires “Test files” parameter. We can again use the **/SampleTestProject.dll syntax to look for any subdirectories for SampleTestProject.dll file.
For search folder we will give the release agent temp folder, because we unzipped the test binaries there. By default the runner will publish test result file into $(Agent.TempDirectory)\TestResults folder and that is fine to us.
Publish test results
To show test results in Azure DevOps, we need to run the “Publish Test Results” task. In this sample we have used XUnit, so we need to also set “Test result format” into XUnit.
Test results files can be set to default (**/*.trx), but for search folder we will define the location of our test result folder, which was $(Agent.TempDirectory)\TestResults. Everything else can be left to default.
Test results
When the release pipeline is finished, we can look for test results from release logs. From there we can see that our only test (Test1) has successfully finished. Hooray! There are no test failures.
Summary
To run XUnit/NUnit/MSTests at release pipeline, we have to first decide to use the already build artifacts, or to build them in the pipeline. If we want to use already build binaries, we need to publish them as part of build artifacts and then use the artifacts in release pipeline. This sample solution works also for .NET Framework projects, just change the dotnet publish task into good old publish artifacts task and ship the test binaries with it.
Remember that we have now two artifacts at drop.zip, which means that we may need to define the target zip file in deployment phase. If we have only one zip file, we don’t have to do it, the Azure DevOps will pick the only that is in the drop.zip.