In this post I show how you can run tests using the xUnit .NET CLI Tool dotnet xunit when building projects using Cake. Cake includes first class support for running test using dotnet test via the DotNetCoreTest alias, but if you want access to the additional configuration provided by the dotnet-xunit tool, you'll currently need to run the tool using DotNetCoreTool instead.

dotnet test vs dotnet xunit

Typically, .NET Core unit tests are run using the dotnet test command. This runs unit tests for a project regardless of which unit test framework was used - MSTest, NUnit, or xUnit. As long as the test framework has an appropriate adapter, the dotnet test command will hook into it, and provide a standard set of features.

However, the suggested approach to run .NET Core tests with xUnit is to use the dotnet-xunit framework tool. This provides more advanced access to xUnit settings, so you can set a whole variety of properties like how test names are listed, whether diagnostics are enabled, or parallelisation options for the test runs.

You can install the dotnet-xunit tool into a project by adding a DotnNetCliToolReference element to your .csproj file. If you add the xunit.runner.visualstudio and Microsoft.NET.Test.Sdk packages too, then you'll still be able to run your tests using dotnet test and Visual Studio:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFrameworks>netcoreapp2.0</TargetFrameworks>
    <IsPackable>false</IsPackable>
  </PropertyGroup>

  <ItemGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="15.3.0" />
    <PackageReference Include="xunit" Version="2.3.0" />
    <PackageReference Include="xunit.runner.visualstudio" Version="2.3.0" />
    <DotNetCliToolReference Include="dotnet-xunit" Version="2.3.0" />
  </ItemGroup>

</Project>

With these packages installed and restored, you'll be able to run your tests using either dotnet test or dotnet xunit, but if you use the latter option, you'll have a whole host of additional arguments available to you. To see all your options, run dotnet xunit --help. The following is just a small selection of the help screen:

> dotnet xunit --help
xUnit.net .NET CLI Console Runner
Copyright (C) .NET Foundation.

usage: dotnet xunit [configFile] [options] [reporter] [resultFormat filename [...]]

Note: Configuration files must end in .json (for JSON) or .config (for XML)
      XML configuration files are only supported on net4x frameworks

Valid options (all frameworks):
  -framework name        : set the framework (default: all targeted frameworks)
  -configuration name    : set the build configuration (default: 'Debug')
  -nobuild               : do not build the test assembly before running
  -nologo                : do not show the copyright message
  -nocolor               : do not output results with colors
  -failskips             : convert skipped tests into failures
  -stoponfail            : stop on first test failure
  -parallel option       : set parallelization based on option
                         :   none        - turn off parallelization
                         :   collections - parallelize test collections
  -maxthreads count      : maximum thread count for collection parallelization
                         :   default   - run with default (1 thread per CPU thread)
                         :   unlimited - run with unbounded thread count
                         :   (number)  - limit task thread pool size to 'count'
  -wait                  : wait for input after completion
  -diagnostics           : enable diagnostics messages for all test assemblies
[Output truncated]

Running tests with Cake

Cake is my preferred build scripting system for .NET Core projects. In their own words:

Cake (C# Make) is a cross platform build automation system with a C# DSL to do things like compiling code, copy files/folders, running unit tests, compress files and build NuGet packages.

I use cake to build my open source projects. A very simple Cake script consisting of a restore, build, and test, might look something like the following:

var configuration = Argument("Configuration", "Release");
var solution = "./test.sln";

// Run dotnet restore to restore all package references.
Task("Restore")
    .Does(() =>
    {
        DotNetCoreRestore();
    });

Task("Build")
    .IsDependentOn("Restore")
    .Does(() =>
    {
        DotNetCoreBuild(solution
           new DotNetCoreBuildSettings()
                {
                    Configuration = configuration
                });
    });

Task("Test")
    .IsDependentOn("Build")
    .Does(() =>
    {
        var projects = GetFiles("./test/**/*.csproj");
        foreach(var project in projects)
        {
            DotNetCoreTest(
                project.FullPath,
                new DotNetCoreTestSettings()
                {
                    Configuration = configuration,
                    NoBuild = true
                });
        }
    });

This will run a restore in the solution directory, build the solution, and then run dotnet test for every project in the test sub-directory. Each of the steps uses a C# alias which calls the dotnet SDK commands:

• DotNetCoreRestore - restores NuGet packages using dotnet restore
• DotNetCoreBuild - builds the solution using dotnet build, using the settings provided in the DotNetCoreBuildSettings object
• DotNetCoreTest - runs the tests in the project using dotnet test and the settings provided in the DotNetCoreTestSettings object.

Customizing the arguments passed to the dotnet tool

In the previous example, we ran tests with dotnet test and were able to set a number of additional options using the strongly typed DotNetCoreTestSettings object. If you want to pass additional options down to the dotnet test call, you can add customise the arguments using the ArgumentCustomization property.

For example, dotnet build implicitly calls dotnet restore, even though we are specifically restoring the solution. You can forgo this second call by passing --no-restore to the dotnet build call using the ArgumentCustomization property:

Task("Build")
    .IsDependentOn("Restore")
    .Does(() =>
    {
        DotNetCoreBuild(solution
           new DotNetCoreBuildSettings()
                {
                    Configuration = configuration,
                    ArgumentCustomization = args => args.Append($"--no-restore")
                });
    });

With this approach, you can customise the arguments that are passed to the dotnet test command. However you can't customise the command itself to call dotnet xunit. For that, you need a different Cake alias - DotNetCoreTool

Running dotnet xunit tests with Cake

Using the strongly typed *Settings objects makes invoking many of the dotnet tools easy with Cake, but it doesn't include first-class support for all of them. The dotnet-xunit tool is one such tool.

If you need to run a dotnet tool that's not directly supported by a Cake alias, you can use the general purpose DotNetCoreTool alias. You can use this to execute any dotnet tool, by providing the tool name, and the command arguments.

For example, imagine you want to run dotnet xunit with diagnostics enabled, and stop on the first failure. If you were running the tool directly from the command line you'd use:

dotnet xunit -diagnostics -stoponfail

In Cake, we can use the DotnetCoreTool, and pass in the command line arguments manually. If we update the previous Cake "Test" target to use DotNetCoreTool we have:

Task("Test")
    .IsDependentOn("Build")
    .Does(() =>
    {
        var projects = GetFiles("./test/**/*.csproj");
        foreach(var project in projects)
        {
            DotNetCoreTool(
                projectPath: project.FullPath, 
                command: "xunit", 
                arguments: $"-configuration {configuration} -diagnostics -stoponfail"
            );
        }
    });

The DotNetCoreTool isn't as convenient as being able to set strongly typed properties on a dedicated settings object. But it does at give a great deal of flexibility, and effectively lets you drop down to the command line inside your Cake build script.

If you build your scripts using Cake, and need/want to use some of the extra features afforded by dotnet xunit then DotNetCoreTool is currently the best approach, but it shouldn't be hard to create a wrapper alias for dotnet xunit that makes these arguments strongly typed. Assuming noone else has already done it and made this post obsolete, then I'll look at sending a PR as soon as I find the time!