Optimising ASP.NET Core apps in Docker - avoiding manually copying csproj files (Part 2)

This is a follow-up to my recent posts on building ASP.NET Core apps in Docker:

• Optimising ASP.NET Core apps in Docker - avoiding manually copying csproj files
• Building ASP.NET Core apps using Cake in Docker
• Exploring the .NET Core Docker files: dotnet vs aspnetcore vs aspnetcore-build

In this post I expand on a comment Aidan made on my last post:

Something that we do instead of the pre-build tarball step is the following, which relies on the pattern of naming the csproj the same as the directory it lives in. This appears to match the structure of your project, so it should work for you too.

I'll walk through the code he provides to show how it works, and how to use it to build a standard ASP.NET Core application with Docker. The technique in this post can be used instead of the tar-based approach from my previous post, as long as your solution conforms to some standard conventions.

I'll start by providing some background to why it's important to optimise the order of your Dockerfile, the options I've already covered, and the solution provided by Aidan in his comment.

Background - optimising your Dockerfile for dotnet restore

When building ASP.NET Core apps using Docker, it's important to consider the way Docker caches layers to build your app. I discussed this process in a previous post on building ASP.NET Core apps using Cake in Docker, so if that's new to you, i suggest checking it out.

A common way to take advantage of the build cache when building your ASP.NET Core app, is to copy across only the .csproj, .sln and nuget.config files for your app before doing dotnet restore, instead of copying the entire source code. The NuGet package restore can be one of the slowest parts of the build, and it only depends on these files. By copying them first, Docker can cache the result of the restore, so it doesn't need to run again if all you do is change a .cs file for example.

Due to the nature of Docker, there are many ways to achieve this, and I've discussed two of them previously, as summarised below.

Option 1 - Manually copying the files across

The easiest, and most obvious way to copy all the .csporj files from the Docker context into the image is to do it manually using the Docker COPY command. For example:

# Build image
FROM microsoft/aspnetcore-build:2.0.6-2.1.101 AS builder
WORKDIR /sln

COPY ./aspnetcore-in-docker.sln ./NuGet.config  ./
COPY ./src/AspNetCoreInDocker.Lib/AspNetCoreInDocker.Lib.csproj  ./src/AspNetCoreInDocker.Lib/AspNetCoreInDocker.Lib.csproj
COPY ./src/AspNetCoreInDocker.Web/AspNetCoreInDocker.Web.csproj  ./src/AspNetCoreInDocker.Web/AspNetCoreInDocker.Web.csproj
COPY ./test/AspNetCoreInDocker.Web.Tests/AspNetCoreInDocker.Web.Tests.csproj  ./test/AspNetCoreInDocker.Web.Tests/AspNetCoreInDocker.Web.Tests.csproj

RUN dotnet restore

Unfortunately, this has one major downside: You have to manually reference every .csproj (and .sln) file in the Dockerfile.

Ideally, you'd be able to do something like the following, but the wildcard expansion doesn't work like you might expect:

# Copy all csproj files (WARNING, this doesn't work!)
COPY ./**/*.csproj ./

That led to my alternative solution: creating a tar-ball of the .csproj files and expanding them inside the image.

Option 2 - Creating a tar-ball of the project files

In order to create a general solution, I settled on an approach that required scripting steps outside of the Dockerfile. For details, see my previous post, but in summary:

1. Create a tarball of the project files using

find . -name "*.csproj" -print0 \
    | tar -cvf projectfiles.tar --null -T -`

2. Expand the tarball in the Dockerfile

FROM microsoft/aspnetcore-build:2.0.6-2.1.101 AS builder
WORKDIR /sln

COPY ./aspnetcore-in-docker.sln ./NuGet.config  ./
COPY projectfiles.tar .
RUN tar -xvf projectfiles.tar

RUN dotnet restore

3. Delete the tarball once build is complete

rm projectfiles.tar

This process works, but it's messy. It involves running bash scripts both before and after docker build, which means you can't do things like build automatically using DockerHub. This brings us to the hybrid alternative, proposed by Aidan.

The new-improved solution

The alternative solution actually uses the wildcard technique I previously dismissed, but with some assumptions about your project structure, a two-stage approach, and a bit of clever bash-work to work around the wildcard limitations.

I'll start by presenting the complete solution, and I'll walk through and explain the steps later.

FROM microsoft/aspnetcore-build:2.0.6-2.1.101 AS builder
WORKDIR /sln

COPY ./*.sln ./NuGet.config  ./

# Copy the main source project files
COPY src/*/*.csproj ./
RUN for file in $(ls *.csproj); do mkdir -p src/${file%.*}/ && mv $file src/${file%.*}/; done

# Copy the test project files
COPY test/*/*.csproj ./
RUN for file in $(ls *.csproj); do mkdir -p test/${file%.*}/ && mv $file test/${file%.*}/; done

RUN dotnet restore

# Remainder of build process

This solution is much cleaner than my previous tar-based effort, as it doesn't require any external scripting, just standard docker COPY and RUN commands. It gets around the wildcard issue by copying across csproj files in the src directory first, moving them to their correct location, and then copying across the test project files.

This requires a project layout similar to the following, where your project files have the same name as their folders. For the Dockerfile in this post, it also requires your projects to all be located in either the src or test sub-directory:

Step-by-step breakdown of the new solution

Just to be thorough, I'll walk through each stage of the Dockerfile below.

1. Set the base image

The first steps of the Dockerfile are the same for all solutions: it sets the base image, and copies across the .sln and NuGet.config file.

FROM microsoft/aspnetcore-build:2.0.6-2.1.101 AS builder
WORKDIR /sln

COPY ./*.sln ./NuGet.config  ./

After this stage, your image will contain 2 files:

2. Copy src .csproj files to root

In the next step, we copy all the .csproj files from the src folder, and dump them in the root directory.

COPY src/*/*.csproj ./

The wildcard expands to match any .csproj files that are one directory down, in the src folder. After it runs, your image contains the following file structure:

3. Restore src folder hierarchy

The next stage is where the magic happens. We take the flat list of csproj files, and move them back to their correct location, nested inside sub-folders of src.

RUN for file in $(ls *.csproj); do mkdir -p src/${file%.*}/ && mv $file src/${file%.*}/; done

I'll break this command down, so we can see what it's doing

1. for file in $(ls *.csproj); do ...; done - List all the .csproj files in the root directory. Loop over them, and assign the file variable to the filename. In our case, the loop will run twice, once with AspNetCoreInDocker.Lib.csproj and once with AspNetCoreInDocker.Web.csproj.

2. ${file%.*} - use bash's string manipulation library to remove the extension from the filename, giving AspNetCoreInDocker.Lib and AspNetCoreInDocker.Web.

3. mkdir -p src/${file%.*}/ - Create the sub-folders based on the file names. the -p parameter ensures the src parent folder is created if it doesn't already exist.

4. mv $file src/${file%.*} - Move the csproj file into the newly created sub-folder.

After this stage executes, your image will contain a file system like the following:

4. Copy test .csproj files to root

Now the src folder is successfully copied, we can work on the test folder. The first step is to copy them all into the root directory again:

COPY test/*/*.csproj ./

Which gives a hierarchy like the following:

5. Restore test folder hierarchy

The final step is to restore the test folder as we did in step 3. We can use pretty much the same code as in step 3, but with src replaced by test:

RUN for file in $(ls *.csproj); do mkdir -p test/${file%.*}/ && mv $file test/${file%.*}/; done

After this stage we have our complete skeleton project, consisting of just our sln, NuGet.config, and .csproj files, all in their correct place.

That leaves us free to build and restore the project while taking advantage of Docker's layer-caching optimisations, without having to litter our Dockerfile with specific project names, or use outside scripting to create a tar-ball.

Summary

For performance purposes, it's important to take advantage of Docker's caching mechanisms when building your ASP.NET Core applications. Some of the biggest gains can be had by caching the restore phase of the build process.

In this post I showed an improved way to achieve this without having to resort to external scripting using tar, or having to list every .csproj file in your Dockerfile. This solution was based on a comment by Aidan on my previous post, so a big thanks to him!