Why isn't my ASP.NET Core environment-specific configuration loading?

I was recently standing up a new ASP.NET Core application running in Docker, and I was seeing some very strange behaviour. The application would start up without any problems when running locally on my Windows machine. But when I pushed it to the build server, the application would immediately fail, citing a "missing connection string" or something similar. I spent a good half an hour trying to figure out the issue, so this post is just in case someone else runs into the same problem!

In this post I'll cover the basic background of environments in ASP.NET Core, and describe how you would typically use environment-specific configuration. Finally, I'll describe the bug that I ran into and why it was an issue. If you just want to see the bug, feel fee to skip ahead.

tl;dr; IHostingEnvironment ignores the case of the current environment when you use the IsDevelopment() extension methods etc. However, if you are using environment-specific configuration files, appsettings.Development.json for example, then you must pay attention to case. Setting the environment to "development" instead of "Development" will result in your configuration files not loading on a case-sensitive OS like Linux.

ASP.NET Core environments

ASP.NET Core has the concept of environments, which represent the different locations your code might be running. You can determine the current environment at runtime, and use the value to change the behaviour of your app somehow. For example, in Startup.Configure(), it's common to configure your middleware pipeline differently if you're running in "Development" as opposed to "Production":

public void Configure(IApplicationBuilder app, IHostingEnvironment env)
{
    // Only added when running in Development
    if (env.IsDevelopment())
    {
        app.UseDeveloperExceptionPage();
    }

    // Only added when running in Production
    if (env.IsProduction())
    {
        app.UseExceptionHandler("/Error");
    }

    app.UseStaticFiles();
    app.UseMvc();
}

You can use IHostingEnvironment anywhere in your application where you want to check the current environment, and behave differently based on the value.

ASP.NET Core has knowledge of three environments by default, and provides extension methods for working with them:

• "Development" - Identified using IHostingEnvironment.IsDevelopment()
• "Staging" - Identified using IHostingEnvironment.IsStaging()
• "Production" - Identified using IHostingEnvironment.IsProduction()

You can also see the value of the current environment by reading IHostingEnvironment.EnvironmentName directly, but it's highly recommended you use one of the extension methods. The extension methods take care to make a case-insensitive comparison between the EnvironmentName and the expected string (e.g. "Development").

While you can litter your code with imperative checks of the environment, a generally cleaner approach is to use environment-specific configuration, which I'll describe shortly.

ASP.NET Core configuration primer

The configuration system in ASP.NET Core is built up of layers of configuration values, compiled from multiple sources. You can load values from JSON files, XML files, environment variables, or you can create a custom provider to load values from pretty much anywhere.

You can build a configuration object by adding providers to an IConfigurationBuilder object. This typically happens in Program.cs, using the IWebHostBuilder.ConfigureAppConfiguration method. WebHost.CreateDefaultBuilder() calls this method behind the scenes in a typical ASP.NET Core 2.x app. Each provider added to the IConfigurationBuilder adds another layer of configuration. For example, the following code adds a JSON file (appsettings.json) and environment variables to the final configuration object:

IHostingEnvironment env;
var builder = new ConfigurationBuilder()
    .SetBasePath(env.ContentRootPath) // the path where the json file should be loaded from
    .AddEnvironmentVariables();

The order of the configuration providers is important here; if any environment variable has the same name as a setting in the JSON file, it will overwrite the JSON setting. The final configuration will be a "flattened" view of the settings in all of the configuration sources.

I think of the flattening of configuration providers as similar to the flattening of layers in a Photoshop image. Each layer overwrites the values from the previous layers, except where it is transparent (i.e. where the layer doesn't have values).

For example, imagine you have the following appsettings.json configuration file;

{
    "Logging": {
        "LogLevel": {
            "Default": "Debug",
            "System": "Information",
            "Microsoft": "Information"
    }
}

On its own, that would generate the following settings:

"Logging:LogLevel:Default" = "Debug";
"Logging:LogLevel:System" = "Information";
"Logging:LogLevel:Microsoft" = "Information";

However, if you also had an environment variable,

Logging__LogLevel__Default=Warning

And loaded it after your JSON file, the final configuration would be the following (note the change in value for the first setting):

"Logging:LogLevel:Default" = "Warning";
"Logging:LogLevel:System" = "Information";
"Logging:LogLevel:Microsoft" = "Information";

Environment-specific configuration

The "flattening" of configuration providers is what allows you to have environment-specific configuration. Take the common case where you want to use a different setting in local development compared to production. There are a number of ways you could achieve this, for example:

• Overwrite default values e.g. only set an environment variable for the setting in Production.
• Use different configuration provider settings e.g. Load settings from Azure Key Vault in production, and User Secrets for local development.
• Load additional configuration providers e.g. load an additional environment-specific JSON file

Those last two points are essentially the same thing, but I wanted to call them out as different because they're typically used for two slightly different things, secrets vs. settings.

Secrets, such as API keys and connection strings shouldn't be stored inside your repository. For local development, sensitive values should be stored in User Secrets. In production, secrets should be retrieved from a provider such as Azure Key Vault.

In contrast, settings are not sensitive values, they just represent something you might want to configure differently between environments. For example, maybe you want to use more caching in production, or write log files to different locations.

As well as environment-specific configuration providers, it's possible to have environment-specific dependency injection configuration, and environment-specific Startup classes.

The typical WebHost.CreateDefaultBuilder() method uses all three approaches: overwriting, different providers, and additional providers. The configuration method for the default builder is shown below:

ConfigureAppConfiguration((hostingContext, config) =>
{
    var env = hostingContext.HostingEnvironment;

    config.AddJsonFile("appsettings.json", optional: true, reloadOnChange: true)
          .AddJsonFile($"appsettings.{env.EnvironmentName}.json", optional: true, reloadOnChange: true); // optional extra provider

    if (env.IsDevelopment()) // different providers in dev
    {
        var appAssembly = Assembly.Load(new AssemblyName(env.ApplicationName));
        if (appAssembly != null)
        {
            config.AddUserSecrets(appAssembly, optional: true);
        }
    }

    config.AddEnvironmentVariables(); // overwrites previous values

    if (args != null)
    {
        config.AddCommandLine(args);
    }
});

The default builder configures up to 5 configuration providers by default:

• A JSON file called appsettings.json
• An environment-specific JSON file called appsettings.ENVIRONMENT.json where ENVIRONMENT is the name of the current environment
• User Secrets, if in the Development environment
• Environment variables
• Command line arguments (if any arguments were passed)

For the rest of this post I'm going to focus on the environment-specific JSON file, as that's what caused the issue I encountered.

The problem: environment-specific configuration not loading

As part of a new .NET Core app I was building, I was running a "smoke test" on the Docker container produced, as described in my last post. This involves running the Docker container on the build server, and checking that the container starts up correctly. The idea is to double check that the initial configuration that occurs on app start up is correct. One such check is that any strongly typed settings validation runs successfully.

This approach of starting the app as part of the CI process won't necessarily be a good (or useful) approach for all apps. I added it to the CI process of several apps as an incredibly basic "sanity check" but I'm not 100% convinced by my approach yet!

When I ran the smoke test for the first time in a new app, the settings validation for a third-party API URL failed. This was very odd, as I had tested the application locally. When running smoke tests, I typically set the Hosting Environment of the app to Development, (or sometimes a testing-specific environment, Testing). Inside the appsettings.Development.json file, I could see the offending configuration value:

{
    "ThirdPartyApi": {
        "BaseUrl": "https://test.example.com"
    }
}

But for some reason, when the application was running in Docker for the smoke tests, the value wasn't being bound correctly. In the next section, I'll briefly describe some of the things I thought of and looked into.

Troubleshooting

I tried debugging locally, adding and removing the file, and changing the setting value. I was trying to confirm that the file was definitely being loaded correctly, and the setting wasn't coming from somewhere else when running locally. Everything was correct.

I checked that there were no unexpected environment variables overwriting the value when the app was running in Docker for the smoke test. There weren't.

I looked inside the Docker container itself, and double checked that the appsettings.Development.json file existed, and was in the right place. Everything looked OK.

Finally, I checked that I was actually running in the environment I expected - Development. Looking at the logs from the container when the smoke test ran I could see that the Hosting environment was correct according to the app:

Hosting environment: Development
Content root path: /app/
Now listening on: https://localhost:80
Application started. Press Ctrl+C to shut down.

At this point, I was somewhat stumped, I had run out of ideas. I made a coffee.

When I sat down and opened the smoke test script file, the answer hit me immediately…

Linux file-system case-sensitivity

The smoke test script I was using is very similar to the script from my last post. The command I was using to run my new app for the smoke test is shown below:

docker run -d --rm \
    --name $SMOKE_TEST_IMAGE \
    -e ASPNETCORE_ENVIRONMENT=development \
    $APP_IMAGE

The problem is the statement where I set the environment variable to define the hosting environment using

ASPNETCORE_ENVIRONMENT=development

This sets the environment to development which is not the same as Development. ASP.NET Core itself is careful to not differentiate between environments based on case - the IHostingEnvironment extension methods like IsDevelopment() are all case insensitive. As long as you use these extension methods and don't use IHostingEnvironment.EnvironmentName directly, you'll be fine.

However, the one place where it's very common to use EnvironmentName directly is in your app configuration. Earlier I described the common approach to environment-specific configuration: adding an extra appsettings.json file:

var env = hostingContext.HostingEnvironment;

config.AddJsonFile("appsettings.json")
      .AddJsonFile($"appsettings.{env.EnvironmentName}.json");

As you can see, we're directly using EnvironmentName to calculate the environment-specific JSON configuration file. In my smoke test script, EnvironmentName="development", so the app was looking for the appsettings.development.json file. The file was actually called appsettings.Development.json.

On Windows, this case difference doesn't matter - ASP.NET Core respects the conventions of the host OS, so it loads the file. Even if you set the environment to DeVelOpMeNt, you'd be fine. Linux, however, is case sensitive, so it won't find the file.

The simple fix was to set the environment with the standard title-casing:

docker run -d --rm \
    --name $SMOKE_TEST_IMAGE \
    -e ASPNETCORE_ENVIRONMENT=Development \
    $APP_IMAGE

With that small change, the app was able to start, and the smoke test succeeded.

Summary

Always be consistent with your environment names. The case may not matter if you're running on Windows, but it definitely will if your app is ever run on Linux. The ASP.NET Core framework itself is careful to ignore case when determining the current environment, but you can't trust the underlying operating system to do the same!