In my last post, I showed how to add secrets to AWS Secrets Manager, and how you could configure your ASP.NET Core application to load them into the .NET Core configuration system at runtime. In this post, I take the process a little further. Instead of loading all secrets into the app, we'll only load those secrets which have a required prefix. This will allows us to have different secrets for different environments and for different apps.
A quick recap on loading secrets from AWS Secrets Manager.
I'm not going to cover the background of why we need secure secrets management, or how to add secrets to AWS Secrets Manager - check out my previous post for those details. At the end of my previous post, I showed how to add the Kralizek.Extensions.Configuration.AWSSecretsManager NuGet package to your app, and how to call AddSecretsManager() in Program.cs. I also showed how to conditionally load the package depending on the hosting environment, and how to replace "__" tokens in the secret name with ":".
This was the code we ended up with:
public class Program
{
public static void Main(string[] args) => BuildWebHost(args).Run();
public static IWebHost BuildWebHost(string[] args) =>
WebHost.CreateDefaultBuilder(args)
.ConfigureAppConfiguration((hostingContext, config) =>
{
// Don't add AWS secrets when running locally
if(!hostingContext.HostingEnvironment.IsDevelopment())
{
config.AddSecretsManager(configurator: ops =>
{
// Replace __ tokens in the configuration key name
opts.KeyGenerator = (secret, name) => name.Replace("__", ":");
});
}
})
.UseStartup<Startup>()
.Build();
}
This worked well for my example, but it falls down a bit in practical usage. There's two main issues here:
- All apps share the same secrets. We load all secrets we have access to and add them all to the
IConfigurationobject. That means currently all apps need to have the sameIConfiguration/appsettings.json schema, and use the same values. - We have no way of differentiating configuration specific to different environments such as testing/staging/production. We would need to have separate sections for each environment, which would quickly get messy and is not recommended. The idiomatic approach in ASP.NET Core is for every hosting environments to have the same configuration schema, but different values. This is easily achieved with configuration layering, but is not possible with our current setup.
The solution to both of these problems in our case, is to add a "prefix" to the secret name stored in AWS. We can use that prefix to filter the secrets we load both by environment and by application.
Note - It you're using Azure Key Vault, the guidance is to not use this approach. Instead, create a separate Key Vault for each app and each environment. Unfortunately, AWS Secrets Manager is currently "global" to an account (and region) so we can't use that approach.
Filtering the secrets loaded from AWS
Before I dive into the solution, I'll take a brief detour to describe how the Kralizek.Extensions.Configuration.AWSSecretsManager package loads your secrets from AWS. When you call AddSecretsManager(), you add a SecretsManagerConfigurationProvider to the list of IConfigurationProviders for your app. This doesn't immediately load the secrets from AWS, it just registers the provider with the IConfigurationBuilder.
When you call IConfigurationBuilder.Build() (or it's called implicitly as part of the standard ASP.NET Core bootstrapping), the provider calls LoadAsync(). An (abbreviated) annotated version of which is shown below:
public class SecretsManagerConfigurationProvider : ConfigurationProvider
{
public SecretsManagerConfigurationProviderOptions Options { get; }
async Task LoadAsync()
{
// Fetch ALL secrets from AWS Secrets Manager
// This does not include the secret value
var allSecrets = await FetchAllSecretsAsync().ConfigureAwait(false);
foreach (var secret in allSecrets)
{
// If we should not load the secret, skip it
if (!Options.SecretFilter(secret)) continue;
// Fetch the secret value from AWS
var secretValue = await Client.GetSecretValueAsync(
new GetSecretValueRequest { SecretId = secret.ARN }).ConfigureAwait(false);
// generate the key
var key = Options.KeyGenerator(secret, secret.Name);
// Save the value in the `IConfiguration` object
Set(key, secretValue);
}
}
}
The key feature we're after here is the SecretFilter() on the SecretsManagerConfigurationProviderOptions class as this lets us filter out which secrets are added to our IConfiguration.
public class SecretsManagerConfigurationProviderOptions
{
public Func<SecretListEntry, bool> SecretFilter { get; set; } = secret => true;
public Func<SecretListEntry, string, string> KeyGenerator { get; set; } = (secret, key) => key;
}
It's important to realise that we list all the secrets available first, so you need to ensure your AWS IAM role has permission to List all secrets, as well as Fetch specific values.
In order to solve our environment/app clashing problems we can build an appropriate predicate to filter our secrets by environment and application name. We'll also need to update the Key Generator, as you'll see later.
Deciding how to filter secrets
From the previous code, you can see the SecretFilter() predicate takes a single SecretListEntry object, and returns true if we should load the secret. That gives you a lot of different options for filtering your secrets.
For example, you could provide a hard list of ARNs that should be loaded, ensuring a very strict list of secrets to load. (ARNs are the unique resource identifiers in AWS, something like arn:aws:secretsmanager:eu-west-1:30123456:secret:ConnectionStrings__MyTestApp-abc123). Alternatively, you could add Tags to your secrets.
Specifying the exact secret name seemed attractive initially for keeping the secrets as locked down as possible. But having to maintain a specific list of ARNs for every environment in each app seemed like too much of a maintenance burden.
Instead I decided to go with a consistent naming convention for secrets based on the environment and a concept of variable "groups", inspired by Octopus Deploy's Variable Sets.
{EnvironmentName}/{SecretGroup}/{ConfigurationKey}
So for example, the connection string for MyTestApp in the Staging environment might have the following key name:
Staging/MyTestApp/ConnectionStrings__MyTestApp
This lets us do two things:
- Filter secrets based on the hosting environment
- Filter secrets based on the "secret groups" an app needs.
A given app will probably not have access to many different groups. Each app would have it's own group, for secrets specific to that app. It might also require access to one or more "shared" groups that contain global settings. That means you don't have to duplicate the same API Keys into app-specific secret groups, for example. The app can instead depend on the shared "Segment", "Twilio", or "Cloudflare" keys as necessary.
Partially building configuration to access stored values
By adding the flexibility to load multiple different secrets groups, we've somewhat inadvertently added some configuration to our app that we need in order to load the secrets. Another chicken and egg problem! Luckily this configuration isn't sensitive so we can store the configuration in appsettings.json.
Whenever possible I like to use strongly typed configuration, so we'll create a simple POCO for this configuration:
public class AwsSecretsManagerSettings
{
/// <summary>
/// The allowed secret groups, e.g. Shared or MyAppsSecrets
/// </summary>
public ICollection<string> SecretGroups { get; } = new List<string>();
}
Now we can add our configuration to appsettings.json:
{
"AwsSecretsManagerSettings": {
"SecretGroups": [
"shared",
"my-test-app"
]
}
}
Ordinarily, you would bind this configuration to the AwsSecretsManagerSettings object by using the Options pattern, calling Configure<T> in Startup.ConfigureServices(). Unfortunately that's not going to work in this case.
We need to access the configuration values before we start doing dependency injection, and before the Startup class is instantiated. The only way to make that work is to partially build the IConfiguration object for the app, and to manually bind our settings to it.
The following diagram shows the "partial build" approach (and is adapted from my book, ASP.NET Core in Action):
The code for this would look something like the following:
public class Program
{
public static void Main(string[] args) => BuildWebHost(args).Run();
public static IWebHost BuildWebHost(string[] args) =>
WebHost.CreateDefaultBuilder(args)
.ConfigureAppConfiguration((hostingContext, configBuilder) =>
{
// Partially build the IConfigurationBuilder. It won't contain
// out AWS secrets, but that's ok, we're going to throw it away soon
IConfiguration partialConfig = configBuilder.Build();
// Create a new instance of the settings object, and bind our configuration to it
var settings = new AwsSecretsManagerSettings();
partialConfig
.GetSection(nameof(AwsSecretsManagerSettings))
.Bind(settings);
// Build the list of allowed prefixes
var env = hostingContext.HostingEnvironment.EnvironmentName;
var allowedPrefixes = settings.SecretGroups
.Select(grp => $"{hostingContext.Host}/{grp}/");
// TODO: Use allowedPrefixes and add AWS secrets
// configBuilder.AddSecretsManager();
})
.UseStartup<Startup>()
.Build();
}
This allows us to access the partial configuration we need to populate our AwsSecretsManagerSettings object and generate the list of secret prefixes to load from AWS. All that's left is to use those prefixes to set the SecretFilter() predicate, and add the SecretsManagerConfigurationProvider to the configuration builder.
Note that the framework will call
IConfigurationBuilder.Build()a second time, when building the finalIConfigurationobject. By that point, we will have added theSecretsManagerConfigurationProviderby callingAddSecretsManager(), so the configuration will contain our AWS secrets.
Creating the filter methods
The filter methods themselves are pretty basic. First the SecretsManagerConfigurationProvider loads the list of available secrets. For each one, it calls the SecretFilter() predicate to decide whether to load the secret's value. Given our list of allowedPrefixes, we can write a predicate that looks something like this:
ICollection<string> allowedPrefixes; // Loaded from configuration
builder.AddSecretsManager(configurator: opts =>
{
// For a given entry, if it's name starts with any of the allowed prefixes
// then load the secret
opts.SecretFilter = entry => allowedPrefixes.Any(prefix => entry.Name.StartsWith(prefix));
opts.KeyGenerator = // TODO:
});
Once the provider has established that it should load a secret's value (because SecretFilter() returned true), and has called AWS to fetch the secret, it then calls our KeyGenerator function. In my last post, this was a simple string.Replace() to swap the "__" tokens for ":". With our "prefix" approach, we need to strip the prefix off first:
ICollectionstring allowedPrefixes; // Loaded from configuration
builder.AddSecretsManager(configurator: opts =>
{
opts.SecretFilter = entry => allowedPrefixes.Any(prefix => entry.Name.StartsWith(prefix));
opts.KeyGenerator = entry =>
{
// We know one of the prefixes matches, this assumes there's only one match,
// So don't use '/' in your environment or secretgroup names!
var prefix = allowedPrefixes.First(text.StartsWith);
// Strip the prefix, and replace "__" with ":"
return secretValue
.Substring(prefix.Length)
.Replace("__", ":");
}
});
We're almost there now, we just need to put all the pieces together.
Putting it all together and extracting into an extension method
We're starting to put quite a lot of code together here, most of which is boilerplate plumbing. In these situations I like to extract the code into an extension method instead of bloating the program.cs file. The code below is the same as shown throughout this post, just extracted into extension methods, and using static functions instead of anonymous delegates. For convenience I've actually created two extension methods:
- An extension method on
IConfigurationBuilderto add the AWS Secrets using the allowed prefixes - An extension method on
IWebHostBuilderwhich skips AWS secrets inDevelopmentenvironments
public static class AwsSecretsConfigurationBuilderExtensions
{
public static IWebHostBuilder AddAwsSecrets(this IWebHostBuilder hostBuilder)
{
return hostBuilder.ConfigureAppConfiguration((hostingContext, configBuilder) =>
{
// Don't add AWS secrets when running in develop
if(!hostingContext.HostingEnvironment.IsDevelopment())
{
// Call our extension method
configBuilder.AddAwsSecrets();
}
})
}
public static IConfigurationBuilder AddAwsSecrets(this IConfigurationBuilder configurationBuilder)
{
IConfiguration partialConfig = configBuilder.Build();
var settings = new AwsSecretsManagerSettings();
partialConfig
.GetSection(nameof(AwsSecretsManagerSettings))
.Bind(settings);
var env = hostingContext.HostingEnvironment.EnvironmentName;
var allowedPrefixes = settings.SecretGroups
.Select(grp => $"{hostingContext.Host}/{grp}/");
builder.AddSecretsManager(configurator: opts =>
{
opts.SecretFilter = entry => HasPrefix(allowedPrefixes, entry);
opts.KeyGenerator = (entry, key) => GenerateKey(allowedPrefixes, key);
});
}
// Only load entries that start with any of the allowed prefixes
private static bool HasPrefix(List<string> allowedPrefixes, SecretListEntry entry)
{
return allowedPrefixes.Any(prefix => entry.Name.StartsWith(prefix));
}
// Strip the prefix and replace '__' with ':'
private static string GenerateKey(IEnumerable<string> prefixes, string secretValue)
{
// We know one of the prefixes matches, this assumes there's only one match,
// So don't use '/' in your environment or secretgroup names!
var prefix = prefixes.First(secretValue.StartsWith);
// Strip the prefix, and replace "__" with ":"
return secretValue
.Substring(prefix.Length)
.Replace("__", ":");
}
}
That leaves our program.cs file nice and clean:
public class Program
{
public static void Main(string[] args) => BuildWebHost(args).Run();
public static IWebHost BuildWebHost(string[] args) =>
WebHost.CreateDefaultBuilder(args)
.AddAwsSecrets() // Add the AWS secrets to the configuration (if we're not in Dev)
.UseStartup<Startup>()
.Build();
}
We now have secure storage of secrets in AWS, loaded dynamically at runtime into an ASP.NET Core app.
Final thoughts
It's important to be aware of the limitations of any approach. If you're running your ASP.NET Core app in AWS then you'll be running under an IAM role which will have a range of security policies attached to it. In order to fetch a secret from AWS Secrets Manager, the role must have permission to fetch the secret. That means you can lock down access to secrets on a per-role basis. However, it also means that if your apps are all running with the same IAM role, then any app will be able to access the secrets from any other app. That's worth thinking about if you're running your applications on a single web server, or aren't using separate roles for each pod in Kubernetes for example.
Another point to consider with this design is that every AWS secret corresponds to a single configuration value. If you have a lot of configuration values, that could mean a lot of secrets stored in AWS, and more secrets to manage. The alternative would be to store all the secrets for a given secret group as a JSON object. That would significantly decrease the number of secrets stored in Secret Manager. It would also secure the actual configuration keys that are stored for each group.
I'm not sure if that's a good or bad thing to be honest - the approach I've described is working for me currently, so I'm inclined to leave it as it is. The good news is that if I change my mind later and start storing multiple values per secret, the Kralizek.Extensions.Configuration.AWSSecretsManager library supports JSON secrets out of the box, so a switch wouldn't mean any changes to my code. It will gracefully deconstruct a JSON payload into separate configuration values per string. See the source code if you're interested how this works.
Summary
In my previous post I showed how to use AWS Secrets Manager to securely store secrets, and how to use the Kralizek.Extensions.Configuration.AWSSecretsManager package to load them at runtime in ASP.NET Core apps. The solution shown in that post had two problems - you couldn't use different secrets for different environments, and secret keys were global across all of your apps. In this post I showed how to use a standard naming prefix and introduced the concept of secrets-groups to work around these issuses.