An important aspect of running ASP.NET Core apps in the cloud is how you secure the secrets your app requires, things like connection strings and API keys. In this post, I show one approach to securing your application when you're running in AWS - using AWS Secrets Manager to access the secrets at runtime.
I'll cover two aspects in this post:
- Adding a secret to AWS Secrets Manager
- Configuring an ASP.NET Core app to load secrets from AWS Secrets Manager at runtime
In my next post, I'll show how to improve this process by filtering out which secrets are loaded by a given application.
Protecting secrets in ASP.NET Core apps
Secrets are configuration values that are in some way sensitive and should not be public. They include things like connection strings, API Keys, and certificates. As a rule of thumb, you should never write these values in appsettings.json files or in any file that is checked-in to a source control repository. Ideally, they should be stored outside your source control working directory.
For local development, User Secrets (also known as Secrets Manager, but I'll stick to User Secrets for this post) is the preferred way to store sensitive values. This tool manages storing configuration values outside the working directory of your project (in your user profile directory). This works well for local development, though it's important to be aware that User Secrets doesn't encrypt the secrets, it just moves them to a location you're less likely to accidentally expose.
User Secrets are added to your ASP.NET Core configuration by default if you're using the default builder WebHost:
public static IWebHostBuilder CreateWebHostBuilder(string[] args) =>
WebHost.CreateDefaultBuilder(args) // Adds User Secrets to configuration
.UseStartup<Startup>();
User Secrets are intended only for development-time storage of secrets, what about in a production environment?
The approach you take will depend a lot on the environment you're running in. Environment variables are often suggested as a good choice, though there are some downsides.
If you're injecting environment variables at runtime (for example, into a docker container), then you need a process to do the injecting. That typically falls on your CI/CD server, which means it (the CI/CD server) needs to have knowledge (or know how to retrieve) all the required secrets for every application.
Alternatively, you could "bake" the the secrets into your docker images themselves, so that when they're run they have all the required secrets available. That's generally a bad idea as anyone can inspect the Docker image and retrieve the secrets directly (assuming they're not previously encrypted in some way).
A better approach, and the approach generally advocated by Microsoft, is to store your secrets in a dedicated "Secrets Vault" such as Azure Key Vault. At runtime, your app requests access to Secrets from the Azure Key Vault service.
One practicality to be aware of - accessing Azure Key Vault requires a "Client Secret" to call the Azure Key Vault API. That leaves you with a chicken-and-egg problem. You store your secrets securely, but you can't access them without an API key, so how do you store that API key securely? If your app is running in Azure, I think Managed Service Identity can solve this problem, but I haven't used it myself so I'm not sure.
AWS Secrets Manager serves an essentially identical role as Azure Key Vault. It securely stores your secrets until you retrieve them at runtime. If your going to be running your ASP.NET Core app in AWS, then AWS Secrets Manager is a great option, as it allows you to finely control the permissions associated with the AWS IAM roles running your apps. It also handles the chicken and egg problem if you're running inside AWS: if the AWS role used to run your app has access to AWS Secrets Manager, you don't have to worry about storing extra API/Access keys; you control access at the AWS role/policy level instead of using API keys.
Adding a secret to AWS Secrets Manager
As with most AWS features, there are several ways to add a secret to AWS Secrets Manager. I'll show how to do so via the CLI tool and the AWS Console.
Note: To generate secrets, you'll need to have the
secretsmanager:CreateSecretpermission granted for your user/role in IAM
AWS Secrets Manager has a lot of different features, that I'm not going to touch on in this post. You can store database credentials, key-value pairs, plaintext strings, encrypt with custom AWS KMS keys, and rotate database credentials automatically. I'm going to store a simple plain text string (the secret value) and use the name of the secret as the key.
The secret I'm going to store is a connection string, and I want to store it so that the final key added to the ASP.NET Core configuration is ConnectionStrings:MyTestApp i.e. it's the MyTestApp key, in the ConnectionStrings section. If we were storing the secret directly in JSON, it would look something like this:
{
"ConnectionStrings": {
"MyTestApp": "Server=127.0.0.1;Port=5432;Database=myDataBase;User Id=myUsername;Password=myPassword;"
}
}
For legacy reasons, I'm actually going to store the : in the secret name as __, and replace it inside the app when the secret is loaded. This is consistent with the behaviour for environment variables in ASP.NET Core, and I think it's a nice convention to preserve. So the secret name stored in AWS will be ConnectionStrings__MyTestApp.
An important point to note is that you'll need to configure the AWS IAM role used to run your ASP.NET Core application with the necessary permissions to List and Fetch stored secrets, plus to decrypt secrets with KMS. An example IAM Policy snippet might look something like the following (but be sure to check for your own environment - this allows the role to read any secret in Secrets Manager)
{
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Action": [
"kms:Decrypt",
"kms:Encrypt"
],
"Resource": "*"
},
{
"Action": ["secretsmanager:ListSecrets"],
"Effect": "Allow",
"Resource": "*"
},
{
"Action": ["secretsmanager:GetSecretValue"],
"Effect": "Allow",
"Resource": "*"
}
]
}
Adding a secret using the CLI
If you already have the AWS CLI configured for your environment, then you can create a new secret with a single command:
aws secretsmanager create-secret \
--name ConnectionStrings__MyTestApp \
--description "The connection string for my test app" \
--secret-string "Server=127.0.0.1;Port=5432;Database=myDataBase;User Id=myUsername;Password=myPassword;"
Alternatively, you could use the contents of a file as the secret value. This avoids printing the secret to the console:
aws secretsmanager create-secret \
--name ConnectionStrings__MyTestApp \
--description "The connection string for my test app" \
--secret-string file://connectionstring.txt
Both of these options will use the default KMS encryption key to store the secret, and won't configure rotation.
Adding a secret using the AWS Console
Does it bug anyone else that AWS calls their web UI the Console? Console always makes me think of command line!
If CLIs aren't your thing, you can create a new secret using the AWS Console. Navigate to Secrets Manager for your desired region, and click "Store a New Secret".
This will take you to the "Store a new Secret" wizard. The first step is to choose the type of secret, and set its value. We'll be using the "Other type of secret" and will store the plaintext value. We'll leave the encryption as the default for now.
Click Next, and on the next page enter the name ConnectionStrings__MyTestApp and a description for the secret.
Click Next to view the rotation option. We'll leave rotation disabled. Click Next again and you're presented with a review of your secret, and sample code for accessing it. Don't worry about this - we'll be using the AWS .NET SDK to abstract away all this complexity. Finally, click Store to store your secret in AWS Secrets Manager.
After storing the secret, you can view the details of the secret from the Secrets Manager home page, and manually retrieve the secret value if necessary.
Loading secrets from AWS Secrets Manager with ASP.NET Core
Now we've got a secret stored, it's on to the interesting part - loading it in our ASP.NET Core application. To do this I used a little NuGet package called Kralizek.Extensions.Configuration.AWSSecretsManager. The library is open source on GitHub - it doesn't have many stars, but its actual code foot print is very small, and it does everything I need so I recommend it.
Add the package to your project using
dotnet add package Kralizek.Extensions.Configuration.AWSSecretsManager
Or update your .csproj file:
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFrameworks>netcoreapp2.1</TargetFrameworks>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Microsoft.AspNetCore.App" />
<!-- Add the package-->
<PackageReference Include="Kralizek.Extensions.Configuration.AWSSecretsManager" Version="1.0.0" />
</ItemGroup>
</Project>
The README for the library includes great instructions for how to add configuration using AWS Secrets Manager to your application. The simplest use case is to add the AWS Secrets to the standard CreateDefaultBuilder call in Program.cs by using ConfigureAppConfiguration():
public class Program
{
public static void Main(string[] args) => BuildWebHost(args).Run();
public static IWebHost BuildWebHost(string[] args) =>
WebHost.CreateDefaultBuilder(args)
.ConfigureAppConfiguration((hostingContext, config) =>
{
// Load AWS Secrets after all other configuration steps
config.AddSecretsManager();
})
.UseStartup<Startup>()
.Build();
}
This adds AWS Secrets loading after all other configuration sources (e.g. appsettings.json, environment variables, command line arguments). It also assumes that AWS credentials are available by default to your application, using the usual AWS SDK mechanisms. If you're running your app inside AWS, then this will likely work out of the box. If not, there are other ways to obtain the necessary credentials, but I won't go into those here, as they circle back to the chicken and egg issue!
This is almost all we need for our application, but there's a couple of tweaks I want to make.
Don't add AWS secrets when developing locally
First, I don't want to use AWS SecretsManager when developing applications locally - .NET User Secrets are a better solution for that problem. It's possible that _my_ connection string for a local development database will be different to someone else's connection string when they're working locally on the app. AWS Secrets Manager configuration would be shared across all users, and would overwrite any local values we'd set. On top of that, it means I don't have to have AWS credentials configured for my local environment to run the application.
Ignoring the AWS Secrets Manager when developing locally is easy to achieve by using Hosting Environments. We can conditionally check if we're in the Development environment and only call AddSecretsManager() if it's appropriate:
public static IWebHost BuildWebHost(string[] args) =>
WebHost.CreateDefaultBuilder(args)
.ConfigureAppConfiguration((hostingContext, config) =>
{
if(!hostingContext.HostingEnvironment.IsDevelopment())
{
// Don't add AWS secrets in local environment
config.AddSecretsManager();
}
})
.UseStartup<Startup>()
.Build();
}
Customising the generated key
There's one more piece of configuration to add. When I created the AWS secret, I said that I would store the secret using the __ token to separate configuration sections. The .NET Core configuration system requires you use : to separate sections, so we need to change the secret name when it's loaded. Luckily, Kralizek.Extensions.Configuration.AWSSecretsManager provides an options object to do just that.
Our key generation function isn't complicated, we are just replacing instances of "__" with ":":
static string GenerateKey(SecretListEntry secret, string secretName)
{
return secretName.Replace("__", ":");
}
Note that you have access to the whole
SecretListEntryobject here, so you can do more complex transformations if necessary. This object includes all the metadata about the secret (LastChangedDateandTagsfor example), but not the value itself.
Add the GenerateKey call to the AwsSecretsManager() call by configuring the SecretsManagerConfigurationProviderOptions object. As the GenerateKey is so simple, I've inlined it here:
public static IWebHost BuildWebHost(string[] args) =>
WebHost.CreateDefaultBuilder(args)
.ConfigureAppConfiguration((hostingContext, config) =>
{
if(!hostingContext.HostingEnvironment.IsDevelopment())
{
config.AddSecretsManager(configurator: ops =>
{
// Replace __ tokens in the configuration key name
opts.KeyGenerator = (secret, name) => name.Replace("__", ":");
});
}
})
.UseStartup<Startup>()
.Build();
And that's it - you now have secrets being stored securely in Secrets Manager, and loaded at runtime based on the AWS IAM role your app is running under! This is great for getting started, but you'll generally want a more sophisticated setup than this provides, especially if you're storing secrets for multiple apps or for multiple environments. In the next post I'll show how you can achieve that with just a few tweaks to this basic configuration.
Summary
In this post I showed how to create a secret in AWS Secrets Manager using the AWS CLI or the web Console. I showed how to configure an ASP.NET Core application to load the secrets at runtime using the Kralizek.Extensions.Configuration.AWSSecretsManager NuGet package. One big advantage to using AWS Secrets Manager if you're already running in AWS is that you don't need to configure additional API keys. That avoids the "chicken and egg" problem you can run into if trying to access secure resources from outside the ecosystem.