Setting environment variables for ASP.NET Core apps in a Helm chart: Deploying ASP.NET Core applications to Kubernetes - Part 5

So far in this series I've provided a general introduction to Kubernetes and Helm, and we've deployed a basic ASP.NET Core solution using a Helm chart. In this post we extend the Helm chart to allow setting configuration values at deploy time, which are added to the application pods as environment variables.

The sample app: a quick refresher

In the previous post I described the .NET solution that we're deploying. It consists of two applications, TestApp.Api which is a default ASP.NET Core web API project, and a TestApp.Service which is an empty web project. The TestApp.Service represents a "headless" service, that would be handling messages from an event queue using something like NServiceBus or MassTransit.

We created Docker images for both of these apps, and created a Helm chart for the solution, that consists of a "top-level" Helm chart test-app containing two sub-charts (test-app-api and test-app-service).

When installed, these charts create a deployment for each app, a service for each app, and an ingress for the test-app-api only.

In the previous post, we saw how to control various settings of the Helm chart by adding values to the Chart's values.yaml file, and also at install-time, by passing --set key=value to the helm upgrade --install command.

We used this approach to set various Kubernetes and Helm related values (what service types to use, which ports to expose etc.) but we didn't change anything in our application. In this post, we want to override configuration in our ASP.NET Core apps, for example to change the HostingEnvironment our apps are using.

Setting pod environment variables in a deployment manifest

In the previous post, when we checked the logs for our API app, we noticed that it was using the default hosting environment, Production, and that it had not been configured to handle HTTPS redirection correctly.

kubectl logs -n=local -l app.kubernetes.io/name=test-app-api

info: Microsoft.Hosting.Lifetime[0]
      Now listening on: http://[::]:80
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Production # The default environment
info: Microsoft.Hosting.Lifetime[0]
      Content root path: /app
warn: Microsoft.AspNetCore.HttpsPolicy.HttpsRedirectionMiddleware[3]
      Failed to determine the https port for redirect.  # The app doesn't know how to redirect from HTTP -> HTTPS

We're deploying to a test environment at the moment, so we want to change the hosting environment to Staging. We'll also let the NGINX ingress controller handle SSL/TLS offloading, so we want to ensure our app uses the correct X-Forwarded-Proto headers to understand whether the original request came over HTTP or HTTPS

When you're deploying in a reverse-proxy environment such as in Kubernetes, it's important you configure the ForwardedHeadersMiddleware and options. This enables things like SSL/TLS offloading (as I'm using), where the application sends HTTPS requests to the ingress, but the ingress forwards the request to your deployment using HTTP. Setting forwarded headers tells your application the original request was over HTTPS.

You can set environment variables in pods by adding an env: dictionary to the deployment.yaml manifest. For example, in the following manifest, I've added an env section underneath the test-app-api container in the spec:containers section.

I find lots of YAML really hard work to look at, but I've included a whole manifest here because it's vital that you get the white-space and indentation correct. Errors in white-space a nightmare to debug!

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-app-api-deployment
spec:
  replicas: 3
  strategy: 
    type: RollingUpdate
  selector:
    matchLabels:
      app: test-app-api
  template:
    metadata:
      labels:
        app: test-app-api
    spec:
      containers:
      - name: test-app-api
        image: andrewlock/my-test-api:0.1.1
        ports:
        - containerPort: 80
        # Environment variable section
        env:
        - name: "ASPNETCORE_ENVIRONMENT"
          value: "Staging"
        - name: "ASPNETCORE_FORWARDEDHEADERS_ENABLED"
          value: "true"

In the above example, I've added two environment variables - one setting the ASPNETCORE_ENVIRONMENT variable, which controls the application's HostingEnvironment, and one which enables the ForwardedHeaders middleware, so the application knows it's behind a reverse-proxy (in this case an NGINX ingress controller).

The ASPNETCORE_FORWARDEDHEADERS_ENABLED environment variable is an easy way to configure the middleware, when you're sure your application is behind a trusted proxy.

If we install that manifest and check the logs, we'll see that the hosting environment has changed to Staging:

kubectl logs -n=local -l app.kubernetes.io/name=test-app-api

info: Microsoft.Hosting.Lifetime[0]
      Now listening on: http://[::]:80
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Staging # Updated to staging
info: Microsoft.Hosting.Lifetime[0]
      Content root path: /app
warn: Microsoft.AspNetCore.HttpsPolicy.HttpsRedirectionMiddleware[3]
      Failed to determine the https port for redirect.

You'll notice that we still have the "Failed to determine the https port for redirect". You can resolve this by setting the ASPNETCORE_HTTPS_PORT variable. I've avoided doing that here, as it causes issues with the liveness probes, which we discuss in later posts.

In the above example we've hard-corded the environment variable configuration into the deployment.yaml manifest. In practice, we want to provide these values at install time so we should use Helm's support for templating and injecting values.

Setting environment variables using Helm variables

First, lets update the deployment.yaml Helm template to use values provided at install time. I'm not going to reproduce the whole template here, just the bits we're interested in. Again, make sure you get the indentation right when you add it to your manifest!

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-app-api-deployment
spec:
  template:
    spec:
      containers:
      - name: test-app-api
        image: andrewlock/my-test-api:0.1.1
        # Environment variable section
        env:
        {{ range $k, $v := .Values.env }}
          - name: {{ $k | quote }}
            value: {{ $v | quote }}
        {{- end }}

The important part is those last 4 lines. That syntax says

• Retrieve .Values.env, that is, the env section of the current Helm values (the values provided in values.yaml merged with the values provided using the --set syntax at install time)
• The env section should be a dictionary/map structure. Repeat the content inside the {{range}} {{- end}} block for each key-value-pair.
• For each key-value pair, assign the key to $k and the value to $v.
• {{ $k | quote }} means "print the variable $k, adding quote (") marks as necessary".

That means we can set values in values.yaml using, for example:

# config for test-app-api 
test-app-api:

  env: 
    "ASPNETCORE_ENVIRONMENT": "Staging"
    "ASPNETCORE_FORWARDEDHEADERS_ENABLED": "true"

  image:
    repository: andrewlock/my-test-api

  # ... other config

# config for test-app-service
test-app-service:

  env: 
    "ASPNETCORE_ENVIRONMENT": "Staging"
    "ASPNETCORE_FORWARDEDHEADERS_ENABLED": "true"

  image:
    repository: andrewlock/my-test-service

  # ... other config

At install time, the template gets rendered as the following:

spec:
  template:
    spec:
      containers:
      - name: test-app-api
        image: andrewlock/my-test-api:0.1.1
        env:
          - name: "ASPNETCORE_ENVIRONMENT"
            value: "Staging"
          - name: "ASPNETCORE_FORWARDEDHEADERS_ENABLED"
            value: "true"

You can also set/override the environment variables using --set arguments in your helm upgrade --install command, for example:

helm upgrade --install my-test-app-release . \
  --namespace=local \
  --set test-app-api.image.tag="0.1.0" \
  --set test-app-service.image.tag="0.1.0" \
  --set test-app-api.env.ASPNETCORE_ENVIRONMENT="Staging" \
  --set test-app-service.env.ASPNETCORE_ENVIRONMENT="Staging"

Of course, there's an obvious annoyance here—we're having to duplicate environment variables for each service, even though we want the exact same values. Luckily, there's an easy way around that using global values.

Using global values to reduce duplication

Helm's global values are exactly what they sound like: they're values you set globally that all sub-charts can access. The values you've seen so far have all been scoped to a specific chart by using a test-app-api: or test-app-service: section in values.yaml. Global values are set at the top-level. For example, if we use global values for our env configuration, then we could just specify them once in values.yaml, in the global: section:

# global config
global:
  env: 
    "ASPNETCORE_ENVIRONMENT": "Staging"
    "ASPNETCORE_FORWARDEDHEADERS_ENABLED": "true"

# config for test-app-api 
test-app-api:
  image:
    repository: andrewlock/my-test-api

  # ... other app-specific config

# config for test-app-service
test-app-service:
  image:
    repository: andrewlock/my-test-service

  # ... other app-specific config

These values are added to a global variable under .Values, so to use these values in our sub-charts, we need to update the manifests to use .Values.global.env instead of .Values.env:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-app-api-deployment
spec:
  template:
    spec:
      containers:
      - name: test-app-api
        image: andrewlock/my-test-api:0.1.1
        env:
        {{ range $k, $v := .Values.global.env }} # instead of .Values.env
          - name: {{ $k | quote }}
            value: {{ $v | quote }}
        {{- end }}

At install time, you can override these values as before using --set global.env.key="value" (note the global. prefix). For example:

helm upgrade --install my-test-app-release . \
  --namespace=local \
  --set test-app-api.image.tag="0.1.0" \
  --set test-app-service.image.tag="0.1.0" \
  --set global.env.ASPNETCORE_ENVIRONMENT="Staging" 

That's definitely better, but what if you want the best of both worlds? You want to be able to globally set environment variables, but you want to be able to set/override them for specific apps too. On the face of it, it seems like you can just combine both of the techniques above, but doing that naïvely can give strange errors…

Merging global and sub-chart-specific values: the wrong way

As an example of the problem, lets imagine you want to be able to set a global value, and override it at the sub-chart level. You might update the template section of your manifest to this:

env:
{{ range $k, $v := .Values.global.env }} # global variables
  - name: {{ $k | quote }}
    value: {{ $v | quote }}
{{- end }}
{{ range $k, $v := .Values.env }} # sub-chart variables
  - name: {{ $k | quote }}
    value: {{ $v | quote }}
{{- end }}

and set different global and sub-chart values at install time:

helm upgrade --install my-test-app-release . \
  --namespace=local \
  --set global.env.ASPNETCORE_ENVIRONMENT="Staging" \          # global value
  --set test-app-api.env.ASPNETCORE_ENVIRONMENT="Development"  # sub-chart value

Unfortunately, the way we've designed our manifest means that we don't get "override" semantics. Instead, you will end up setting the environment variable twice, with two different values:

env:
  - name: "ASPNETCORE_ENVIRONMENT"
    value: "Staging"
  - name: "ASPNETCORE_ENVIRONMENT"
    value: "Development"

Unfortunately, this gives a horrendously, confusing error message, which only seems to appear when you update a chart

Error: UPGRADE FAILED: The order in patch list:
[map[name:ASPNETCORE_ENVIRONMENT value:Staging] map[name:ASPNETCORE_ENVIRONMENT value:Development] map[name:ASPNETCORE_FORWARDEDHEADERS_ENABLED value:true]]
 doesn't match $setElementOrder list:
[map[name:ASPNETCORE_ENVIRONMENT] map[ASPNETCORE_FORWARDEDHEADERS_ENABLED]]

In this very basic example, you might figure it out, but I'd be impressed! One way around this is the manual approach of avoiding setting global variables if you set them for sub-charts. That works around the issue but isn't particularly user friendly.

Instead, you can use a bit of dictionary manipulation to correctly merge the values in the two dictionaries before you write them in your manifest

Merging global and sub-chart specific variables: the right way

To solve our problem we're going to use a function from an underlying package, sprig, that helm uses to provide the templating functionality. In particular, we're going to use the dict function, to create an empty dictionary, and the merge function, to merge two dictionaries.

Update your deployment.yaml manifests to the following:

env:
{{- $env := merge (.Values.env | default dict) (.Values.global.env | default dict) -}}
{{ range $k, $v := $env }}
  - name: {{ $k | quote }}
    value: {{ $v | quote }}
{{- end }}

This does the following:

• {{- $env := ... -}} Defines a variable $env, which will be the final dictionary containing the variables we want to merge
• (.Values.env | default dict) use the values provided in the env: section. If that section doesn't exist, create an empty dictionary instead.
• (.Values.global.env | default dict) as above, but for the global values.
• merge a b Merge the values of b into a. The order is important here—keys in a will not be overridden if they appear in b too, so we need a to be the most specific values, and b to be the most general values.

With this configuration, you can now set values globally and override them for specific sub-charts:

helm upgrade --install my-test-app-release . \
  --namespace=local \
  --set global.env.ASPNETCORE_ENVIRONMENT="Staging" \          # global value
  --set test-app-api.env.ASPNETCORE_ENVIRONMENT="Development"  # sub-chart value

For the test-app-api sub-chart, that now renders (correctly) as:

env:
  - name: "ASPNETCORE_ENVIRONMENT"
    value: "Development"

Setting environment variables like this is the preferred way for getting configuration values into your app when you're deploying in Kubernetes. You can still use appsettings.json for "static" configuration, but for any configuration that is environment specific, environment variables are the way to go.

Injecting secrets into your apps is a whole other aspect, as it can be tricky to do safely! I've blogged previously about using AWS Secrets Manager directly from your apps, but there are also (complicated) approaches which plug directly into Kubernetes.

The approach we've seen so far is great for setting environment variables when the configuration values are known at the time you install the chart. But in some situations, your app might need to know details about its configuration in the Kubernetes environment, such as its IP address. For those circumstances, you'll need a slightly different configuration.

Exposing pod information to your applications

When Kubernetes runs your application in a pod, it knows various things about your pod, for example:

• The name of the Node it's running on
• What service account it's running under
• The IP Address of the pod
• The IP Address of the host Node

In most cases, your application shouldn't care about those values. Ideally, your application should need to know as little about its environment as possible.

However, in some cases, you may find you need to access these values. One example might be that you're running DataDog's StatsD agent on a Node, and you need to set the IP address in your application's config.

There are obviously multiple ways to obtain metrics from your app, and this isn't necessarily the best one, it's just an example!

You can "inject" values that Kubernetes knows about a pod as environment variables into your pod. This uses a similar syntax the -name/value configuration you've already seen, but it uses valueFrom instead. For example:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-app-api-deployment
spec:
  template:
    spec:
      containers:
      - name: test-app-api
        image: andrewlock/my-test-api:0.1.1
        env:
          - name: "ASPNETCORE_ENVIRONMENT"
            value: "Development"             # A "static" value
          - name: "MyPodIp"
            valueFrom: 
              fieldRef: 
                fieldPath: status.hostIP     # A dynamic variable, set when the pod is provisioned

In the example above, the pod will have two environment variables:

• ASPNETCORE_ENVIRONMENT is set "statically" using the value provided in the manifest. You can also use a static value using set when calling helm upgrade --install, by using the templating approach I've already described
• MyPodIp is set to the host Node's IP address, This is set dynamically when the pod is created. Different pods in the same deployment may have different values if they're deployed on different Nodes.

There are a variety of different values available, sourced from the manifest used to deploy the pod or from runtime values taken from status. The only ones I've used personally are status.hostIP to get the host Node's IP address, and status.podIP to get the pod's IP address.

You can read more about this approach in the documentation. This also shows how to inject container-specific values, in addition to pod-specific values.

Rather than hard-coding values and mappings into your deployment.yaml manifest, as I did above, it's better to use Helm's templating capabilities to extract this into configuration. We can use a similar approach as I showed in previous sections to create envValuesFrom sections, which define an environment variable-to-fieldPath mapping. For example:

env:
{{ range $k, $v := .Values.global.envValuesFrom }}
  - name: {{ $k | quote }}
    valueFrom:
      fieldRef:
        fieldPath: {{ $v | quote }}
{{- end }}

You could then create a mapping between the Runtime__IpAddress environment variable and the status.podIP field by using the following configuration in your values.yaml (or alternatively using --set when installing the chart):

global:
  # Environment variables shared between all the pods, populated with valueFrom: fieldRef
  envValuesFrom:
    Runtime__IpAddress: status.podIP

Note that I've used the double underscore __ in the environment variable name. The translates to a "section" in ASP.NET Core's configuration, so this would set the configuration value Runtime:IpAdress to the pod's IP address.

When Helm renders the manifest, it will create an env section like the following:

env:
  - name: "Runtime__IpAddress"
    valueFrom: 
      fieldRef: 
        fieldPath: "status.podIP"

You can allow "overriding" envValuesFrom using the same dictionary-merging technique I described previously, but I've not found much of a need for that personally. You can also use envValuesFrom in conjunction with env to give a combination of static and dynamic environment variables. I typically just render both lists in my manifest—I don't use envValuesFrom very often, and there's never been any overlap with env values:

env:
{{ range $k, $v := .Values.global.envValuesFrom }} # dynamic values
  - name: {{ $k | quote }}
    valueFrom:
      fieldRef:
        fieldPath: {{ $v | quote }}
{{- end }}

{{- $env := merge (.Values.env | default dict) (.Values.global.env | default dict) -}} # static values, merged together
{{ range $k, $v := $env }}
  - name: {{ $k | quote }}
    value: {{ $v | quote }}
{{- end }}

That covers how I handle injecting configuration into ASP.NET Core applications when installing helm charts. In the next post we'll cover another important aspect: liveness probes.

Summary

In this post I showed how you can use Helm values to inject values into your ASP.NET Core applications as environment variables. I showed how to use templating so that you can set these values at runtime, and how to reduce duplication by using global values. I also showed how to safely combine global and sub-chart-specific values using the merge function. Finally, I showed how to inject dynamic environment variables, such as a pod's IP address, using the valueFrom syntax.