Gotchas upgrading from IdentityServer 3 to IdentityServer 4

This post covers a couple of gotchas I experienced upgrading an IdentityServer 3 implementation to IdentityServer 4. I've written about a previous issue I ran into with an OWIN app in this scenario - where JWTs could not be validated correctly after upgrading. In this post I'll discuss two other minor issues I ran into:

1. The URL of the JSON Web Key Set (JWKS) has changed from /.well-known/jwks to .well-known/openid-configuration/jwks.
2. The KeyId of the X509 certificate signing material (used to validate the identity token) changes between IdentityServer 3 and IdentityServer 4. That means a token issued by IdentityServer 3 will not be validated using IdentityServer 4, leaving users stuck in a redirect loop.

Both of these issues are actually quite minor, and weren't a problem for us to solve, they just caused a bit of confusion initially! This is just a quick post about these problems - if you're looking for more information on upgrading from IdentityServer 3 to 4 in general, I suggest checking out the docs, the announcement post, or this article by Scott Brady.

1. The JWKS URL has changed

OpenID Connect uses a "discovery document" to describe the capabilities and settings of the server - in this case, IdentityServer. This includes things like the Claims and Scopes that are available and the supported grants and response types. It also includes a number of URLs indicating other available endpoints. As a very compressed example, it might look like the following:

{
    "issuer": "https://example.com",
    "jwks_uri": "https://example.com/.well-known/openid-configuration/jwks",
    "authorization_endpoint": "https://example.com/connect/authorize",
    "token_endpoint": "https://example.com/connect/token",
    "userinfo_endpoint": "https://example.com/connect/userinfo",
    "end_session_endpoint": "https://example.com/connect/endsession",
    "scopes_supported": [
        "openid",
        "profile",
        "email"
    ],
    "claims_supported": [
        "sub",
        "name",
        "family_name",
        "given_name"
    ],
    "grant_types_supported": [
        "authorization_code",
        "client_credentials",
        "refresh_token",
        "implicit"
    ],
    "response_types_supported": [
        "code",
        "token",
        "id_token",
        "id_token token",
    ],
    "id_token_signing_alg_values_supported": [
        "RS256"
    ],
    "code_challenge_methods_supported": [
        "plain",
        "S256"
    ]
}

The discovery document is always located at the URL /.well-known/openid-configuration, so a new client connecting to the server knows where to look, but the other endpoints are free to move, as long as the discovery document reflects that.

In our move from IdentityServer 3 to IdentityServ4, the JSWKs URL did just that - it moved from /.well-known/jwks to /.well-known/openid-configuration/jwks. The discovery document obviously reflected that, and all of the IdentityServer .NET client libraries for doing token validation, both with .NET Core and for OWIN, switched to the correct URLs without any problems.

What I didn't appreciate, was that we had a Python app which was using IdentityServer for authentication, but which wasn't using the discovery document. Rather than go to the effort of calling the discovery document and parsing out the URL, and knowing that we controlled the IdentityServer implementation, the /.well-known/jwks URL was hard coded.

Oops!

Obviously it was a simple hack to update the hard coded URL to the new location, though a much better solution would be to properly parse the discovery document.

2. The KeyId of the signing material has changed

This is a slightly complex issue, and I confess, this has been on my backlog to write up for so long that I can't remember all the details myself! I do, however, remember the symptom quite vividly - a crazy, endless, redirect loop on the client!

The sequence of events looked something like this:

1. The client side app authenticates with IdentityServer 3, obtaining an id and access token.
2. Upgrade IdentityServer to IdentityServer 4.
3. The client side app calls the API, which tries to validate the token using the public keys exposed by IdenntityServer 4. However IdentityServer 4 can't seem to find the key that was used to sign the token, so this validation fails causing a 401 redirect.
4. The client side app handles the 401, and redirects to IdentityServer 4 to login.
5. However, you're already logged in (the cookie persists across IdentityServer versions), so IdentityServer 4 redirects you back.
6. Go to 4.

It's possible that this issue manifested as it did due to something awry in the client side app, but the root cause of the issue was the fact a token issued by IdentityServer 3 could not be validated using the exposed public keys of IdentityServer 4, even though both implementations were using the same signing material - an X509 certificate.

The same public and private keypair is used in both IdentityServer 3 and IdentityServer4, but they have different identifiers, so IdentityServer thinks they are different keys.

In order to validate an access token, an app must obtain the public key material from IdentityServer, which it can use to confirm the token was signed with the associated private key. The public keys are exposed at the jwks endpoint (mentioned earlier), something like the following (truncated for brevity):

{
  "keys": [
    {
      "kty": "RSA",
      "use": "sig",
      "kid": "E23F0643F144C997D6FEEB320F00773286C2FB09",
      "x5t": "4j8GQ_FEyZfW_usyDwB3MobC-wk",
      "e": "AQAB",
      "n": "rHRhPtwUwp-i3lA_CINLooJygpJwukbw",
      "x5c": [
        "MIIDLjCCAhagAwIBAgIQ9tul\/q5XHX10l7GMTDK3zCna+mQ="
      ],
      "alg": "RS256"
    }
  ]
}

As you can see, this JSON object contains a keys property which is an array of objects (though we only have one here). Therefore, when validating an access token, the API server needs to know which key to use for the validation.

The JWT itself contains metadata indicating which signing material was used:

{
  "alg": "RS256",
  "kid": "E23F0643F144C997D6FEEB320F00773286C2FB09",
  "typ": "JWT",
  "x5t": "4j8GQ_FEyZfW_usyDwB3MobC-wk"
}

As you can see, there's a kid property (KeyId) which matches in both the jwks response and the value in the JWT header. The API token validator uses the kid contained in the JWT to locate the appropriate signing material from the jwks endpoint, and can confirm the access token hasn't been tampered with.

Unfortunately, the kid was not consistent across IdentityServer 3 and IdentityServer 4. When trying to use a token issued by IdentityServer 3, IdentityServer 4 was unable to find a matching token, and validation failed.

For those interested, IdentityServer3 uses the bae 64 encoded certificate thumbprint as the KeyId - Base64Url.Encode(x509Key.Certificate.GetCertHash()). IdentityServer 4 [uses X509SecurityKey.KeyId] (https://github.com/IdentityServer/IdentityServer4/blob/993103d51bff929e4b0330f6c0ef9e3ffdcf8de3/src/IdentityServer4/ResponseHandling/DiscoveryResponseGenerator.cs#L316) which is slightly different - a base 16 encoded version of the hash.

Our simple solution to this was to do the upgrade of IdentityServer out of hours - in the morning, the IdentityServer cookies had expired and so everyone had to re-authenticate anyway. IdentityServer 4 issued new access tokens with a kid that matched its jwks values, so there were no issues 🙂

In practice, this solution might not work for everyone, for example if you're not able to enforce a period of downtime. There are other options, like explicitly providing the kid material yourself as described in this issue if you need it. If the kid doesn't change between versions, you shouldn't have any issues validating old tokens in the upgrade.

Alternatively, you could add the signing material to IdentityServer 4 using both the old and new kids. That way, IdentityServer 4 can validate tokens issued by IdentityServer 3 (using the old kid), while also issuing (and validating) new tokens using the new kid.

Summary

This post describes a couple of minor issues upgrading a deployment from IdentityServer 3 to IdentitySerrver4. The first issue, the jwks URL changing, is not an issue I expect many people to run into - if you're using the discovery document you won't have this problem. The second issue is one you might run into when upgrading from IdentityServer 3 to IdentityServer 4 in production; even if you use the same X509 certificate in both implementations, tokens issued by IdentityServer 3 can not be validated by IdentityServer 4 due to mis-matching kids.