Automatic key management doesn't work as expected

[ivan-sedlak-visma]

Which version of Duende IdentityServer are you using?
6.3.3

Which version of .NET are you using?
.NET 6

Describe the bug
We enabled the automatic key management and initially, it worked as expected, generating new signing keys in db which are visible in the disco doc. We still keep the old key registered manually.
In the meantime, we disabled some of the algorithms and left only the RSA256 (X509). The other autogenerated signing keys disappeared from disco doc, all is good thus far. But the issue is, seems like new keys are not generated as the existing one (autogenerated) already expired. We still sign tokens with our manual key but we can't disable it because new keys are not being announced.

To Reproduce

• Enable automatic key management
• Keep the old (but valid) certificate manually registered
• Enable 3 signing algorithms
• Run the app and check disco doc. The new keys are in disco doc.
• Disable 2 signing algorithms and change the rotation interval, propagation time, and retention duration
• One autogenerated and one manual signing key in the disco doc, all good.
• It stays this way, autogenerated one expires, and no new signing keys are announced, this is bad.

Expected behavior

• New signing key is announced as specified by the rotation interval

[josephdecock]

@ivan-sedlak-visma, thanks for this bug report. We'll investigate and update here soon.

[josephdecock]

@ivan-sedlak-visma - so far I've not been able to reproduce this issue in my environment. Can you go into more detail about the values you're using for all the relevant configurations? For anything that you're changing, before and after values would be helpful too.

[ivan-sedlak-visma]

Thanks @josephdecock. If the feature is up and running properly on your side it might be something we do with injecting a signing certificate and a validation certificate the manual way or something with our db store. When I get the time I will go in-depth and report back with more info.

[ivan-sedlak-visma]

@josephdecock Here are more details.

I've noticed that when I run IdentityServer in localhost, with a fresh db instance (no cert data), it generates the RS256 certificate as specified by current configuration (same key management config in all envs):

		options.KeyManagement.Enabled = true;
		options.KeyManagement.RotationInterval = TimeSpan.FromDays(90);
		options.KeyManagement.PropagationTime = TimeSpan.FromDays(14);
		options.KeyManagement.RetentionDuration = TimeSpan.FromDays(7);
		options.KeyManagement.DeleteRetiredKeys = true;
		options.KeyManagement.SigningAlgorithms = new[]
		{
			new SigningAlgorithmOptions(SecurityAlgorithms.RsaSha256) { UseX509Certificate = true }
		};

However, in staging and production environments, where db is not fresh (has data) it still keeps outdated certificate. I enabled the Verbose logging and this is what I got:

Getting all the keys.
Cache miss when loading all keys.
Loading keys from store.
Loaded keys from store: "88CF32355BAE0F4E0DD9E73BFA6E232E,3030AAC5B919B4567A0BCC47C5074B83,5B49719B0EF3414A640251AAEAAC236F"
Remaining keys after filter: "88CF32355BAE0F4E0DD9E73BFA6E232E,3030AAC5B919B4567A0BCC47C5074B83,5B49719B0EF3414A640251AAEAAC236F"
Keys with allowed alg from store: "88CF32355BAE0F4E0DD9E73BFA6E232E"
Keys successfully returned from store.
Caching keys with KeyCacheDuration for 1.00:00:00
Looking for active signing keys.
Looking for an active signing key for alg "RS256".
Checking if key with kid "88CF32355BAE0F4E0DD9E73BFA6E232E" is active (respecting activation delay).
Key with kid "88CF32355BAE0F4E0DD9E73BFA6E232E" is active.
Active signing key found (respecting the activation delay) with kid: "88CF32355BAE0F4E0DD9E73BFA6E232E".
Found active signing key for alg "RS256" with kid "88CF32355BAE0F4E0DD9E73BFA6E232E".
Checking if key with kid "88CF32355BAE0F4E0DD9E73BFA6E232E" is active (respecting activation delay).
Key with kid "88CF32355BAE0F4E0DD9E73BFA6E232E" is active.
Active signing key found (respecting the activation delay) with kid: "88CF32355BAE0F4E0DD9E73BFA6E232E".
Key rotation not required for alg "RS256"; New key expected to be created in 11.20:59:48
Getting the current key.
Cache hit when loading all keys.
Looking for active signing keys.
Looking for an active signing key for alg "RS256".
Checking if key with kid "88CF32355BAE0F4E0DD9E73BFA6E232E" is active (respecting activation delay).
Key with kid "88CF32355BAE0F4E0DD9E73BFA6E232E" is active.
Active signing key found (respecting the activation delay) with kid: "88CF32355BAE0F4E0DD9E73BFA6E232E".
Found active signing key for alg "RS256" with kid "88CF32355BAE0F4E0DD9E73BFA6E232E".
Checking if key with kid "88CF32355BAE0F4E0DD9E73BFA6E232E" is active (respecting activation delay).
Key with kid "88CF32355BAE0F4E0DD9E73BFA6E232E" is active.
Active signing key found (respecting the activation delay) with kid: "88CF32355BAE0F4E0DD9E73BFA6E232E".
Key rotation not required for alg "RS256"; New key expected to be created in 11.20:59:48

Below is the jwks endpoint result. The second key is registered manually, not a part of the key management. The key with id "88CF32355BAE0F4E0DD9E73BFA6E232E" expired on 24th of June 2023. No other keys are registered.

{
  "keys": [
    {
      "kty": "RSA",
      "use": "sig",
      "kid": "88CF32355BAE0F4E0DD9E73BFA6E232E",
      "x5t": "vj9T2ohOOVkNBXUhdsziVWRnKHI",
      "e": "AQAB",
      "n": "ssWj...YvJEw",
      "x5c": [
        "MIIDDD...eDCQ=="
      ],
      "alg": "RS256"
    },
    {
      "kty": "RSA",
      "use": "sig",
      "kid": "AA9FD220B3D04F4F5E1B6344A47EA9CB2C379E57RS256",
      "x5t": "qp_SILPQT09eG2NEpH6pyyw3nlc",
      "e": "AQAB",
      "n": "oUAB...u-2Q",
      "x5c": [
        "MIIDmD...0TtE3dR"
      ],
      "alg": "RS256"
    }
  ]
}

More cert details:

Valid From: 18 May 2023, 8:59 a.m.
Valid To: 24 Jun 2023, 8:59 a.m.

Previous key management config:

		options.KeyManagement.Enabled = true;
		options.KeyManagement.RotationInterval = TimeSpan.FromDays(30);
		options.KeyManagement.PropagationTime = TimeSpan.FromDays(7);
		options.KeyManagement.RetentionDuration = TimeSpan.FromDays(7);
		options.KeyManagement.DeleteRetiredKeys = true;
		options.KeyManagement.SigningAlgorithms = new[]
		{
			new SigningAlgorithmOptions(SecurityAlgorithms.RsaSha256) { UseX509Certificate = true },
			new SigningAlgorithmOptions(SecurityAlgorithms.RsaSsaPssSha256),
			new SigningAlgorithmOptions(SecurityAlgorithms.EcdsaSha256)
		};

Please let me know if you need more details. Thank you.

[ivan-sedlak-visma]

I think I have found the bug in KeyManager.cs

It happens because the rotation interval was changed to a higher value and the diff is bigger than the propagation interval. In our particular case, the issue should fix itself in approximately 12 days and as long as we don't change the key management config anymore it should keep working as expected from that point on. I guess the proper fix would be to keep the config alongside the generated certificate and check against those values instead of the current ones. I am not sure that is the best strategy so I don't want to create any PRs at this point in time to suggest a bugfix.

Here is the code snippet to reproduce:

		DateTime created = DateTime.Parse("2023-05-18");
		TimeSpan interval = TimeSpan.FromDays(90);
		TimeSpan propagation = TimeSpan.FromDays(14);
		DateTime now = DateTime.Parse("2023-07-21");

		TimeSpan age = now.Subtract(created);
		TimeSpan diff = interval.Subtract(age);
		bool needed = (diff <= propagation);

		Console.WriteLine($"Age: {age}");
		Console.WriteLine($"Diff: {diff}");
		Console.WriteLine($"Needed? {needed}");

		if (!needed) {
		  Console.WriteLine("Key rotation not required for alg {0}; New key expected to be created in {1}", "ALG", diff.Subtract(propagation));
		} else {
		  Console.WriteLine("Key rotation required now for alg {0}.", "ALG");
		}

Result:

Age: 64.00:00:00
Diff: 26.00:00:00
Needed? False
Key rotation not required for alg ALG; New key expected to be created in 12.00:00:00

[josephdecock]

Signing keys never really expire. They can be stored in an x.509 certificate and that certificate might have an expiration date on it, but IdentityServer isn't validating any of the x.509 certificate details. It's just using the certificate as a container for the key material, and will happily use a certificate regardless of expiration. This is a common point of confusion, and I can see why folks would think "oh, my certificate is about to expire, I have a problem". But as I say, that's not how IdentityServer uses certificates.

Given this, do you still think there is a bug?

[ivan-sedlak-visma]

Signing keys never really expire. They can be stored in an x.509 certificate and that certificate might have an expiration date on it, but IdentityServer isn't validating any of the x.509 certificate details. It's just using the certificate as a container for the key material, and will happily use a certificate regardless of expiration. This is a common point of confusion, and I can see why folks would think "oh, my certificate is about to expire, I have a problem". But as I say, that's not how IdentityServer uses certificates.

Given this, do you still think there is a bug?

In that context, the behavior seems valid, but then why bother with creating a certificate with precise expiration dates? Using an expired certificate might trigger some clients to reject it? Idk seems a bit backward to me but if you think this is expected behavior please feel free to close the issue. Thank you.

[josephdecock]

In that context, the behavior seems valid, but then why bother with creating a certificate with precise expiration dates?

There's not a lot of benefit to creating x509 certificates at all. From a security point of view in OIDC, there's no benefit at all, but saml requires it, and sometimes corporate IT policies will require that as well.

Using an expired certificate might trigger some clients to reject it?

We haven't run into this in practice. Clients in general trust the public key material because they authenticate the server using TLS (they got it over https, so it has to be legit). If you run into a client that is rejecting these certs, we'd be interested to hear about it, and could consider adding additional checks to the rotation based on the expiration of existing certs. In an unusual circumstance like that, there's always the option of manual key management as well. For now, I'll close this issue. Thanks for all your thoughts and feedback!

[DenL]

Found this when researching an issue we're facing. I want to raise that Microsoft's default token signing key validator does check for lifetime of the signing key's certificate, please see here: https://github.com/AzureAD/azure-activedirectory-identitymodel-extensions-for-dotnet/blob/dev/src/Microsoft.IdentityModel.Tokens/Validation/Validators.IssuerSigningKey.cs#L68

[josephdecock]

@DenL thanks for your comment. Can you expand a bit more about the issue you're seeing? I just tried getting IdentityServer to generate a key using x.509 and a very short lifetime. Then I changed IdentityServer configuration so that the key with the expired certificate would continue to be used, and then was able to use that key to log in to a client using OpenId Connect built using the ASP.NET Core OIDC handler. At least historically, the typical OAuth and OpenId Connect related code paths haven't run into this check that you've found, so I'm curious what your use case is, and how you're using Microsoft.IdentityModel to cause an issue.

[DenL]

@josephdecock we use the key to sign a jwt token. That token is used by client elsewhere to authenticate with another app. The app is using jwtbearer to authenticate with TokenValidationParameters.ValidateIssuerSigningKey = true

I’m on mobile so apologize for the lack of formatting

[josephdecock]

Okay, interesting. And what's the goal of setting that flag in the other app? If you control that app, why not just disable that flag? In general, clients can trust the public key material that IdentityServer serves from the jwks endpoint because the authorization server was authenticated by TLS.

[DenL]

That was an info sec decision made before my time. I’m guessing it’s to ensure tokens are authentic and actually served by our identity server.

[josephdecock]

My advice is to revisit that old info sec decision. There really isn't a security benefit to it in OIDC. You can have confidence in the authenticity of the tokens because you can validate the signature of tokens using public key material that was retrieved over TLS. In other words, you know that you got the JWK from your-identity-server.example.com because it was sent over HTTPS, so only an entity that controls the TLS certificate for that domain could do so.

[DenL]

We already manually deployed a new valid cert and resolved our issue that way. I just wanted to bring this to your attention as I imagine it’s a reasonable expectation that the public key container cert is still valid. I’m not a security expert but there could be many reasons to enforcing validity. In our case one of the reasons I’m aware of is to force a rotation of keys as having one set valid forever means it could eventually be compromised, and that does not meet compliance standards in our industry.

…

On Mon, Feb 3, 2025 at 4:32 PM Joe DeCock ***@***.***> wrote: My advice is to revisit that old info sec decision. There really isn't a security benefit to it in OIDC. You can have confidence in the authenticity of the tokens because you can validate the signature of tokens using public key material that was retrieved over TLS. In other words, you know that you got the JWK from your-identity-server.example.com because it was sent over HTTPS, so only an entity that controls the TLS certificate for that domain could do so. — Reply to this email directly, view it on GitHub <DuendeArchive/Support#764 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABK7BWZXFKJJHSIOT6SKLOL2N7N5XAVCNFSM6AAAAABVTGALPGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMZSGEZDCOJVGI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[maartenba]

(note: we're moving this issue to our new community discussions)