JWKS and Signing Key Rotation Runbook: Prevent JWT Verification Failures

Overview

A JSON Web Key Set (JWKS) endpoint publishes the public keys that relying parties use to verify tokens signed by an authorization server or application. The private signing key must remain protected by the issuer; only the corresponding public key belongs in the JWKS document.

During a normal JWT signing key rotation, the issuer should publish the new public key first, begin signing new tokens with it, and continue publishing the previous public key until tokens signed with that key can no longer be accepted. This overlap prevents a verifier from receiving a valid token whose signing key has already disappeared from the JWKS response.

The kid (key ID) header helps a verifier select the correct public key. A kid mismatch can result from an incorrectly deployed key, duplicate IDs, stale JWKS caches, an issuer using a different key set than expected, or a token routed to the wrong environment. A rotation plan should address each possibility before production changes begin.

Use this runbook alongside a controlled token policy. Define the maximum token lifetime, JWKS cache behavior, clock tolerance, rollback method, and owner for each environment. For safe token inspection, see the JWT decoder and debugging guide, but do not paste production credentials or sensitive tokens into an unapproved tool.

Checklist by scenario

Before any planned rotation

• Record the current issuer URL, JWKS URL, signing algorithm, active kid, and accepted audiences.
• Confirm that every verifier validates the expected issuer, audience, signature algorithm, token time claims, and key use. Key discovery must not replace claim validation.
• Inventory token lifetimes and refresh-token behavior. The overlap period should cover the longest time an old access token can still be accepted, plus a documented allowance for cache and clock differences.
• Check whether verifiers cache JWKS responses, how they refresh them, and whether they retry after an unknown kid. Avoid assuming all libraries behave alike.
• Generate the new key in an approved key-management location. Restrict private-key access, record ownership, and confirm that backups and recovery procedures are appropriate.
• Choose a unique, stable kid. Do not reuse an old ID for a different key.
• Test the complete process in staging with production-like caching, clock settings, token lifetimes, and deployment topology.

For a standard planned rotation

1. Publish the new public JWK in the JWKS response while retaining the old public JWK.
2. Wait for the expected distribution and cache period, or verify that representative verifiers can retrieve the new key.
3. Switch the issuer to sign newly issued tokens with the new private key and new kid.
4. Monitor token issuance, verification errors, unknown-key events, authentication success rates, and refresh flows.
5. Keep the old public key available through the overlap window. Do not remove it merely because new tokens use the replacement key.
6. After the window, remove the old public JWK, verify that no supported token should still depend on it, and retain an auditable record of the change.

For an emergency or suspected key compromise

• Follow the incident-response plan rather than the normal overlap schedule. The priority may be to stop accepting tokens signed by the affected key.
• Disable or revoke the compromised private key where the platform supports that control, and issue a replacement key with a new kid.
• Publish only the keys that should remain trusted. Removing a key can cause a broad outage, so coordinate the change with application and platform owners.
• Assess refresh tokens, sessions, downstream services, logs, and any credentials that may have been exposed. Key rotation alone may not invalidate already issued tokens unless verifiers enforce a revocation or denylist strategy.
• Document the timeline, affected environments, verification behavior, and follow-up improvements.

Provider and deployment comparison

The operational controls differ by where signing and key publication are managed. Use this comparison to identify questions for your platform owner; it is not a ranking of providers.

Implementation template

Record the following before approving a change:

Environment: staging | production
Issuer: https://identity.example
JWKS URL: https://identity.example/.well-known/jwks.json
Current kid: <old-key-id>
Replacement kid: <new-key-id>
Algorithm: <approved-algorithm>
Maximum accepted token lifetime: <duration>
JWKS cache and refresh allowance: <duration>
Clock-skew allowance: <duration>
Overlap ends: <timestamp>
Change owner: <team or person>
Rollback: restore signing configuration; keep both public keys available
Success signals: new tokens verify; old tokens behave as expected; no unknown-kid spike

What to double-check

• JWKS content: Confirm the new JWK contains the required public parameters, the intended algorithm and key use, and a unique kid. Never expose private parameters.
• Issuer routing: Ensure the token's iss value maps to the intended JWKS endpoint. A valid signature from another issuer is not automatically acceptable.
• Algorithm enforcement: Configure verifiers with an explicit allowlist. Do not infer trust solely from the token header.
• Cache behavior: A verifier may continue using an old response, while another may request the new set immediately. Test both paths and make the endpoint highly available.
• Time validation: Check server time synchronization and the treatment of exp, nbf, and iat. Small clock differences can look like signing failures.
• Rollback: A rollback should restore the issuer's signing configuration without deleting the replacement public key prematurely. Decide whether tokens issued during the failed deployment remain trusted.
• Observability: Log the issuer, kid, verification outcome, and failure category without logging raw tokens or sensitive claims unnecessarily.

Incident troubleshooting flow

1. Capture the failure category and timestamp. Separate signature failure, unknown kid, expired token, invalid issuer, invalid audience, and unavailable JWKS endpoint.
2. Inspect the token header and claims in a controlled environment. Confirm the algorithm, kid, issuer, audience, and time values.
3. Fetch the JWKS response from the failing verifier's network location. Compare its keys with the issuer's intended configuration.
4. If the kid is absent from a stale response, use the library's supported refresh mechanism. Avoid disabling signature validation or accepting arbitrary keys as a workaround.
5. Compare system clocks, region-specific configuration, DNS, proxies, and environment variables.
6. After remediation, test both newly issued tokens and, where appropriate, tokens signed by the previous key during the overlap window.

Common mistakes

• Removing the old key too early: This breaks verifiers that still need to validate unexpired tokens or have a cached response.
• Publishing after signing: If new tokens appear before the replacement public key is discoverable, verification can fail immediately. Publish first, then switch signing.
• Reusing a kid: A reused identifier can cause caches and key-selection logic to associate a token with the wrong key.
• Using one JWKS set for every environment: Shared trust can allow a token from the wrong environment to pass an incomplete validation policy.
• Refreshing on every request: This can create unnecessary dependency on the JWKS endpoint and amplify an outage. Use bounded caching and a safe refresh path instead.
• Treating a JWT decoder as a verifier: Decoding reveals structure; it does not prove authenticity. Signature and claim checks must occur in the application or a trusted validation component.
• Relying on rotation as revocation: A new signing key does not necessarily invalidate tokens already issued under the old key. Define separate session and revocation controls.

For broader developer identity tooling, the identity tool stack guide covers related controls such as SSO, MFA, passkeys, SCIM, and audit logs. For integrity-focused workflows, review the hash generator and checker guide.

When to revisit

Use this runbook before every planned signing-key change, new environment rollout, authorization-server migration, major verifier-library upgrade, or change to token lifetime and caching. Revisit it after a JWT verification incident, suspected key exposure, region or gateway redesign, or change in ownership of the identity platform.

At least during each regular planning cycle, confirm that the documented JWKS URL, active kid, algorithms, overlap calculation, cache assumptions, monitoring, and rollback steps still match production. Rehearse the procedure in staging when workflows or tools change. End each review with one practical action: update the rotation record, test the emergency path, remove obsolete keys, or assign an owner for an unresolved gap.

A reliable rotation is not just a cryptographic event. It is a coordinated change across issuers, caches, verifiers, clocks, deployment regions, and incident responders. Keeping those dependencies visible is the simplest way to prevent a routine key change from becoming an authentication outage.