We've got Umbraco hosted in Azure, in a load-balanced environment with deployment slots. We're seeing a lot of warnings in the logs. Umbraco Version 13.0.1 W : Error unprotecting the session cookie. Stack Trace : System.Security.Cryptography.CryptographicException: The key {078a31d2-3d8c-4b43-baeb-a639524b05c5} was not found in the key ring. For more information go to https://aka.ms/aspnet/dataprotectionwarning at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.UnprotectCore(Byte[] protectedData, Boolean allowOperationsOnRevokedKeys, UnprotectStatus& status) at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.Unprotect(Byte[] protectedData) at Microsoft.AspNetCore.Session.CookieProtection.Unprotect(IDataProtector protector, String protectedText, ILogger logger) We also see a rare instance of an error, but this seems much less frequent Microsoft.AspNetCore.Antiforgery.AntiforgeryValidationException: The antiforgery token could not be decrypted. Stacktrace : System.Security.Cryptography.CryptographicException: The key {84cea658-5bcd-439a-b175-d1c4bb7956f7} was not found in the key ring. For more information go to https://aka.ms/aspnet/dataprotectionwarning at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.UnprotectCore(Byte[] protectedData, Boolean allowOperationsOnRevokedKeys, UnprotectStatus& status) at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.Unprotect(Byte[] protectedData) at Microsoft.AspNetCore.Antiforgery.DefaultAntiforgeryTokenSerializer.Deserialize(String serializedToken) --- End of inner exception stack trace --- at Microsoft.AspNetCore.Antiforgery.DefaultAntiforgeryTokenSerializer.Deserialize(String serializedToken) at Microsoft.AspNetCore.Antiforgery.DefaultAntiforgery.GetCookieTokenDoesNotThrow(HttpContext httpContext) Anyone seen this issue, or similar?

Yes this is quite common, or at least something I see quite often when developing locally It essentially means that the specific key was not found in the key ring, which points to an issue with the data protection mechanism in ASP.NET Core. The data protection system uses a set of keys to encrypt and decrypt data, including antiforgery tokens. If the expected key is missing from the key ring, the system cannot decrypt the token, which is what this error says. It's a pretty common error to see when the application pool is recycling or when the server restarts. For you since your working in a Load-balanced environment where a request from a user can be handled by different servers it most likely means that the key ring is not synchronized across all servers. Meaning a server might receive a token that was encrypted by a key that server does not have. How to solve this: Configure the data protection system to persist keys in a central location that all instances of the application can access. Could be a Azure storage blob or Redis, if it works with an azure keyvault thats what i would use. For the loadbalancing solution the solution is pretty much the same: Central storage of the keys, and make sure you sync between all environments. Then point it out in your startup.cs or program.cs (depending on .net version) where the keys are located

.net
// psudo-code
services.AddDataProtection().PersistKeysToFileSystem(new DirectoryInfo(@"\\path\to\your\keys\"));

Yes, def. Currently seeing both of them a lot on a new, live 13.2.2 site.

The "Data Protection system" i'am referring to, in this context is the ASP.NET Core built-in feature designed to protect sensitive data. It's the service that automatically encrypts and decrypts data like auth tokens, antiforgery tokens, and session states with cryptographic keys. By default, the data protection system in .Net stores the keys in memory. So for example, on a single server, without any additional configuration, the keys used by the data protection system are stored in memory. This means that once the server is rebooted or the application pool is recycled, the keys stored in memory are lost and not automatically available upon restart. New keys are generated when the application starts again, which can lead to issues like failed decryption of data that was encrypted with the old keys. The in-memory solution is a quite effective especially when the app doesn't need to scale and is rarly restarted. The default settings also automatically generates, rotates, and retires cryptographic keys. A new key is added to the key ring at regular intervals, if i recall correctly by default, every 90 days, and the key is used to protect data operations. If you want to get rid of the error in the log you should not use the default in-memory version but instead save the persistant keys to a file system (Like the central storage solution)

Thanks for the help with the @2lach - If we're persisting keys to the filesystem, then won't it have the same problem if therequest goes to a different node?

yes that is correct, thats why you need a central storage solution where you save all the keys and sync them between all nodes

Could we use bllo stoage for this then?

I would use Azure key vault as a first option (but i haven't done that so not sure how the implementation would look) I would rather use Azure blob storage than the db

ok - I'll dig around and see if I can use key vault or blob storage for this - thanks for your help

My pleasure

I'm guessing from your prev message this is more an annoying problem than necessarily a show stopper

On a single node yes, but if it's a reccuring problem between your nodes that happens when they have been running for a while i would build a solution for it since it means data is not reaching it's expected endpoint

yeah - ok

Whenever we have more than one node running we make sure to setup a blob store and swap the keys over to be stored there.

I'm looking at an implemetation with blob storage right now

On a single node it more depends if you need sessions to persist accross app-pool recycles

or at least trying to put it together. We have a back office and a front end node, so we probably will want this

It's pretty straightforward - one line in startup or composer plus connection string in app settings / env vars

Ok - I might ping you if I can't put that together if you don't mind - unless you have the code handy?

Funnily enough I have that page open anyway for my current project 😁 https://docs.umbraco.com/umbraco-cms/fundamentals/setup/server-setup/load-balancing#data-protection

yeah - I saw ths

try if it works, but it's likely you need to implement a solution that syncs the keys between your instances

Seems to have done the trick - perhaps an old key was in memory, but that does concern me for future deployments, and slot swaps. @Andrew McKaskill did you need to implement anything to keep keys across nodes? My code in startup was just a single line once I'd loaded the blob connectionstring, and all nodes use the same containers services.AddDataProtection().PersistKeysToAzureBlobStorage(mediaConnectionString, "datakeys", "datakeys.xml");

It's probably an old session token hanging around.

Cool - I'll keep an eye out for any further instances of those errors, not seen any so far - thanks folks 🙂

Great explanation, thanks.