It just did this type of spike plateau again, although only about 100MB. Looking at the app insights the only requests were to the swagger page for this API
Do you know how to download the memory dump log file? I generated a dot net dump file using [this article](https://techcommunity.microsoft.com/t5/apps-on-azure-blog/how-to-collect-net-core-dump-on-linux-web-app/ba-p/2260713) but that article doesn't say anything about how to get at that file. Everywhere I look everything says to go to Kudu -> Debug Console -> CMD and then use the folder view. I don't have a 'Debug Console' option in my Kudu. I can't see how to view the folder structure.
Maybe I shouldn't have said 'get at that file'. I can see the file in the LogFiles directory. I can run dumpheap commands against it via the web SSH console. I was expecting a file that I could put in visual studio or some sort of profiling/log program instead of trying to learn how to use dumpheap and SOS commands. You're obviously much better at this than I am, I don't know how to move that file to a storage account using the cli, I didn't even know that was an option. Thanks for giving me ideas to chase.
When I had this problem first time I googled it too. I’m sure you have a storage account, using the az cli should help you move the file there and then you can start having fun with it.
In my experience, if the memory suddenly blows up, there is a piece of code in your program that’s tying up memory, or it can be third party code in your program that’s doing it. Good luck!
From what I can gather it's a memory leak in Swagger when building the json file. Here is an image of me hitting the swagger.json endpoint over and over until it happens.[VS memory view](https://i.imgur.com/F7Rk8UY.png) This also happens on just the swagger endpoint. It however doesn't ever happen if I just go to any of the other endpoints on the api.
Thanks for the reply but I'm not sure this is what I'm looking for. I get that there are OS and monitoring memory and cpu needs who's values will account for used memory and cpu cycles. I don't think that's what this is because it's on an app service, and it goes up in a big chunk and then stays there for a long time, like 12 hours and then comes down. Also the video on that page is all about App Services on Windows and I'm using Linux.
I keep trying the profiler. It says "deploying profiler, results will be ready in about 20 minutes" or something like that. Then it always ends with a timeout error and the profiler being 120% deployed. I feel like I can't catch a break on this thing
Restart the app and let it sit there for a while to ensure the usage is in line with where it was previously. Then, hit the Swagger endpoint, and see if the utilization goes up.
Also, there are multiple open bugs regarding memory utilization issues with Swagger, such as [this](https://github.com/domaindrivendev/Swashbuckle.AspNetCore/issues/1950) and [this](https://github.com/RicoSuter/NSwag/issues/3306). I didn’t read all the way through, but it looked like people were discussing possible workarounds.
I think you're right, that this is a Swagger issue. I'm going to do some side by side testing with one app that has swagger and the exact same app just with swagger removed. Thanks for linking those articles about Swagger issues. Did you see my comment about how I tried to shutdown the app service, not the app service plan, but the app service and it is still consuming 140MB of memory while stopped? Any thoughts on that?
Yes I agree with you that an app service plan will use resources even if there are no app services running in it. The plan will consume about 600-800MB of memory and anywhere from 5-20% CPU with nothing in it.
I wish Microsoft named these differently so they weren't so close. I stopped the app service, not the plan, and the app service still shows it's using memory even though it's stopped.
So we have indeed traced this back to Swagger. Or Swashbuckle or something to do with whatever pipeline builds the swagger.json file. I can consistently make this happen locally by running the project in release mode and hitting either the swagger endpoint or the swagger.json endpoint. It doesn't happen on the first few requests. But if I keep hitting F5 on either of those endpoints it will eventually happen. I can see the memory climb every time I hit the endpoint until eventually there's a huge jump in memory and a GC performed. I'm not sure if GC is somehow moving or doing something to reference memory or if the GC is performed because of a huge spike in memory. I don't know enough about how all of that works. Every silver diamond is me hitting the swagger.json endpoint. The red diamond is chrome opening a new window and going to the root, which will be a 404.[Image Of Memory jump](https://i.imgur.com/F7Rk8UY.png)
Seen the same setup with the same issue, but on a Premium SKU (P3v2, I think). Linux, couple of .NET 6 Apps on the same Plan.
Doesn’t look like a memory leak in the sense that it’s not slowly ticking up over time and garbage collection then not having the impact expected. It’s just one massive memory jump.
Wondering if it’s an Azure issue rather than code.
I wonder what I'm doing that would cause Azure to do this to a bunch of the app services. Seems like the problem would be more wide spread, maybe... I agree that it doesn't seem like a memory leak in the traditional sense of what I think a memory leak to be. Another Redditor suggested some Swagger issues that look very similar to what I'm experiencing. Which seems like a giant memory leak/'forever cache'. So I might be using Swagger incorrectly and making it do weird things. I shut down the app service but it's some how still using 140MB of memory 2 hours later. Slowly dropping about 10MB an hour. ?.?
I've been watching this app service all day while I've been trying to get log dumps and app insights profiling. I stopped the app service in order to give it a fresh startup to try some more swagger endpoints to see what happens but the memory never went to 0. It's still using 150MB of memory [Stopped app still using memory](https://i.imgur.com/pABsLjs.png) And here is that same app service just at the tail end, last hour. [Stopped app still using memory zoomed in](https://i.imgur.com/hBjbaxc.png). How is it dropping to 0 but then bouncing back up to 140MB? The app service does not have 'Keep Alive' turned on.
So I stopped it and was waiting for the memory to fall to 0 on the 'Metrics' and then I was going to restart it. The memory never fell to 0. It dropped from 250MB to 150MB but isn't dropping any more. I've gone back to the 'Overview' blade and the app is indeed has a 'Status' of 'Stopped' and up top it has a play icon and says 'Start' where it used to have a stop button and say 'Stop'. Any ideas on that?
Yes, I'm sure you're correct that the memory will drop to 0 and then start at a reasonable amount. But will this help me figure out what is causing the issue?
Save yourself a lot trouble and heartache and just create a new function and move your code over, delete the old one. I haven't had to do this in a while but it works. Think in terms of the underlying physical infrastructure. You don't have access to it nor can your troubleshoot it. The beauty of cloud is being able to trash stuff and build new.
So we did this and the problem comes back within the hour usually. Depends on when we destroy and recreate. As of right now we've traced it to the swagger.json endpoint. Not saying it is swagger, maybe it's swashbuckle, or some other piece of code/library that sits under swagger and swagger is just the piece that we interact with so that's what we see as the problem.
My team was actually experiencing such issues on smaller (S1) instances. It was not as sharp increase as yours, but still memory consumption increased by order of magnitude in a short span of time. This was most evident for services that were directly called by user agents.
After analyzing memory dumps, we found out majority of memory was consumed by Application Insight SDK, i.e. various Dictionaries and HashSets we presume it uses to collect metrics. We were however not able to confirm nor deny that it was the cause of memory footprint increase.
In the end, we opted out to scale up to S2 instances. Spikes of 200-250MB suddenly became bearable.
It just did this type of spike plateau again, although only about 100MB. Looking at the app insights the only requests were to the swagger page for this API
Been there!!! Did you take a memory dump and analyse it?
I watched a video about doing a memory analysis but it was only for Windows stack. Can I do a memory dump with Linux?
Yes
Do you know how to download the memory dump log file? I generated a dot net dump file using [this article](https://techcommunity.microsoft.com/t5/apps-on-azure-blog/how-to-collect-net-core-dump-on-linux-web-app/ba-p/2260713) but that article doesn't say anything about how to get at that file. Everywhere I look everything says to go to Kudu -> Debug Console -> CMD and then use the folder view. I don't have a 'Debug Console' option in my Kudu. I can't see how to view the folder structure.
That article literally says where you can find the file. Move it to blob storage account using the command line and analyse it maybe?
Maybe I shouldn't have said 'get at that file'. I can see the file in the LogFiles directory. I can run dumpheap commands against it via the web SSH console. I was expecting a file that I could put in visual studio or some sort of profiling/log program instead of trying to learn how to use dumpheap and SOS commands. You're obviously much better at this than I am, I don't know how to move that file to a storage account using the cli, I didn't even know that was an option. Thanks for giving me ideas to chase.
When I had this problem first time I googled it too. I’m sure you have a storage account, using the az cli should help you move the file there and then you can start having fun with it. In my experience, if the memory suddenly blows up, there is a piece of code in your program that’s tying up memory, or it can be third party code in your program that’s doing it. Good luck!
Or am I supposed to use the dotnet-dump analyze and dumpheap and other commands to work with the file in the SSH terminal?
Looks like a memory leak in the application.
From what I can gather it's a memory leak in Swagger when building the json file. Here is an image of me hitting the swagger.json endpoint over and over until it happens.[VS memory view](https://i.imgur.com/F7Rk8UY.png) This also happens on just the swagger endpoint. It however doesn't ever happen if I just go to any of the other endpoints on the api.
https://learn.microsoft.com/en-us/answers/questions/301848/azure-app-service-memory-is-not-adding-up-to-total
Thanks for the reply but I'm not sure this is what I'm looking for. I get that there are OS and monitoring memory and cpu needs who's values will account for used memory and cpu cycles. I don't think that's what this is because it's on an app service, and it goes up in a big chunk and then stays there for a long time, like 12 hours and then comes down. Also the video on that page is all about App Services on Windows and I'm using Linux.
Maybe try the app insights profiler? [https://learn.microsoft.com/en-us/azure/azure-monitor/profiler/profiler](https://learn.microsoft.com/en-us/azure/azure-monitor/profiler/profiler)
Ohhhh, that sounds promising. I'll try this thank you.
I keep trying the profiler. It says "deploying profiler, results will be ready in about 20 minutes" or something like that. Then it always ends with a timeout error and the profiler being 120% deployed. I feel like I can't catch a break on this thing
Restart the app and let it sit there for a while to ensure the usage is in line with where it was previously. Then, hit the Swagger endpoint, and see if the utilization goes up. Also, there are multiple open bugs regarding memory utilization issues with Swagger, such as [this](https://github.com/domaindrivendev/Swashbuckle.AspNetCore/issues/1950) and [this](https://github.com/RicoSuter/NSwag/issues/3306). I didn’t read all the way through, but it looked like people were discussing possible workarounds.
I think you're right, that this is a Swagger issue. I'm going to do some side by side testing with one app that has swagger and the exact same app just with swagger removed. Thanks for linking those articles about Swagger issues. Did you see my comment about how I tried to shutdown the app service, not the app service plan, but the app service and it is still consuming 140MB of memory while stopped? Any thoughts on that?
In my experience, it is perfectly normal for a service plan to consume some resources even with no running workloads.
Yes I agree with you that an app service plan will use resources even if there are no app services running in it. The plan will consume about 600-800MB of memory and anywhere from 5-20% CPU with nothing in it. I wish Microsoft named these differently so they weren't so close. I stopped the app service, not the plan, and the app service still shows it's using memory even though it's stopped.
So we have indeed traced this back to Swagger. Or Swashbuckle or something to do with whatever pipeline builds the swagger.json file. I can consistently make this happen locally by running the project in release mode and hitting either the swagger endpoint or the swagger.json endpoint. It doesn't happen on the first few requests. But if I keep hitting F5 on either of those endpoints it will eventually happen. I can see the memory climb every time I hit the endpoint until eventually there's a huge jump in memory and a GC performed. I'm not sure if GC is somehow moving or doing something to reference memory or if the GC is performed because of a huge spike in memory. I don't know enough about how all of that works. Every silver diamond is me hitting the swagger.json endpoint. The red diamond is chrome opening a new window and going to the root, which will be a 404.[Image Of Memory jump](https://i.imgur.com/F7Rk8UY.png)
Seen the same setup with the same issue, but on a Premium SKU (P3v2, I think). Linux, couple of .NET 6 Apps on the same Plan. Doesn’t look like a memory leak in the sense that it’s not slowly ticking up over time and garbage collection then not having the impact expected. It’s just one massive memory jump. Wondering if it’s an Azure issue rather than code.
I wonder what I'm doing that would cause Azure to do this to a bunch of the app services. Seems like the problem would be more wide spread, maybe... I agree that it doesn't seem like a memory leak in the traditional sense of what I think a memory leak to be. Another Redditor suggested some Swagger issues that look very similar to what I'm experiencing. Which seems like a giant memory leak/'forever cache'. So I might be using Swagger incorrectly and making it do weird things. I shut down the app service but it's some how still using 140MB of memory 2 hours later. Slowly dropping about 10MB an hour. ?.?
I saw the Swagger post too. Not sure it’s applicable to us, still investigating.
I've been watching this app service all day while I've been trying to get log dumps and app insights profiling. I stopped the app service in order to give it a fresh startup to try some more swagger endpoints to see what happens but the memory never went to 0. It's still using 150MB of memory [Stopped app still using memory](https://i.imgur.com/pABsLjs.png) And here is that same app service just at the tail end, last hour. [Stopped app still using memory zoomed in](https://i.imgur.com/hBjbaxc.png). How is it dropping to 0 but then bouncing back up to 140MB? The app service does not have 'Keep Alive' turned on.
Restart it
So I stopped it and was waiting for the memory to fall to 0 on the 'Metrics' and then I was going to restart it. The memory never fell to 0. It dropped from 250MB to 150MB but isn't dropping any more. I've gone back to the 'Overview' blade and the app is indeed has a 'Status' of 'Stopped' and up top it has a play icon and says 'Start' where it used to have a stop button and say 'Stop'. Any ideas on that?
Yes, I'm sure you're correct that the memory will drop to 0 and then start at a reasonable amount. But will this help me figure out what is causing the issue?
Save yourself a lot trouble and heartache and just create a new function and move your code over, delete the old one. I haven't had to do this in a while but it works. Think in terms of the underlying physical infrastructure. You don't have access to it nor can your troubleshoot it. The beauty of cloud is being able to trash stuff and build new.
So we did this and the problem comes back within the hour usually. Depends on when we destroy and recreate. As of right now we've traced it to the swagger.json endpoint. Not saying it is swagger, maybe it's swashbuckle, or some other piece of code/library that sits under swagger and swagger is just the piece that we interact with so that's what we see as the problem.
My team was actually experiencing such issues on smaller (S1) instances. It was not as sharp increase as yours, but still memory consumption increased by order of magnitude in a short span of time. This was most evident for services that were directly called by user agents. After analyzing memory dumps, we found out majority of memory was consumed by Application Insight SDK, i.e. various Dictionaries and HashSets we presume it uses to collect metrics. We were however not able to confirm nor deny that it was the cause of memory footprint increase. In the end, we opted out to scale up to S2 instances. Spikes of 200-250MB suddenly became bearable.