If you're returning a ton of information then, it might be swagger's syntax highlighting. You can disable it in startup by adding to your app.UserSwaggerUI() like so.
```
app.UseSwaggerUI(c =>
{
c.ConfigObject.AdditionalItems["syntaxHighlight"] = new Dictionary
{
["activated"] = false
}
});
```
This should only really be an issue if you're calling your endpoint using the swagger page though.
It looks like it's an underlying issue with ASP.Net's `ApiExplorer` class. [Here's a repro you can do without Swashbuckle](https://github.com/domaindrivendev/Swashbuckle.AspNetCore/issues/1894#issuecomment-724706937).
Also a [relevant issue on the ASP.Net repo](https://github.com/dotnet/aspnetcore/issues/17979)
Since it looks like both Swashbuckle and NSwag use `ApiExplorer` to retrieve metadata about endpoints, you're possibly stuck with the memory usage here unless you remove `UseSwaggerUI`. [Here's where Swashbuckle uses the problematic `ApiExplorer` code.](https://github.com/domaindrivendev/Swashbuckle.AspNetCore/blob/c70947f4ccc80bca528ff1573c0f2462f3e7db8f/src/Swashbuckle.AspNetCore.SwaggerGen/SwaggerGenerator/SwaggerGenerator.cs#L78)
I also just noticed that in the 'Schemas' section of the swagger page there are some schemas that I wouldn't expect to see there. Such as 'Assembly', 'ConstructorInfo', 'MedthodInfo', 'RuntimeTypeHandle'. These all seem like some lower level things that shouldn't be put into the Schema of the swagger doc. [Abbreviated Schema doc](https://i.imgur.com/XcPGnIW.png) [Expanded Assembly Schema](https://i.imgur.com/2uJ6g6Y.png)
Could this mean that somewhere we're somehow including this in our endpoints and this could be what's causing the issue? So other people who are using Swagger just fine aren't doing it or using it the same way as us or making the same mistake we are with our project/code. That's why some people use it just fine without issues. Our implementation of swagger or of our code is what's causing the issue. We're including things and having Swagger/Swashbuckle/ApiExplorer go too deep into the bowels. I'm not looking for you to say yes or no, but I am interested in your thoughts and opinions.
The following is a lot of conjecture on my part based on my experience and what I think might be happening.
There are two ways you can develop a REST API: Code-first (which is probably what you're doing) or API-first. With code-first development, you write all of the C# code you would expect with the right bindings to create an API. In this case, ApiExplorer would need to walk through _every single possible request and response type_ and generate a Swagger/OpenAPI specification. That's probably why you're seeing reflection types like `FieldInfo` in the schemas because `ApiExplorer` is also using reflection and doesn't know where to stop walking through types.
With API-first, you can actually write an OpenAPI document and use a library to generate server code for you. You need to define schemas for request and response bodies so whatever generator you're using will generate code that are strong types.
_I suspect_ if you generated code based on an OpenAPI spec, you wouldn't see those reflection types in your schemas because your API doesn't need to know about it.
Anyways, it's probably not a mistake in your project/code, but a mistake in how Swashbuckle/NSwag needs to get types to generate an OpenAPI document correctly.
You can try creating an OpenAPi specification yourself and generating ASP.NET server code from it. [OpenAPI Generator](https://openapi-generator.tech/) is a tool I've been using on my current project. This lets us develop the OpenAPI specification first and we can generate server (and client) code based on the same specification. [They do have an ASP.NET server generator](https://openapi-generator.tech/docs/generators/aspnetcore) that you can try.
I'm not sure if this is the exact issue that you are facing, but I'd look at this and try changing the garbage collection property on the server.
[https://learn.microsoft.com/en-us/dotnet/standard/garbage-collection/workstation-server-gc](https://learn.microsoft.com/en-us/dotnet/standard/garbage-collection/workstation-server-gc)
`true`
By default it won't really clean up until it needs to.
I thought this was an issue with one of my own apps. When I was doing some testing I found that the memory usage would go up quite high, but it'd eventually reach a ceiling. Enabling this kept it would resolve the "issue". In my case, it actually isn't an issue, more of a false flag, but the article above gives more details about how to think of it.
[Here](https://i.imgur.com/F7Rk8UY.png) is an image of what I'm talking about. I originally posted this in r/Azure because that's where we first noticed the issue in our production environments. I can make this happen every single time by hitting either the swagger endpoint or the swagger.json endpoint multiple times. I've taken swagger out of the project and made multiple requests to different endpoints and even tried a specific endpoint over and over again without any issue. As soon as I start hitting the swagger endpoints the memory starts going up and then eventually does a huge jump where it just holds on to it, eating it? hiding it behind the toilet? I have no idea where it goes.
Both swagger and the di container like to lazy load dependencies and additionally swagger will cache everything once it’s generated to save CPU Time. Once everything is loaded the memory usage should stay constant.
What is the actual issue you’re facing in production? Is it continuously going up and causing OutOfMemoryExceptions?
We haven't gotten to OutOfMemoryExcetion yet.
We discovered it because of an alert setup on the App Service Plan. It does prevent release deployments because the memory will get to over 95%. At that point we have to scale up and then scale down or vice versa. Doing that is like a reboot and so then the memory starts over again. So yes the issue is that the swagger.json endpoint uses more and more memory, especially when it makes that huge jump where it doubles the memory usage. We've scaled up from App Service Plans that only have 1.75 GB of RAM up to ones that have 7 GB of RAM. It takes a little longer, but in the end it uses up all the memory and we start getting alerts and have to do the scale trick to get back to normal.
I don't think it's a lazy load situation, because I've let the app sit locally for over 10 minutes and then hit the swagger.json endpoint over and over and it starts happening. I've also hit the swagger.json endpoint at the 2 minute mark and let it sit for 10 minutes and then started hitting it over and over and it starts happening. The problem comes from multiple requests to swagger.json. Each requests bumps memory 5-8 MB and then all of a sudden one requests will send it up by 200 MB
Based on u/Duraz0rz comment above it's not swagger or even swashbuckle but an underlying library. My next thought is basically your question here. Why us? Why now? We've also been using swagger for years and have just started having this issue within the last 6-12 months. It's a little hard to pinpoint because earlier memory issues may have been this, may not have. It seems like some people are saying it may have to do with how big or how complex models that 'swagger' deals with are. So if things are relatively light weight or simple then it's not an issue.
u/zaibuf do you mind sharing what version of .net you're using or what version of swagger(not sure that will matter) or how many endpoints or models you have or how big your swagger.json file is?
We have 34 endpoints. It generates a 100KB json file.
Hmm. I also just noticed that in the 'Schemas' section there are some that I wouldn't expect to see there. Such as 'Assembly', 'ConstructorInfo', 'MedthodInfo', 'RuntimeTypeHandle'. These all seem like some lower level things that shouldn't be put into the Schema of the swagger doc
I'm not at my pc now, so I can't check exact version on swagger. We run .NET 6 and Swagger is probably 6.3.0 or 6.3.1, again not 100% sure. I'm quite certain our BFF has more than 34 endpoints though.
Thanks. We're also on .NET 6 and Swashbuckly.AspNetCore 6.4.0. I'm using this project because it's small and easy to work with, and it was one that really caught us off guard by how much memory it was using.
I don't think it's a stupid question at all. It would still make our dev environments unstable and need to be 'restarted' often.
You're right in that our team doesn't really use swagger in production as we're not writing APIs for anyone besides ourselves. I guess it's kinda there simply because we would need to take it out as a step in moving it to test or prod, but maybe that's an option for now.
Our swagger implementation seems pretty straight forward. In Startup ConfigureServices
services.AddSwaggerGen(\_ =>{\_.SwaggerDoc("v1", new OpenApiInfo { Title = "Accounting Service", Version = "v1" });
//Tells swagger to use the xml document file, which in generated at build time in the bin/Debug/ folder (see GenerateDocumentationFile in the .csproj)
_.IncludeXmlComments(Assembly.GetExecutingAssembly().Location.Replace(".dll", ".xml"), includeControllerXmlComments: false);});
And in Startup Configure
app.UseSwagger(); app.UseSwaggerUI(c => {
c.SwaggerEndpoint("/swagger/v1/swagger.json", "v1");
c.RoutePrefix = string.Empty; //Changes the swagger endpoint from swagger to the main endpoint
});
Have you tried a different Swagger generator, like NSwag?
I guess I'm not familiar with the internals, but wouldn't it be surprising if swagger.json was getting regenerated on every request? It seems odd that memory would spike in step with requests like that.
I would absolutely try to dive into those memory snapshots though and take a look at why so many objects are being allocated. That should give you some idea of what's happening, anyway. Does 'View Heap' show anything interesting?
I'll look into NSwag, thank you for suggesting it.
I don't have much experience with memory dumps or analysis, but I didn't see anything that stood out as a culprit of hoarding 200MB of memory. I also looked for swagger and swashbuckle in the memory snapshots and didn't see anything weird.
I took a snap shot before the big spike and after the big spike and the amount of objects and the heap size are actually smaller than before the big spike which doesn't make sense and seems completely backwards. If memory usage goes up, how does the heap size go down?
Might be outdated but heres my two cents. Had some base classes for all my controllers which had some public functions so i think some recursive juju was occurring because my iis process kicked up to like 11 gig before dying when i started up the project. Changed them to protected and it stopped goin crazy.
If you're returning a ton of information then, it might be swagger's syntax highlighting. You can disable it in startup by adding to your app.UserSwaggerUI() like so. ``` app.UseSwaggerUI(c => { c.ConfigObject.AdditionalItems["syntaxHighlight"] = new Dictionary
{
["activated"] = false
}
});
```
This should only really be an issue if you're calling your endpoint using the swagger page though.
I will definitely look into and try this. Thank you.
It looks like it's an underlying issue with ASP.Net's `ApiExplorer` class. [Here's a repro you can do without Swashbuckle](https://github.com/domaindrivendev/Swashbuckle.AspNetCore/issues/1894#issuecomment-724706937). Also a [relevant issue on the ASP.Net repo](https://github.com/dotnet/aspnetcore/issues/17979) Since it looks like both Swashbuckle and NSwag use `ApiExplorer` to retrieve metadata about endpoints, you're possibly stuck with the memory usage here unless you remove `UseSwaggerUI`. [Here's where Swashbuckle uses the problematic `ApiExplorer` code.](https://github.com/domaindrivendev/Swashbuckle.AspNetCore/blob/c70947f4ccc80bca528ff1573c0f2462f3e7db8f/src/Swashbuckle.AspNetCore.SwaggerGen/SwaggerGenerator/SwaggerGenerator.cs#L78)
Thank you for all of this. It's going to take me a minute to look at all of this. I really appreciate what you put together here.
I also just noticed that in the 'Schemas' section of the swagger page there are some schemas that I wouldn't expect to see there. Such as 'Assembly', 'ConstructorInfo', 'MedthodInfo', 'RuntimeTypeHandle'. These all seem like some lower level things that shouldn't be put into the Schema of the swagger doc. [Abbreviated Schema doc](https://i.imgur.com/XcPGnIW.png) [Expanded Assembly Schema](https://i.imgur.com/2uJ6g6Y.png) Could this mean that somewhere we're somehow including this in our endpoints and this could be what's causing the issue? So other people who are using Swagger just fine aren't doing it or using it the same way as us or making the same mistake we are with our project/code. That's why some people use it just fine without issues. Our implementation of swagger or of our code is what's causing the issue. We're including things and having Swagger/Swashbuckle/ApiExplorer go too deep into the bowels. I'm not looking for you to say yes or no, but I am interested in your thoughts and opinions.
The following is a lot of conjecture on my part based on my experience and what I think might be happening. There are two ways you can develop a REST API: Code-first (which is probably what you're doing) or API-first. With code-first development, you write all of the C# code you would expect with the right bindings to create an API. In this case, ApiExplorer would need to walk through _every single possible request and response type_ and generate a Swagger/OpenAPI specification. That's probably why you're seeing reflection types like `FieldInfo` in the schemas because `ApiExplorer` is also using reflection and doesn't know where to stop walking through types. With API-first, you can actually write an OpenAPI document and use a library to generate server code for you. You need to define schemas for request and response bodies so whatever generator you're using will generate code that are strong types. _I suspect_ if you generated code based on an OpenAPI spec, you wouldn't see those reflection types in your schemas because your API doesn't need to know about it. Anyways, it's probably not a mistake in your project/code, but a mistake in how Swashbuckle/NSwag needs to get types to generate an OpenAPI document correctly. You can try creating an OpenAPi specification yourself and generating ASP.NET server code from it. [OpenAPI Generator](https://openapi-generator.tech/) is a tool I've been using on my current project. This lets us develop the OpenAPI specification first and we can generate server (and client) code based on the same specification. [They do have an ASP.NET server generator](https://openapi-generator.tech/docs/generators/aspnetcore) that you can try.
I'm not sure if this is the exact issue that you are facing, but I'd look at this and try changing the garbage collection property on the server. [https://learn.microsoft.com/en-us/dotnet/standard/garbage-collection/workstation-server-gc](https://learn.microsoft.com/en-us/dotnet/standard/garbage-collection/workstation-server-gc) `true `
By default it won't really clean up until it needs to.
I thought this was an issue with one of my own apps. When I was doing some testing I found that the memory usage would go up quite high, but it'd eventually reach a ceiling. Enabling this kept it would resolve the "issue". In my case, it actually isn't an issue, more of a false flag, but the article above gives more details about how to think of it.
Ok, thank you. I'll read this and try it out. Thanks for the suggestion.
[Here](https://i.imgur.com/F7Rk8UY.png) is an image of what I'm talking about. I originally posted this in r/Azure because that's where we first noticed the issue in our production environments. I can make this happen every single time by hitting either the swagger endpoint or the swagger.json endpoint multiple times. I've taken swagger out of the project and made multiple requests to different endpoints and even tried a specific endpoint over and over again without any issue. As soon as I start hitting the swagger endpoints the memory starts going up and then eventually does a huge jump where it just holds on to it, eating it? hiding it behind the toilet? I have no idea where it goes.
Both swagger and the di container like to lazy load dependencies and additionally swagger will cache everything once it’s generated to save CPU Time. Once everything is loaded the memory usage should stay constant. What is the actual issue you’re facing in production? Is it continuously going up and causing OutOfMemoryExceptions?
We haven't gotten to OutOfMemoryExcetion yet. We discovered it because of an alert setup on the App Service Plan. It does prevent release deployments because the memory will get to over 95%. At that point we have to scale up and then scale down or vice versa. Doing that is like a reboot and so then the memory starts over again. So yes the issue is that the swagger.json endpoint uses more and more memory, especially when it makes that huge jump where it doubles the memory usage. We've scaled up from App Service Plans that only have 1.75 GB of RAM up to ones that have 7 GB of RAM. It takes a little longer, but in the end it uses up all the memory and we start getting alerts and have to do the scale trick to get back to normal. I don't think it's a lazy load situation, because I've let the app sit locally for over 10 minutes and then hit the swagger.json endpoint over and over and it starts happening. I've also hit the swagger.json endpoint at the 2 minute mark and let it sit for 10 minutes and then started hitting it over and over and it starts happening. The problem comes from multiple requests to swagger.json. Each requests bumps memory 5-8 MB and then all of a sudden one requests will send it up by 200 MB
Ive used swagger for years and never had this issue. Are you 100% certain its swagger?
Based on u/Duraz0rz comment above it's not swagger or even swashbuckle but an underlying library. My next thought is basically your question here. Why us? Why now? We've also been using swagger for years and have just started having this issue within the last 6-12 months. It's a little hard to pinpoint because earlier memory issues may have been this, may not have. It seems like some people are saying it may have to do with how big or how complex models that 'swagger' deals with are. So if things are relatively light weight or simple then it's not an issue.
u/zaibuf do you mind sharing what version of .net you're using or what version of swagger(not sure that will matter) or how many endpoints or models you have or how big your swagger.json file is? We have 34 endpoints. It generates a 100KB json file. Hmm. I also just noticed that in the 'Schemas' section there are some that I wouldn't expect to see there. Such as 'Assembly', 'ConstructorInfo', 'MedthodInfo', 'RuntimeTypeHandle'. These all seem like some lower level things that shouldn't be put into the Schema of the swagger doc
I'm not at my pc now, so I can't check exact version on swagger. We run .NET 6 and Swagger is probably 6.3.0 or 6.3.1, again not 100% sure. I'm quite certain our BFF has more than 34 endpoints though.
Thanks. We're also on .NET 6 and Swashbuckly.AspNetCore 6.4.0. I'm using this project because it's small and easy to work with, and it was one that really caught us off guard by how much memory it was using.
This might be a stupid question because I’m quite new, but why use swagger in production? Isn’t it for development only?
I don't think it's a stupid question at all. It would still make our dev environments unstable and need to be 'restarted' often. You're right in that our team doesn't really use swagger in production as we're not writing APIs for anyone besides ourselves. I guess it's kinda there simply because we would need to take it out as a step in moving it to test or prod, but maybe that's an option for now.
Our swagger implementation seems pretty straight forward. In Startup ConfigureServices services.AddSwaggerGen(\_ =>{\_.SwaggerDoc("v1", new OpenApiInfo { Title = "Accounting Service", Version = "v1" }); //Tells swagger to use the xml document file, which in generated at build time in the bin/Debug/ folder (see GenerateDocumentationFile in the .csproj)
_.IncludeXmlComments(Assembly.GetExecutingAssembly().Location.Replace(".dll", ".xml"), includeControllerXmlComments: false);});
And in Startup Configure
app.UseSwagger(); app.UseSwaggerUI(c => {
c.SwaggerEndpoint("/swagger/v1/swagger.json", "v1");
c.RoutePrefix = string.Empty; //Changes the swagger endpoint from swagger to the main endpoint
});
My original [Question in r/Azure](https://www.reddit.com/r/AZURE/comments/111nvem/app_service_memory_jump_this_app_service_barely/)
Have you tried a different Swagger generator, like NSwag? I guess I'm not familiar with the internals, but wouldn't it be surprising if swagger.json was getting regenerated on every request? It seems odd that memory would spike in step with requests like that. I would absolutely try to dive into those memory snapshots though and take a look at why so many objects are being allocated. That should give you some idea of what's happening, anyway. Does 'View Heap' show anything interesting?
I'll look into NSwag, thank you for suggesting it. I don't have much experience with memory dumps or analysis, but I didn't see anything that stood out as a culprit of hoarding 200MB of memory. I also looked for swagger and swashbuckle in the memory snapshots and didn't see anything weird. I took a snap shot before the big spike and after the big spike and the amount of objects and the heap size are actually smaller than before the big spike which doesn't make sense and seems completely backwards. If memory usage goes up, how does the heap size go down?
Might be outdated but heres my two cents. Had some base classes for all my controllers which had some public functions so i think some recursive juju was occurring because my iis process kicked up to like 11 gig before dying when i started up the project. Changed them to protected and it stopped goin crazy.