Excessive Memory Size using the Microsoft Service Bus
During periods of high activity on the Resgrid system NewRelic would send out a large amount of warnings due to excessive memory usage. This was great information but I didn’t know why we would have excessive memory usage without corresponding CPU to match.
I like to design my API’s as stateless as possible and the Resgrid API’s are no different. They do a fair amount of business logic and work, but all the heavy stuff is all offloaded to backend workers, databases or Azure itself via the Service Bus. So this slow progressive rise in memory was troubling.
With work on moving Resgrid to App Services from Cloud Services I decided to run some JMeter tests against the API to see what was going on. After a bunch of read based operations tests, no memory leak. Then I analyzed what was occurring during the NewRelic alerts, there were a lot of sets.
I altered the JMeter tests and found this:
That’s around 500MB in memory allocated to just the Microsoft.ServiceBus operations, particularly the Microsoft.ServiceBus.Common.IOThreadTimer. Some quick goggling and I found this issue on GitHub, so it seems at least as of end of 2014 this was a known issue.
I checked and the version of Microsoft.ServiceBus I had installed from Nuget was 2.4.1 which was from July 2014. Which was before the GitHub issue was posted (in late 2014). The current version of Microsoft.ServiceBus is 3.0.9 from November 2015. I updated and re-ran my JMeter tests and this was the result:
After the update, no more memory leak, and an inspection of the objects in memory reveals that no large service bus objects in the heap. So if your using the Azure Service Bus and experiencing memory leaks I highly recommend looking at your Microsoft.ServiceBus version.
If you’re a First Responder or know one check out Resgrid which is a SaaS product utilizing Microsoft Azure, providing logistics, management and communication tools to first responder organizations like volunteer fire departments, career fire departments, EMS, search and rescue, CERT, public safety, disaster relief organizations.
