Resolving the Memory Saturation Error

In this blog, we are going to look at the resolution to the memory saturation error reported by Dynatrace on a machine that hosts multiple JVMs. 

The memory saturation error is thrown from Dynatrace when the memory usage goes beyond 80% and the page faults go beyond 20-page faults/second on a Linux machine. 

For this blog, let's assume that we are using a Linux machine with 128GB memory and 6 application JVMs are running with 16GB set as minimum and maximum heap size. The memory saturation error might get thrown if the JVMs started consuming more memory. When the minimum (-Xms) and maximum (-Xmx) heap value is set at 16GB, the system will commit 16GB of RAM space to the heap for each JVM. The JVM will use this 16GB for heap, and in addition to that, it will take some more memory to do its processing. Overall each JVM might take around 20GB of memory to do the processing depending upon requests that are coming to each JVM. At a certain point, it is possible that all 6 JVMs might use around 120GB of memory resulting in memory saturation errors and page faults occurring in that machine. 

There are three ways to resolve this issue. The first and easiest approach is to add more memory to the existing machine. If we double the memory by adding another 128GB to this machine, it will make the machine have 256GB of memory in total. Now, in our situation, if the process consumes 120GB out of 256GB, which is less than 50% of the overall memory usage, and will not touch the 80% memory usage threshold. Also, page faults will not occur as it has space to hold all the pages in the memory. The memory saturation error will go away from Dynatrace. The downside of this approach is it will end up in hardware procurement costs if it is a machine in the data center. If it is a cloud instance, it will result in taking up an instance with more memory which will result in additional cost. 

The second approach is to investigate if the JVM requires that much amount of memory or not and optimize that. This approach needs a detailed analysis to identify the areas which take more memory, optimize and bring the memory usage down. This will help to reduce the memory footprint taken for all 6 JVMs and run the application without any memory saturation error on the same hardware without any upgrade. This approach will take time, depending upon the complexity involved in optimizing the application memory usage. 

The third approach is not to set equal values to the minimum and maximum heap size parameters. Instead, set a minimum value to the minimum heap size argument that the application requires. For example, set the minimum heap size(-Xms) as 4GB and the maximum heap size (-Xmx) as 16GB. This will make the system commit only 4GB for heap size initially per JVM, thereby reducing the overall JVM memory usage to less than 8GB per JVM. All 6 JVMs will take around 58GB, which will bring down the overall memory usage to less than 50% and avoids the memory saturation error. This will work if the application really consumes less heap memory. It might become a problem if the heap size grows to the maximum heap size, which will make the system commit 16GB resulting in a memory saturation error if all the JVMs reach that point. 

This approach can be taken after understanding the heap usage pattern, and it will help if the heap usage of the application is less. Setting different values to minimum and maximum heap size might have little impact on heap shrinkage and expansion. That impact will be less when compared to the page faults occurring in the system. 

Based on the scenario in which the memory saturation error occurs, any one or all of the above-recommended approaches can be applied to resolve the memory saturation error in Dynatrace.