Problem
Wir hatten bisher schon ein paar JBOSS-Abstürze wegen outofmem
13:28:28,208 INFO [appHibernateInterceptor] DAO;354368;net.corpintra.app.job.domain.JobDao;findForProcessing;2
13:28:28,210 DEBUG [JobServiceImpl$EnhancerByCGLIB$9688bd47] *** end findSummaryJobs (JobSearchRequestTO)
returning JobSearchResponseTO: [JobSearchResponseTO|id=null|secId=e2jXRDr…]
13:28:31,228 FATAL [DamServiceFacadeImpl$EnhancerByCGLIB$d53b91d9] [2196010f-f242-44c8-842e-4e603ebec486] Technical exception occured!
13:28:31,228 FATAL [DamServiceFacadeImpl$EnhancerByCGLIB$d53b91d9] [2196010f-f242-44c8-842e-4e603ebec486] Root-Cause:
java.lang.OutOfMemoryError: PermGen space
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2395)
at java.lang.Class.getMethod0(Class.java:2642)
Erläuterung
zum java.lang.OutOfMemoryError: PermGen space
One more interesting flavor of the same error message, less common but hence even more troublesome is: “java.lang.OutOfMemoryError: PermGen space”. Most of the memory profiler tools are unable to detect this problem, so it is even more troublesome and therefor – interesting.
To understand this error message and fix it, we have to remember that, for optimized, more efficient garbage-collecting Java Heap is managed in generations – memory segments holding objects of different ages.
Garbage collection algorithms in each generation are different. Objects are allocated in a generation for younger objects – the Young Generation, and because of infant mortality most objects die there. When the young generation fills up it causes a Minor Collection. Assuming high infant mortality, minor collections are garbage- collected frequently.
Some surviving objects are moved to a Tenured Generation. When the Tenured Generation needs to be collected there is a Major Collection that is often much slower because it involves all live objects. Each generation contains variables of different length of life and different GC policies are applied to them.
There is a third generation too – Permanent Generation. The permanent generation is special because it holds meta-data describing user classes (classes that are ot part of the Java language). Examples of such meta-data are objects describing classes and methods and they are stored in the Permanent Generation.
Applications with large code-base can quickly fill up this segment of the heap which will cause java.lang.OutOfMemoryError: PermGen no matter how high your -Xmx and how much memory you have on the machine.
Sun JVMs allow you to resize the different generations of the heap, including the permanent generation. On a Sun JVM (1.3.1 and above) you can configure the initial permanent generation size and the maximum permanent generation size.
To set a new initial size on Sun JVM use the -XX:PermSize=64m option when starting the virtual machine. To set the maximum permanent generation size use -XX:MaxPermSize=128m option. If you set the initial size and maximum size to equal values you may be able to avoid some full garbage collections that may occur if/when the permanent generation needs to be resized. The default values differ from among different versions but for Sun JVMs upper limit is typically 64MB.
Massnahme:
der PermGenSpace ein JVM beinhaltet den Platz für Classes der Anwendung app10 und ist offenbar zu klein.
Durch den Start-Parameter -XX:MaxPermSize=128m wird dieser Platz verdoppelt (default=64m)
Links
Monitoring
Understanding
“java.lang.OutOfMemoryError: PermGen space ”: have you seen this error before ? If you have never seen it, you haven’t probably been involved in creating large java server applications as of lately. Let’s understand what this error means.
When you get OutOfMemoryError, it means that (so obvious) your application doesn’t have enough memory to go on. But the rest of the message, “PermGen space” means that it’s not the common object heap space that it’s lacking, but only the memory space for the binary code of classes and methods is lacking.
Let’s go deeper into this subject to understand properly what PermGen means. Well, PermGen is an acronym for “Permanent Generation”. The term Generation comes from the concept of “generational collection” and the term “generational collection” refers to an heuristic in the garbage collection mechanism that is used to intelligently identify only the most probable objects for garbage collection. There are 3 kinds of Generations in the VM memory: the young generation, the tenured generation and the permanent generation.
Let’s go back to the original “Permanent Generation” term. As I said, it identifies a particular set of objects which are candidate for garbage collection and handled differently by the GC. These kind of objects which belong to the “Permanent Generation” are only the binary codes of all classes and methods. So, the binary code of classes and methods are not stored in the same memory area as the object instances. Now, I think you already understand what this error mean. “java.lang.OutOfMemoryError: PermGen space ” means that the virtual machine ran out of space for loading the binary code of classes and methods.
There are two steps ahead: knowing understanding why this error has happened to your application, learning to monitor the permanent generation size in order to prevent the occurrence of this error; and learning how to increase the permanent generation size.
Why has this error happened ?
The default size for permanent generatoin is 64M. But how come my not so large web application, which totals 2 megabytes of class code, could exceed this limit ? You’re probably not counting it correctly. Your web application probably runs inside Jboss, Tomcat or another J2EE server. Add the size of Java SE libraries, the size of Jboss, the size of all libraries used by Jboss internally, the size of all libraries in the WEB-INF/lib directory of you WAR and you will get a large, large binary code area.
Aren’t Java classes loaded on demand ? Yes, they are loaded on demand. So, you will probably have to factor that large number by 3 or 4. But there are other factors for the enlargement of the permanent generation area. If you web application has JSP pages, remember that every JSP page is converted into a servlet class file, before being executed. So, you should add that to the previous number. Also, maybe there are other web applications running under Jboss than you own. If you use Hibernate, for each persistent class that you create, Hibernate will dynamically create an enhanced class. If you use RMI or EJB, many kinds of proxy classes will be created and loaded in memory.
Now that you believe that the size of the binary code of classes and methods can exceed 64M, you have probably given up summing the sizes of all classes created and loaded by the VM, Jboss and you application. You’d better off monitoring the use of PermGen area than trying to calculate its size. That leads us to the next step: how can I monitor the increase in PermGen area.
How can I monitor the increase of PermGen area ?
jstat
For monitoring PermGen, we’ll learn to use a very simple tool, which is part of Java 5, called JSTAT. For those who use Linux or any other Unix-like OS, jstat is very similar to the TOP tool. JSTAT will connect to some VM, local or remote, and monitor some critical resources. Let’s see how it works. Just type JSTAT and you’ll see its command-line options:
Usage: jstat -help|-options
jstat -<option> [-t] [-h<lines>] <vmid> [<interval> [<count>]]
Definitions:
<option> An option reported by the -options option
<vmid> Virtual Machine Identifier. A vmid takes the following form:
<lvmid>[@<hostname>[:<port>]]
Where <lvmid> is the local vm identifier for the target
Java virtual machine, typically a process id; <hostname> is
the name of the host running the target Java virtual machine;
and <port> is the port number for the rmiregistry on the
target host. See the jvmstat documentation for a more complete
description of the Virtual Machine Identifier.
<lines> Number of samples between header lines.
<interval> Sampling interval. The following forms are allowed:
<n>[“ms”|”s”]
Where <n> is an integer and the suffix specifies the units as
milliseconds(“ms”) or seconds(“s”). The default units are “ms”.
<count> Number of samples to take before terminating.
-J<flag> Pass <flag> directly to the runtime system.
Let’s understand the options. The 1st thing you need to know is the VMID (Virtual Machine ID). What is the VMID ? Right now, there are 2 Java applications running on my computer, each in its own virtual machine. Each virtual machine has its own VMID.
jps
To discover the VMID that the application you need to monitor is using, you’ll have to learn another tool: JPS (Java Process Status). There is a very similar tool in the Unix world called PS. JPS will list all running VMs with their VMIDs. Just type JPS on the command-line and you’ll see this:
app-psrv1:/usr/java/jdk1.5.0_21/bin # ./jps -l
4020 /app/tools/activemq/bin/run.jar
29521 sun.tools.jps.Jps
21163 org.jboss.Main
Let’s understand JPS’s output. In its basic form, jps outputs the VMID and the simple name of the executable class that is running in the virtual machine. JPS itself runs inside a virtual machine and therefore, is always reported by jps. If, by the simple name of main class you cannot discover the VMID of your application, I suggest you type “jps -l” which will show the complete name of the main class for each VM. In most cases, that will do.
Now that we know that the VMID of Jboss is 21163, let’s use JSTAT to monitor Jboss usage of PermGen memory. To use JSTAT, you need specify a single parameter that will identify the class of monitors you want to use. In our case, we will use “-gcpermcapacity” which will show statistics of the sizes of the Permanent Generation.
app-psrv1:/usr/java/jdk1.5.0_21/bin # ./jstat -gcpermcapacity 21163
PGCMN PGCMX PGC PC YGC FGC FGCT GCT
16384.0 131072.0 67840.0 67840.0 297 19 14.141 20.087
- Explanation of Parameter from man jstat
- -gcpermcapacity Option
Permanent Generation Statistics
+——-+———————————————+
|Column | Description |
+——-+———————————————+
|PGCMN | Minimum permanent generation capacity (KB). |
|PGCMX | Maximum permanent generation capacity (KB). |
|PGC | Current permanent generation capacity (KB). |
|PC | Current permanent space capacity (KB). |
|YGC | Number of young generation GC events. |
|FGC | Number of Full GC events. |
|FGCT | Full garbage collection time. |
|GCT | Total garbage collection time. |
+——-+———————————————+
The important metrics
There are 3 important numbers to monitor regarding permanent generation size: the maximum permanent size, the current permanent size, and the current permanent utilization. Now, look at jstat’s ouput.
- The 1st column (PGCMN) shows the minumum size for the permsize. That’s a parameter you can set in the initialization of the virtual machine.
- The 2nd column (PGCMX) shows the maximum size for permsize. This is also a paramter you can set in the initialization of the virtual machine.
- Now, compare the 2nd column to the 3rd column. The permanent size fluctuate between the minimum and the maximum number.
- But that doesn’t mean it’s all used. To know how much of the current permanent size is being used, we’ll have to look at another number, the Permanent Capacity Utilization, which you can see using the “-gcutil” parameter.
- app-psrv1:/usr/java/jdk1.5.0_21/bin # ./jstat -gcutil 21163
S0 S1 E O P YGC YGCT FGC FGCT GCT
5.68 0.00 21.17 31.51 91.47 304 6.081 19 14.141 20.222 - The Parameter P shows the Utilization of the current permanent size.
- Explanation of Parameter from man jstat
- -gcutil Option
Summary of Garbage Collection Statistics
+——-+——————————————-+
|Column | Description |
+——-+——————————————-+
|S0 | Survivor space 0 utilization as a per- |
| | centage of the space’s current capacity. |
|S1 | Survivor space 1 utilization as a per- |
| | centage of the space’s current capacity. |
|E | Eden space utilization as a percentage of |
| | the space’s current capacity. |
|O | Old space utilization as a percentage of |
| | the space’s current capacity. |
|P | Permanent space utilization as a percent- |
| | age of the space’s current capacity. |
|YGC | Number of young generation GC events. |
|YGCT | Young generation garbage collection time. |
|FGC | Number of Full GC events. |
|FGCT | Full garbage collection time. |
|GCT | Total garbage collection time. |
+——-+——————————————-+
- Ifminimum =maximum permanent size, thencurrent =maximum =minimum permanent size. Therefore, this number will not be very useful.
- But, ifminimum <maximum, knowing ifcurrent is getting closer tomaximum is of vital importance. Ifcurrent is only a few megabytes bellow maximum, maybe this is the right time to change Jboss Installation, so that its virtual machine is initialized with a greater MAXIMUM permanent size.
Fix the JVM-Problem
There are two command-line parameters that you must use to define the size of the permgen area: -XX:PermSize -XX:MaxPermSize. See the example bellow:
java -Xms64m -Xmx128m -XX:PermSize=64m -XX:MaxPermSize=256m MainClass
In this example, we are instructing the Java Virtual Machine to create an initial object heap size of 64 megabytes (-Xms) and maximum size of 128 megabytes (-Xmx). And we are defining a PermGen heap area of initially 64 megabytes (-XX:PermSize) and maximum size of 256 megabytes (-XX:MaxPermSize). Remember that the initial size of the heap area in this example, is the sum of the initial object heap area and the initial permgen heap area, that is, 64+64=128 megabytes. And the same for the maximum size of the heap area: 128+256=384 megabytes.
As a practical example, let’s see how to increase permgen area size in Jboss. Locate the jboss.bat (in Windows) or jboss.conf (in Unix). Look for a line containing a redefinition of the shell script variable JAVA_OPTS.
JAVA_OPTS=”-Xms512m -Xmx1024m -XX:PermSize=64m -XX:MaxPermSize=128m”
Now that we have learned how to monitor and fix permgen errors, I advice you not to blindly increase the permgen size everytime there is a permgen error. Monitor how the size of the permgen area is growing. The Permgen area shouldn’t be constantly growing over a long period. The Java language features dynamic class loading, so classes will be loaded on demand and the permgen heap size will grow as well. But, if in the long term, your permgen area is still steadly growing, you are probably using some kind of bytecode generation framework which is misbehaving or you are using it incorrectly. Analyse that and rethink what you are doing.