java.lang.OutOfMemoryError occurs when you call dispose method of frame. It is reproducible since JDK8 b04 with VMWare virtual machine and it is not reproducible if VirtualBox is used. Not reproducible on physical systems.
JDK-7157018Three api/java_awt/Window/indexTGF_* tests fail on OEL 6.2 x86
JDK-7186840Labels are not showed in elements of AWT Panel on VMWare linux virtual machine