Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-8005607

Recursion in J2DXErrHandler() Causes a Stack Overflow on Linux

    Details

    • Subcomponent:
    • Resolved In Build:
      b31
    • CPU:
      x86
    • OS:
      linux, solaris

      Backports

        Description

        FULL PRODUCT VERSION :
        Any update of 32-bit JDK 6, JDK 7, JDK 8

        ADDITIONAL OS VERSION INFORMATION :
        32-bit Linux OS, 32-bit Oracle Solaris OS (for example Oracle Solaris 10 8/11 s10x_u10wos_17b X86)

        A DESCRIPTION OF THE PROBLEM :
        Infinite recursion in AWT native function J2DXErrHandler() causes a stack overflow and a crash under certain circumstances on 32-bit Linux OS or 32-bit Oracle Solaris OS. An approximation of J2DXErrHandler() function in pseudo-code is provided below.

        J2DXErrHandler(parms) {
            if (errorBelongsToJvm(parms))
                letJvmHandleIt(parms);
            else if (pOldHandler)
                pOldHandler(parms);
        }
         
        The problem occurs, if another thread (for example, GTK thread) is doing the same sort of thing concurrently. This can lead to the following situation.

        JVM thread: Sets J2DXErrHandler(), saves ANY_PREVIOUS_HANDLER as previous
            GTK thread: Sets some GTK_HANDLER, saves J2DXErrHandler() as previous
        JVM thread: Restores ANY_PREVIOUS_HANDLER
            GTK thread: Restores J2DXErrHandler()
        JVM thread: Sets J2DXErrHandler(), saves J2DXErrHandler() as previous

        Then, if J2DXErrHandler() is called for an error that JVM doesn't own, this error handler calls a previous handler, which was saved, and passes the error for processing. Unfortunately the previous handler is also set to J2DXErrHandler(), thus the result is an infinite recursion, a stack overflow, and a crash.

        STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
        1. Make sure that OS used for this purpose is 32-bit Linux OS or 32-bit Oracle Solaris OS.
        2. Download the attached "J2DXErrHandlerTestcase.zip" or "J2DXErrHandlerTestcase-Solaris.zip" archive depending on the OS type and unpack it.
        3. Execute: java -cp xtest.jar -Djava.library.path=. xtest
            or unpack "src.zip", compile the test case using "makefile" script and execute the previously mentioned command line.
        4. When Java2Demo window appears select the "Images" tab.
        5. Select the menu item "Options->Background Color".
        6. After the window appears, hit "Escape" to close it.
        7. Repeat steps 5 and 6 in quick succession until the process crashes (may take 10-20 attempts).

        Notes:
        - The problem can be reproduced using tabs other than "Images".
        - The problem may also be easier to reproduce on multi-core hosts.

        REPRODUCIBILITY :
        The problem is intermittent and may be reproduced using the test case after different number of attempts.

        ERROR MESSAGES/STACK TRACES THAT OCCUR :
        $ /shared/jdks/oracle/6/jdk6_37/bin/java -cp xtest.jar -Djava.library.path=. xtest
        Loading library
        Starting xtest
        Starting xtest: xesetTime=10 xeclearTime=10
        Opening X display
        Starting thread...
        Waiting for thread to run...
        Thread started
        Thread running
        Launch java2 demo
        ODD 0x9fd9f950 != 0xb6b4b902
        ODD 0x9f45b280 != 0xb6b4b902
        ODD 0x9f465c40 != 0xb6b4b902
        Launch complete
        ODD 0x9f465c40 != 0xb6b4b902
        URK! newXeh called
        URK! newXeh called
        URK! newXeh called
        URK! newXeh called
        ODD 0x9f465c40 != 0xb6b4b902
        ODD 0x9f465c40 != 0xb6b4b902
        Segmentation fault

          Attachments

            Issue Links

              Activity

                People

                • Assignee:
                  alitvinov Anton Litvinov
                  Reporter:
                  alitvinov Anton Litvinov
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  7 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: