Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-6497221

Multithreaded stresstest for Derby (Java DB) hangs



    • Type: Bug
    • Status: Closed
    • Priority: P4
    • Resolution: Duplicate
    • Affects Version/s: 6
    • Fix Version/s: None
    • Component/s: hotspot
    • Labels:
    • Subcomponent:
    • CPU:
    • OS:


      The problem I see is a hang in one of the tests that my group (DBTG) runs as part of nightly testing of the Derby (Java DB) database. The test is a multithreaded (10 load threads) stress test that is supposed to run for 10 minutes, and is run twice every night on two branches. The hang occurs now and then, 7-8 times the past month, and we have seen it on both branches of the product. It has been observed when running on the latest build of Mustang (b104) on Solaris 10 on x86 (v20s).

      We test with jdk1.4 and jdk1.5, on Solaris 8 and 9, on x86 and Sparc, but have not observed this hang in any other configuration.

      Failed attempts at debugging with the jdk tools made me suspect there might be a bug in the jvm, so instead I have generated a core file using gcore, it can be found here: /net/khepri32/export/home/tmp/cores/siv08.670
      The core file is from a run on siv08.norway.sun.com:
      vo136787@siv08:~> uname -a
      SunOS siv08 5.10 Generic_118855-14 i86pc i386 i86pc

      The behaviour of the hang is that the jvm-process will be using 98-99% cpu, it can not be connected to with jconsole or jdb. Using jinfo to print for instance system properties somehow resolves the hang and the process continues to run. Connecting with jstack fails, but when using the force option I get a list of threads but no stack (this also resolves the hang):

      Deadlock Detection:

      No deadlocks found.

      Thread t@25: (state = IN_VM)

      Thread t@24: (state = IN_JAVA)

      Thread t@23: (state = IN_VM)

      Thread t@22: (state = IN_VM)

      Thread t@21: (state = IN_VM)

      Thread t@20: (state = IN_VM)

      Thread t@19: (state = IN_VM)

      Thread t@18: (state = IN_JAVA)

      Thread t@17: (state = IN_VM)

      Thread t@16: (state = IN_VM)

      Thread t@15: (state = IN_VM)

      Thread t@14: (state = BLOCKED)

      Thread t@13: (state = BLOCKED)

      Thread t@8: (state = BLOCKED)

      Thread t@7: (state = BLOCKED)

      Thread t@6: (state = BLOCKED)

      Thread t@2: (state = BLOCKED)

      Below follows some output from dbx:

      vo136787@siv08:~> /opt/studio11/SUNWspro/bin/dbx /usr/local/java/jdk1.6.0_b104/bin/java /tmp/siv08.670
      t@null (l@25) terminated by signal 0 (UNKNOWN SIGNAL)
      0xfef309b7: ___lwp_mutex_timedlock+0x0007: jae ___lwp_mutex_timedlock+0x15 [ 0xfef309c5, .+0xe ]
      (dbx) where
      =>[1] ___lwp_mutex_timedlock(0x80727c0), at 0xfef309b7
        [2] Mutex::lock_without_safepoint_check(0x8072768), at 0xfe893a13
        [3] SafepointSynchronize::block(0x8748400), at 0xfe8ddbce
        [4] ObjectMonitor::wait(0x816a0d8, 0xbb8, 0x0, 0x1, 0x8748400), at 0xfec94008
        [5] ObjectSynchronizer::wait(0x8748db4, 0xbb8, 0x0, 0x8748400, 0x1a1), at 0xfe94184b
        [6] JVM_MonitorWait(0x87484e8, 0xf264261c, 0xbb8, 0x0, 0xf73d8370, 0xf26425e4, 0x0, 0xf264261c, 0xf3273ac0, 0x0, 0xf32058b8, 0x0, 0xf2642620, 0xfad8c0e8, 0xfb965aec, 0x0, 0xfb4081f9, 0xbb8, 0x0, 0xfad8c0e8), at 0xfe94157c
        [7] 0xfb40a578(0x0, 0xfb4081f9, 0xbb8, 0x0, 0xfad8c0e8, 0xbb8), at 0xfb40a578
        [8] 0xfb965aec(0xfab85830, 0xf99c18e0, 0xf73d83d0, 0x0, 0x0, 0x0), at 0xfb965aec
      (dbx) threads
            t@1 a l@1 ?() running in __lwp_wait()
            t@2 a l@2 JavaMain() running in ___lwp_mutex_timedlock()
            t@3 b l@3 java_start() running in ___lwp_cond_wait()
            t@4 b l@4 java_start() running in ___lwp_cond_wait()
            t@5 b l@5 java_start() running in ___lwp_cond_wait()
            t@6 b l@6 java_start() running in ___lwp_cond_wait()
            t@7 b l@7 java_start() running in ___lwp_cond_wait()
            t@8 b l@8 java_start() sleep on 0xfee1ffd0 in __lwp_park()
            t@9 b l@9 java_start() running in ___lwp_cond_wait()
           t@10 b l@10 java_start() running in ___lwp_cond_wait()
           t@11 b l@11 java_start() running in ___lwp_cond_wait()
           t@12 b l@12 java_start() running in ___lwp_mutex_timedlock()
           t@13 b l@13 java_start() running in ___lwp_cond_wait()
           t@14 b l@14 java_start() running in ___lwp_cond_wait()
           t@15 b l@15 java_start() running in ___lwp_mutex_timedlock()
           t@16 b l@16 java_start() running in inflate()
           t@17 b l@17 java_start() running in ___lwp_mutex_timedlock()
           t@18 b l@18 java_start() running in Stall()
           t@19 b l@19 java_start() running in ___lwp_cond_wait()
           t@20 b l@20 java_start() running in Stall()
           t@21 b l@21 java_start() running in inflate()
           t@22 b l@22 java_start() running in inflate()
           t@23 b l@23 java_start() running in ___lwp_mutex_timedlock()
           t@24 b l@24 java_start() running in ___lwp_mutex_timedlock()
           t@25 b l@25 java_start() running in ___lwp_mutex_timedlock()


          Issue Links



              collins Gary Collins (Inactive)
              vostgaar Vemund Østgaard
              0 Vote for this issue
              2 Start watching this issue