Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-6822370

ReentrantReadWriteLock: threads hung when there are no threads holding onto the lock (Netra x4450)

    Details

    • Type: Bug
    • Status: Closed
    • Priority: P2
    • Resolution: Fixed
    • Affects Version/s: hs14, 6u12, 6u14
    • Fix Version/s: hs17
    • Component/s: hotspot
    • Subcomponent:
    • Resolved In Build:
      b06
    • CPU:
      x86
    • OS:
      solaris_10
    • Verification:
      Verified

      Backports

        Description

        Only happen on Hardware: x4450
        OS = Solaris 10 (does not happen with Linux)
        JDK = JDK 6.0 upd 10 and 12

        Problem Description

        Java threads are blocked waiting for a lock that is not held by
        any threads ie. a lock that has already been released. thread is
        seen waiting on this synchronizer -
        - parking to wait for <0xfffffd72f0e32118> (a java.util.concurrent.lock
        s.ReentrantReadWriteLock$NonfairSync)

        Troubleshooting results are available to show that
        the lock is not being held by any thread. See details in comments.

        This hung issue is only happening on Netra x4450 but not
        any other platforms such as x2200s, x4440s, x4600s, and Niagara-based systems like a 5440.

        A testcase and more details are available in comments.
        A similar problem was reported via the concurrency-interest mailing list and then on the core-libs dev list.

        http://mail.openjdk.java.net/pipermail/core-libs-dev/2009-July/002040.html

        The problem is not restricted to x4450 but shows up on 4-8way Intel systems. Same basic scenario - the application hangs with a bunch of threads all inside a LinkedBlockingQueue trying to acquire the internal ReentrantLock that nobody seems to own. Again -XX:+UseMembar avoids the problem.

        I used the attached test program to reproduce the problem on an 8-way Intel machine that I have access to. However when I tried to probe deeper by using modified classes that allowed me to examine the internal lock state when the hang occurs, it ceased to occur.

        Two different hangs are possible with the test program:
        a) use of the application LinkedBlockingDeque
        b) use of the ThreadPoolExecutors LinkedBlockingQueue

        Thanks to Ariel Weisburg and Ryan Betts for reporting the problem and working to get a small reproducible test case.

          Issue Links

            Activity

            Hide
            dholmes David Holmes added a comment -
            BT2:SUGGESTED FIX

            *** 4654,4663 ****
            --- 4654,4664 ----
              void Parker::park(bool isAbsolute, jlong time) {
                // Optional fast-path check:
                // Return immediately if a permit is available.
                if (_counter > 0) {
                    _counter = 0 ;
            + OrderAccess::fence();
                    return ;
                }
              
                Thread* thread = Thread::current();
                assert(thread->is_Java_thread(), "Must be JavaThread");
            *** 4696,4705 ****
            --- 4697,4707 ----
                int status ;
                if (_counter > 0) { // no wait needed
                  _counter = 0;
                  status = pthread_mutex_unlock(_mutex);
                  assert (status == 0, "invariant") ;
            + OrderAccess::fence();
                  return;
                }
              
              #ifdef ASSERT
                // Don't catch signals while blocked; let the running threads have the signals.
            *** 4735,4745 ****
                assert_status(status == 0, status, "invariant") ;
                // If externally suspended while waiting, re-suspend
                if (jt->handle_special_suspend_equivalent_condition()) {
                  jt->java_suspend_self();
                }
            !
              }
              
              void Parker::unpark() {
                int s, status ;
                status = pthread_mutex_lock(_mutex);
            --- 4737,4747 ----
                assert_status(status == 0, status, "invariant") ;
                // If externally suspended while waiting, re-suspend
                if (jt->handle_special_suspend_equivalent_condition()) {
                  jt->java_suspend_self();
                }
            ! OrderAccess::fence();
              }
              
              void Parker::unpark() {
                int s, status ;
                status = pthread_mutex_lock(_mutex);
            Show
            dholmes David Holmes added a comment - BT2:SUGGESTED FIX *** 4654,4663 **** --- 4654,4664 ----   void Parker::park(bool isAbsolute, jlong time) {     // Optional fast-path check:     // Return immediately if a permit is available.     if (_counter > 0) {         _counter = 0 ; + OrderAccess::fence();         return ;     }        Thread* thread = Thread::current();     assert(thread->is_Java_thread(), "Must be JavaThread"); *** 4696,4705 **** --- 4697,4707 ----     int status ;     if (_counter > 0) { // no wait needed       _counter = 0;       status = pthread_mutex_unlock(_mutex);       assert (status == 0, "invariant") ; + OrderAccess::fence();       return;     }      #ifdef ASSERT     // Don't catch signals while blocked; let the running threads have the signals. *** 4735,4745 ****     assert_status(status == 0, status, "invariant") ;     // If externally suspended while waiting, re-suspend     if (jt->handle_special_suspend_equivalent_condition()) {       jt->java_suspend_self();     } !   }      void Parker::unpark() {     int s, status ;     status = pthread_mutex_lock(_mutex); --- 4737,4747 ----     assert_status(status == 0, status, "invariant") ;     // If externally suspended while waiting, re-suspend     if (jt->handle_special_suspend_equivalent_condition()) {       jt->java_suspend_self();     } ! OrderAccess::fence();   }      void Parker::unpark() {     int s, status ;     status = pthread_mutex_lock(_mutex);
            Hide
            jprtbugupd JPRT Bug Updates (Inactive) added a comment -
            BT2:EVALUATION

            < wrong sub-CR >
            Show
            jprtbugupd JPRT Bug Updates (Inactive) added a comment - BT2:EVALUATION < wrong sub-CR >
            Show
            jprtbugupd JPRT Bug Updates (Inactive) added a comment - BT2:EVALUATION http://hg.openjdk.java.net/jdk7/hotspot-rt/hotspot/rev/95e9083cf4a7
            Hide
            jprtbugupd JPRT Bug Updates (Inactive) added a comment -
            BT2:EVALUATION

            --
            Show
            jprtbugupd JPRT Bug Updates (Inactive) added a comment - BT2:EVALUATION --
            Show
            jprtbugupd JPRT Bug Updates (Inactive) added a comment - BT2:EVALUATION http://hg.openjdk.java.net/jdk7/hotspot/hotspot/rev/95e9083cf4a7

              People

              • Assignee:
                dholmes David Holmes
                Reporter:
                kbteo Kim Teo
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:
                  Imported:
                  Indexed: