Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-6369541

glibc gethostbyaddr_r()/gethostbyname_r() segfault under heavy concurrent calls

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: P4
    • Resolution: Not an Issue
    • Affects Version/s: 5.0, 5.0u9, 5.0u5, 6
    • Fix Version/s: None
    • Component/s: core-libs
    • Labels:
    • Subcomponent:
    • CPU:
      x86
    • OS:
      linux, linux_2.6, linux_redhat_4.0

      Description

      FULL PRODUCT VERSION :
      1.5.0_01 to 1.5.0_05

      ADDITIONAL OS VERSION INFORMATION :
      Linux 2.6 + glibc 2.3

      A DESCRIPTION OF THE PROBLEM :
      gethostbyaddr_r() and gethostbyname_r() in glibc are unstable under load. Concurrent calls through Inet4AddressImpl methods cause malloc corruption, stack corruption, etc. resulting in a SEGV crash of the JVM.

      See discussion thread here for multiple reports and more details:
      http://forum.java.sun.com/thread.jspa?threadID=643360

      STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
      All reported crashes involve running tomcat under load in production or pre-production environments.

      EXPECTED VERSUS ACTUAL BEHAVIOR :
      EXPECTED -
      No crash.
      ACTUAL -
      Crash.

      ERROR MESSAGES/STACK TRACES THAT OCCUR :
      I have a big collection of these. This is representative:

      #
      # An unexpected error has been detected by HotSpot Virtual Machine:
      #
      # SIGSEGV (0xb) at pc=0xb7f24900, pid=31459, tid=705854384
      #
      # Java VM: Java HotSpot(TM) Client VM (1.5.0_06-b05 mixed mode)
      # Problematic frame:
      # C [libc.so.6+0x79900] memchr+0x70
      #

      --------------- T H R E A D ---------------

      Current thread (0x085c9c50): JavaThread "http-80-Processor196" daemon [_thread_
      in_native, id=32032]

      siginfo:si_signo=11, si_errno=0, si_code=1, si_addr=0xb26b307c

      Registers:
      EAX=0xb26b307c, EBX=0xb7fdce80, ECX=0x00000a0a, EDX=0x0a0a0a0a
      ESP=0x2a1253f4, EBP=0x2a125424, ESI=0x000000e7, EDI=0xb26b307c
      EIP=0xb7f24900, CR2=0xb26b307c, EFLAGS=0x00010206

        Top of Stack: (sp=0x2a1253f4)
      0x2a1253f4: b26b307c 000000f7 b7f0d1fe b26b307c
      0x2a125404: 0000000a 000000f7 00000000 b26b307c
      0x2a125414: 2a1254e8 b7fdce80 28c2c738 fbad2418
      0x2a125424: 2a125448 b7f0d181 28c2c738 2a1254e8
      0x2a125434: 000003e7 0000000a 00000001 00000000
      0x2a125444: b7fdce80 2a125474 b7f1661d 28c2c738
      0x2a125454: 2a1254e8 000003e7 0000000a 00000001
      0x2a125464: 00000000 b7e851dc 2a1254d0 000003e8

      Instructions: (pc=0xb7f24900)
      0xb7f248f0: 00 40 4e e9 8e 00 00 00 90 8d b4 26 00 00 00 00
      0xb7f24900: 8b 08 bf ff fe fe fe 31 d1 01 cf 0f 83 04 01 00

      Stack: [0x2a0a8000,0x2a128000), sp=0x2a1253f4, free space=500k
      Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
      C [libc.so.6+0x79900] memchr+0x70
      C [libc.so.6+0x62181] _IO_getline+0x41
      C [libc.so.6+0x6b61d] fgets_unlocked+0x5d
      C [libnss_files.so.2+0x3e31]
      C [libnss_files.so.2+0x3471] _nss_files_gethostbyname_r+0x211
      C [libc.so.6+0xeb19f] gethostbyname_r+0x10f
      C [libnet.so+0x4d61] Java_java_net_Inet4AddressImpl_getLocalHostName+0x91
      j java.net.Inet4AddressImpl.getLocalHostName()Ljava/lang/String;+0
      j java.net.InetAddress.getLocalHost()Ljava/net/InetAddress;+7
      j com.sun.mail.smtp.SMTPTransport.getLocalHost()Ljava/lang/String;+122
      j com.sun.mail.smtp.SMTPTransport.protocolConnect(Ljava/lang/String;ILjava/lang
      /String;Ljava/lang/String;)Z+254
      j javax.mail.Service.connect(Ljava/lang/String;ILjava/lang/String;Ljava/lang/St
      ring;)V+378


      REPRODUCIBILITY :
      This bug can be reproduced occasionally.

      CUSTOMER SUBMITTED WORKAROUND :
      This is a patch which I have applied which has eliminated this problem.

      A better solution would be to cache the results for concurrent reads by Java threads in InetAddress, and serialize reads to fill the cache through the safer non-reentrant functions in glibc.

      --- ./java/net/Inet4AddressImpl.java.orig 2005-11-11 02:20:32.000000000 +0800
      +++ ./java/net/Inet4AddressImpl.java 2005-12-30 17:20:27.000000000 +0800
      @@ -13,10 +13,10 @@
        * @since 1.4
        */
       class Inet4AddressImpl implements InetAddressImpl {
      - public native String getLocalHostName() throws UnknownHostException;
      - public native byte[][]
      + public synchronized native String getLocalHostName() throws UnknownHostException;
      + public synchronized native byte[][]
               lookupAllHostAddr(String hostname) throws UnknownHostException;
      - public native String getHostByAddr(byte[] addr) throws UnknownHostException;
      + public synchronized native String getHostByAddr(byte[] addr) throws UnknownHostException;
           private native boolean isReachable0(byte[] addr, int timeout, byte[] ifaddr, int ttl) throws IOException;
       
           public synchronized InetAddress anyLocalAddress() {

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              jccollet Jean-Christophe Collet (Inactive)
              Reporter:
              ndcosta Nelson Dcosta (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Imported:
                Indexed: