Details

    • Subcomponent:
    • Resolved In Build:
      b07
    • CPU:
      sparc
    • OS:
      solaris_10
    • Verification:
      Verified

      Backports

        Description

        This is logged by Oracle Weblogic team
        This is tracked in BugDB
        BUG 9502049 - STRESS BPM - MEMORY LEAK DETECTED.
        BUG 9527439 - STRESS MEMORY LEAK IN SERVERNOTIFFORWARDER

        repro.jar repro test case attached

        The WL developer who reported this issue had been dialoging with
        ###@###.###
        and the analysis from the above BUGDB bugs follows
        *****************************************************
        Email to Eamonn
        ---------------

        Eamonn,

        I believe we are running into a memory leak in a long running JMX connection.
        I am wondering if it is a known issue. The details are as follows

        - WLS has an admin server and managed servers. We have implemented a
        federated model where the admin server delegates requests to the managed
        servers. The admin server has a long running JMX connection to each managed
        server.

        - a user adds a notification listener on the adminserver side of the
        connection (client side)

        - ArrayNotificationBuffer registers a listener for every object in the
        MBeanServer on the managed server

               for (ObjectName name : names)
                   addBufferListener(name);

        - ArrayNotificationBuffer registers a listener to be notified about new
        MBeans added to the managed server

                  addNotificationListener(MBeanServerDelegate.DELEGATE_NAME,
                                           creationListener, creationFilter, null);


        - ServerNotifForwarder keeps a listenerMap of notification listeners (server
        side)

                  listenerMap.put(nn, set);

        - it does not appear that these listeners entries are removed unless the
        connection is closed. The ArrayNotificationBuffer does not remove them, it
        does not appear that they are removed if a MBean is unregistered, and the
        code prevents them from being removed if the MBean is not registered

              if (name != null && !name.isPattern()) {
                   if (!mbeanServer.isRegistered(name)) {
                       throw new InstanceNotFoundException("The MBean " + name +
                           " is not registered.");
                   }
               }

        So it appears if you have a number of short lived MBeans, then the
        listenerMap will continue to grow until the connection is closed.

        My questions are:

        - do you agree this is an issue with the JMX implementation? Is this a known
        issue?

        - is there anything we can do to workaround the issue? [Somehow cycling the
        connections is a fairly big change to make and we are at the very end of our
        release cycle.]

        - is there a mechanism for patching the JMX implementation?

        If you are not the right person, please let me know who to contact.

        Thanks,

        Peter


        *** PBOWER 03/26/10 08:47 am ***
        Response from Éamonn

        Hi Peter,

        Yes, I am still the right person to contact with technical questions about
        JMX. The former JMX team is no longer working on it full time but we are
        still available to help with urgent problems.

        I believe the behaviour you describe is deliberate. As I recall, our
        reasoning was that clients do not need to have an actual network connection
        open to be valid. For example, the connector could be using an datagram
        protocol or the like. Such a client should nevertheless receive
        notifications, and since it cannot distinguish between, on the one hand, an
        MBean that was unregistered and then re-registered between two of its
        transient connections, and on the other, an MBean that remained there the
        whole time, it would not know in the former case that it needed to re-add its
        listeners to the new MBean. The downside is the behaviour you observe when
        many MBeans come and go, and the client (typically a federating client) adds
        a listener automatically to each of them.

        The code you cite, that tests whether the MBean is registered before removing
        the listenerMap entry, does not seem correct to me. I think it should throw
        InstanceNotFoundException only if there is no entry in the listenerMap. It
        would choose between InstanceNotFoundException and ListenerNotFoundException
        based on whether the MBean exists. That would mean that you could remove a
        listener "ghost" from an MBean that has already been unregistered. It is a
        slight deformation of JMX semantics but does not seem unreasonable.

        Unfortunately, since the JMX API is part of the JDK, the only way to patch it
        is as part of a JDK update release. We could set the wheels in motion for
        this if you think it would help, but it would be months before you would see
        the patch, and, even then, only customers who had updated to the very latest
        JDK 6 would be able to take advantage of it.

        If the problem is very critical, then the best workaround I can think of
        would be to abuse reflection to stick your hand into the motor and tweak
        things while it's running. :-/ I'm thinking you could have a thread that
        accesses ServerNotifForwarder and thence listenerMap directly (via
        Field.setAccessible), and that periodically purges listenerMap of entries for
        MBeans that no longer exist. It's very nasty, and it requires your code to be
        privileged, but I think it would work. (I would suggest failing safe, though,
        so if you don't find ServerNotifForwarder or listenerMap on whatever JRE you
        are running on, you just ignore the problem.) Alternatively, as you suggest,
        you could cycle your connections periodically.

        Both delegation and notifications are much improved in JMX 2.0, and you would
        have a much better solution there, but unfortunately that has been deferred
        from JDK 7 so you won't be able to use it any time soon.
        Regards,

        @ Éamonn McManus · JMX Spec Lead · http://weblogs.java.net/blog/emcmanus
        ***************************************************

          Attachments

            Issue Links

              Activity

                People

                • Assignee:
                  coffeys Sean Coffey
                  Reporter:
                  dvanero Dennis Van Eron
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  0 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:
                    Imported:
                    Indexed: