Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-8213430

Long term memory leak when using PKCS11 and JCE exceeds 32 bit process address space

    Details

    • Type: CSR
    • Status: Closed
    • Priority: P3
    • Resolution: Approved
    • Fix Version/s: 13
    • Component/s: security-libs
    • Labels:
      None
    • Subcomponent:
    • Compatibility Kind:
      behavioral
    • Compatibility Risk:
      low
    • Compatibility Risk Description:
      Hide
      This fix has been tested against different versions of NSS libraries and no issues were found. However, to mitigate the low risk of not being able to re-build a previously extracted key, a new system property "sun.security.pkcs11.disableKeyExtraction" is introduced as part of this fix. When needed, users can set this system property to "true" to disable the key extraction. Note that this is applicable to the NSS PKCS11 library only and won't affect other vendors. In addition, if the first key extraction fails by any means, this fix will also disable key extraction. Lastly, key extraction is not performed on token keys. There is no risk of losing stored keys nor key attributes. Only session keys are subject to key extraction.
      Show
      This fix has been tested against different versions of NSS libraries and no issues were found. However, to mitigate the low risk of not being able to re-build a previously extracted key, a new system property "sun.security.pkcs11.disableKeyExtraction" is introduced as part of this fix. When needed, users can set this system property to "true" to disable the key extraction. Note that this is applicable to the NSS PKCS11 library only and won't affect other vendors. In addition, if the first key extraction fails by any means, this fix will also disable key extraction. Lastly, key extraction is not performed on token keys. There is no risk of losing stored keys nor key attributes. Only session keys are subject to key extraction.
    • Interface Kind:
      System or security property
    • Scope:
      JDK

      Description

      Summary

      A memory growth issue is observed when using the SunPKCS11 provider with the NSS PKCS#11 library. This is due to the clean up of native handles which is managed by java GC and which may not kick in before the native heap is exhausted. To address the memory growth issue, this fix extracts native cryptographic keys into Java byte arrays when not in use and re-creates them when needed.

      Problem

      The SunPKCS11 provider is functionally a java wrapper for the native PKCS#11 library. The cryptographic keys (sun.security.pkcs11.P11Key objects) of the SunPKCS11 provider encapsulate native key handles created by the native PKCS#11 library. When the keys are no longer needed, their memory is freed through the phantom reference mechanism which is managed by java GC. However, the native heap may be exhausted when java GC doesn't kick in or is frequent enough.

      This has been a long standing issue, manifested in 32 bit processors.

      Solution

      The proposed solution is to extract native cryptographic keys when they are no longer in use and hold them in a Java byte array in each P11Key object. Key extraction is achieved by retrieving all PKCS#11 attributes (I.e.: CKA_CLASS, CKA_VALUE, CKA_ID, etc.) and saving the result into a Java byte array. With this approach, the native key handle is only kept for a short duration and for most of the time it will not take up space in the native heap.

      The feature has been implemented to minimize a performance penalty through different levels of optimizations. As an example, P11Key keys that have been just created keep their native key handle for its first use. JNI methods were implemented to query all native key attributes at once, and minimize context switching between Java and C code.

      Since this issue was reported when using the NSS PKCS#11 library, this fix applies key extraction only when the NSS PKCS#11 library is used. However, if the same issue is observed with another PKCS#11 library, it is possible to update this fix to cover other PKCS#11 libraries.

      Review discussion: http://mail.openjdk.java.net/pipermail/security-dev/2017-October/016400.html

      Specification

      As part of this change, a property to completely disable this change has been introduced: sun.security.pkcs11.disableKeyExtraction (defaults to false). See the "Compatibility Risk Description" above.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                mbalao Martin Balao
                Reporter:
                bchapman Bruce Chapman
                Reviewed By:
                Valerie Peng
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: