Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-8271510

Failure in P11Signature sign with 2048-bit DSA keys and NSS Software Token (JDK-11)



    • Type: Bug
    • Status: In Progress
    • Priority: P4
    • Resolution: Unresolved
    • Affects Version/s: 11
    • Fix Version/s: None
    • Component/s: security-libs
    • Labels:


      The C_SignFinal API, according to the PKCS#11 standard, is reasonably expected to return a CKR_BUFFER_TOO_SMALL error when the allocated buffer to hold the signature does not have an enough size [1]. Note that this is the only error that would not terminate an ongoing signature operation (in case of multi-part), so it can be retried with the right size (returned from the first call by means of an in/out argument). However, the NSS Software Token implementation (checked on NSS v3.67) does not return a CKR_BUFFER_TOO_SMALL error in the aforementioned scenario when the mechanism is CKM_DSA_SHA1 -or others-, and terminates the operation [2] [3] [4] [5] [6].

      For DSA signatures, the signature length is expected to be twice the size of 'q'. Thus, for a 256-bits 'q', the signature will be 64 bytes long.

      If C_SignFinal is called with a 40 bytes signature buffer -or any value below the expected one-, dsa_SignDigest will detect the error and a SEC_ERROR_INVALID_ARGS error will be returned (which is then translated to a CKR_ARGUMENTS_BAD error for the API caller) [7].

      What JDK-11 is doing for DSA keys in P11Signature is setting the expected signature length to be 40 bytes from the Java side [8]. On the native side, the first call to C_SignFinal is tried with that size but, for the reasons described above, the expected CKR_BUFFER_TOO_SMALL error is not seen and the 2nd call is never tried [9] [10].

      JDK-16+ is not affected by this bug because, as part of the JDK-8242332 enhancement, it was fixed [11] [12]. The fix works because when the expected size is set to 0 from the Java side, the native side will set the expected size to the maximum before the call to C_SignFinal [13] [14].

      Note: releases between 11 and 16 may be affected by this bug.

      Even when backporting JDK-8242332 will indirectly fix this bug, JDK-8242332 is a large enhancement affecting APIs and the decision to take it to 11u (or previous releases) involves different risks, effort, priority, etc. so it should be independent in my view.

      [1] - https://docs.oasis-open.org/pkcs11/pkcs11-base/v2.40/os/pkcs11-base-v2.40-os.html
      [2] - https://github.com/nss-dev/nss/blob/8c2610c237ed23d5ce99ad640fa1d4d5167fc3e8/lib/softoken/pkcs11c.c#L2894
      [3] - https://github.com/nss-dev/nss/blob/8c2610c237ed23d5ce99ad640fa1d4d5167fc3e8/lib/softoken/pkcs11c.c#L2052
      [4] - https://github.com/nss-dev/nss/blob/8c2610c237ed23d5ce99ad640fa1d4d5167fc3e8/lib/softoken/pkcs11c.c#L3319
      [5] - https://github.com/nss-dev/nss/blob/8c2610c237ed23d5ce99ad640fa1d4d5167fc3e8/lib/softoken/pkcs11c.c#L3330
      [6] - https://github.com/nss-dev/nss/blob/8c2610c237ed23d5ce99ad640fa1d4d5167fc3e8/lib/softoken/pkcs11c.c#L3348
      [7] - https://github.com/nss-dev/nss/blob/8c2610c237ed23d5ce99ad640fa1d4d5167fc3e8/lib/freebl/dsa.c#L338
      [8] - https://github.com/openjdk/jdk11u/blob/d1541f3012754e0bbda2524bb865c6a971fc4574/src/jdk.crypto.cryptoki/share/classes/sun/security/pkcs11/P11Signature.java#L634
      [9] - https://github.com/openjdk/jdk11u/blob/d1541f3012754e0bbda2524bb865c6a971fc4574/src/jdk.crypto.cryptoki/share/native/libj2pkcs11/p11_sign.c#L252
      [10] - https://github.com/openjdk/jdk11u/blob/d1541f3012754e0bbda2524bb865c6a971fc4574/src/jdk.crypto.cryptoki/share/native/libj2pkcs11/p11_sign.c#L259
      [11] - https://bugs.openjdk.java.net/browse/JDK-8242332
      [12] - https://github.com/openjdk/jdk/commit/78be334c#diff-c6f6f725937a727ae58024887255fb326313a422d767c78b1bb62ed50c8c8ef8L627
      [13] - https://github.com/openjdk/jdk/blob/78be334c3817a1b5840922a9bf1339a40dcc5185/src/jdk.crypto.cryptoki/share/classes/sun/security/pkcs11/P11Signature.java#L613
      [14] - https://github.com/openjdk/jdk/blob/78be334c3817a1b5840922a9bf1339a40dcc5185/src/jdk.crypto.cryptoki/share/native/libj2pkcs11/p11_sign.c#L241


          Issue Links



              mbalao Martin Balao
              mbalao Martin Balao
              0 Vote for this issue
              1 Start watching this issue