Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-8271199

Mutual TLS handshake fails signing client certificate with custom sensitive PKCS11 key

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open
    • Priority: P3
    • Resolution: Unresolved
    • Affects Version/s: 8, 11, 13, 15, 17, 18
    • Fix Version/s: None
    • Component/s: security-libs

      Description

      There is a use case with a custom PKCS11 provider (IAIK) which fails during the mutual TLS handshake.
      TLS handshake fails to create a signature for the client certificate using sensitive PKCS11 RSA key.

      EXPECTED BEHAVIOUR:
      JDK selects custom RSASSA-PSS Signature implementation from the IAIK provider, signs client certificate, and completes TLS handshake

      ACTUAL BEHAVIOUR:
      JDK selects RSASSA-PSS Signature implementation from the SunRSASign provider and fails signing client certificate with the following stack trace:
        java.lang.UnsupportedOperationException: Prime P value is sensitive.
        at iaik.pkcs.pkcs11.provider.keys.IAIKPKCS11RsaPrivateKey.getPrimeP(Unknown Source)
        at java.base/sun.security.rsa.RSACore.crtCrypt(RSACore.java:176)
        at java.base/sun.security.rsa.RSACore.rsa(RSACore.java:130)
        at java.base/sun.security.rsa.RSAPSSSignature.engineSign(RSAPSSSignature.java:385)
        at java.base/java.security.Signature$Delegate.engineSign(Signature.java:1404)
        at java.base/java.security.Signature.sign(Signature.java:712)
        at java.base/sun.security.ssl.CertificateVerify$T12CertificateVerifyMessage.<init>(CertificateVerify.java:612)
        at java.base/sun.security.ssl.CertificateVerify$T12CertificateVerifyProducer.produce(CertificateVerify.java:764)
        at java.base/sun.security.ssl.SSLHandshake.produce(SSLHandshake.java:440)
        at java.base/sun.security.ssl.ServerHelloDone$ServerHelloDoneConsumer.consume(ServerHelloDone.java:182)
        at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396)
        at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1277)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1264)
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1209)

      The implementation uses a private PKCS11 key from the IAIK provider :
      http://javadoc.iaik.tugraz.at/pkcs11_provider/current/iaik/pkcs/pkcs11/provider/keys/IAIKPKCS11RsaPrivateKey.html

      This issue happens because of SignatureScheme.getSigner() selects RSAPSSSignature signer from the SunRSASign provider and RSAPSSSignature.engineInitSign() successfully initialises signature object.

      Stack trace to RSAPSSSignature.engineInitSign():
      java.base/sun.security.rsa.RSAPSSSignature.engineInitSign(RSAPSSSignature.java:148)
         at java.base/java.security.SignatureSpi.engineInitSign(SignatureSpi.java:167)
         at java.base/java.security.Signature$Delegate.tryOperation(Signature.java:1318)
         at java.base/java.security.Signature$Delegate.chooseProvider(Signature.java:1270)
         at java.base/java.security.Signature$Delegate.engineInitSign(Signature.java:1382)
         at java.base/java.security.Signature.initSign(Signature.java:683)
         at java.base/java.security.Signature$1.initSign(Signature.java:147)
         at java.base/sun.security.util.SignatureUtil.initSignWithParam(SignatureUtil.java:194)
         at java.base/sun.security.ssl.SignatureScheme.getSigner(SignatureScheme.java:595)
         at java.base/sun.security.ssl.SignatureScheme.getSignerOfPreferableAlgorithm(SignatureScheme.java:542)
         at java.base/sun.security.ssl.CertificateVerify$T12CertificateVerifyMessage.<init>(CertificateVerify.java:593)
         at java.base/sun.security.ssl.CertificateVerify$T12CertificateVerifyProducer.produce(CertificateVerify.java:764)

      As result, JDK does not try other signature providers. Later, RSAPSSSignature fails to sign the client certificate with the PKCS11 sensitive key.

      This issue is possibly related to JDK-8226374 and JDK-8222937

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              abakhtin Alexey Bakhtin
              Reporter:
              abakhtin Alexey Bakhtin
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Dates

                Created:
                Updated: