Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-8223940

Private key not supported by chosen signature algorithm

    Details

    • Subcomponent:
    • Resolved In Build:
      b23
    • CPU:
      x86_64
    • OS:
      windows_7
    • Verification:
      Not verified

      Backports

        Description

        A DESCRIPTION OF THE PROBLEM :
        When using a PKCS11 keystore (using the SunPKCS11 Provider) for client authentication in TLSv1.2, the client chooses a signature algorithm which does not support the P11PrivateKey.

        The server sends a list of supported SignatureSchemes for use in client authentication. The method sun.security.ssl.SignatureScheme.getPreferableAlgorithm(List<SignatureScheme>, PrivateKey, ProtocolVersion) chooses which SignatureScheme to use in client authentication. It can choose any algorithm for which a provider exists, even if the algorithm's java.security.Provider$Service.supportsParameter(privateKey) == false. In this case it chooses RSASSA-PSS from the provider sun.security.rsa.SunRsaSign.

        Because of this, signing with P11PrivateKey fails (with a misleading message stating no providers support the key, when it's a question of the <algorithm name> implementations not supporting the key):

        Caused by: javax.net.ssl.SSLHandshakeException: Cannot produce CertificateVerify signature
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:128)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:321)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:264)
        at java.base/sun.security.ssl.CertificateVerify$T12CertificateVerifyMessage.<init>(CertificateVerify.java:590)
        at java.base/sun.security.ssl.CertificateVerify$T12CertificateVerifyProducer.produce(CertificateVerify.java:740)
        at java.base/sun.security.ssl.SSLHandshake.produce(SSLHandshake.java:436)
        at java.base/sun.security.ssl.ServerHelloDone$ServerHelloDoneConsumer.consume(ServerHelloDone.java:173)
        at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
        at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444)
        at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:421)
        at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:178)
        at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:164)
        at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1152)
        at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1063)
        at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:402)
        at java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:567)
        at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1356)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1331)
        at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:241)
        <snip>
        Caused by: java.security.InvalidKeyException: No installed provider supports this key: sun.security.pkcs11.P11Key$P11PrivateKey
        at java.base/java.security.Signature$Delegate.chooseProvider(Signature.java:1163)
        at java.base/java.security.Signature$Delegate.engineInitSign(Signature.java:1204)
        at java.base/java.security.Signature.initSign(Signature.java:546)
        at java.base/sun.security.ssl.SignatureScheme.getSignature(SignatureScheme.java:473)
        at java.base/sun.security.ssl.CertificateVerify$T12CertificateVerifyMessage.<init>(CertificateVerify.java:580)
        ... 26 more

        The problem cannot be worked around by adding RSASSA-PSS to the java.security configuration jdk.tls.disabledAlgorithms property.

        STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
        Regular Java TLS client using PKCS11 KeyStore from the SunPKCS11 provider for client authentication.

        TLSv1.2 server with client authentication, with supported_signature_algorithms preferring RSASSA-PSS (not supported by PKCS11 provider) over RSA_PKCS1_SHA256 (supported by PKCS11 provider).

        Open connection from client to server.

        EXPECTED VERSUS ACTUAL BEHAVIOR :
        EXPECTED -
        The client chooses the RSA_PKCS1_SHA256 signature algorithm. The TLS connection is established with PKCS11 for client authentication.
        ACTUAL -
        The client chooses the RSASSA-PSS signature algorithm. The client cannot complete TLS handshake because it cannot use the P11PrivateKey in the signature algorithm.

        CUSTOMER SUBMITTED WORKAROUND :
        Configure server so that supported_signature_algorithms prefers signature algorithms supported by the SunPKCS11 provider (RSA_PKCS1_SHA256, RSA_PKCS1_SHA384, RSA_PKCS1_SHA_512, RSA_SHA224, RSA_PKCS1_SHA1).

        FREQUENCY : always


          Attachments

            Issue Links

              Activity

                People

                • Assignee:
                  xuelei Xue-Lei Fan
                  Reporter:
                  webbuggrp Webbug Group
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  11 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: