Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-8160425

Vectorization with signalling NaN returns wrong result

    Details

    • Type: Bug
    • Status: Closed
    • Priority: P2
    • Resolution: Fixed
    • Affects Version/s: 9
    • Fix Version/s: 9
    • Component/s: hotspot
    • Labels:
      None
    • Subcomponent:
    • Resolved In Build:
      b131
    • CPU:
      x86
    • Verification:
      Verified

      Description

      Attached TestNaNVector.java triggers the problem on 32-bit x86.

      We vectorize the following loop:
      for (int i = 0; i < 1024; ++i) {
        array[i] = (char)0xfff7;
      }
      The array store will be replaced by a 64-bit vector store to four subsequent array elements. The vector should look like this '0xfff7fff7fff7fff7' and is read from the constant table. However, in floating point arithmetic this is a signalling NaN which may be converted to a quiet NaN when processed by the x87 FPU. If the signalling bit is set, the vector ends up in the constant table as '0xfffffff7fff7fff7' which leads to an incorrect result.

      The problem is that the constant vector is passed around as a double in the C code. On x86 32-bit, floating point values are returned via the FPU stack and since the value is a signalling NaN, it's converted to a quiet NaN by the FPU instructions.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                thartmann Tobias Hartmann
                Reporter:
                thartmann Tobias Hartmann
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: