Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-4781903

(fs) Scattering reads seem slow (lnx)

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: P4
    • Resolution: Duplicate
    • Affects Version/s: 1.4.0
    • Fix Version/s: None
    • Component/s: core-libs
    • Labels:
    • Subcomponent:
    • CPU:
      x86
    • OS:
      linux

      Description

      Name: nt126004 Date: 11/20/2002


      FULL PRODUCT VERSION :
      java version "1.4.0"
      Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.0-b92)
      Java HotSpot(TM) Client VM (build 1.4.0-b92, mixed mode)

      and

      java version "1.4.1"
      Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.1-b21)
      Java HotSpot(TM) Client VM (build 1.4.1-b21, mixed mode)


      FULL OPERATING SYSTEM VERSION :
      Linux 2.4.18-5smp
      Red Hat Linux 7.3 2.96-110
      glibc-2.2.5-36

      EXTRA RELEVANT SYSTEM CONFIGURATION :
      total: used: free: shared: buffers: cached:
      Mem: 393924608 386863104 7061504 0 33955840 204648448
      Swap: 797843456 0 797843456


      A DESCRIPTION OF THE PROBLEM :
      I am doing nio performance prototyping, and attempted to
      transpose a 3d, 153x701x1501 byte data volume from i,j,k to
      k,i,j order, using ScatterByteChannel (see attached test case).

      Using vmstat and/or top while the program is running shows
      RSS and SIZE increasing monotonically. Eventually the
      system starts to swap, then thrash, and sometime after that,
      the kernel kills the java processes with an "Out of
      Memory..." error.

      I have run this on Windows 2000/JDK1.4 without any problem
      (see Expected and Actual Results). I am guessing there is a
      leak in the native IO code on Linux.

      STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
      1.java -Xms16m -Xmx200m ijk2kij <infile> "/dev/null" ni nj nk
      where ni, nj, nk are the dimensions of the data volume.
      2. Using -verbose:gc shows garbage collection occuring
      several times during each islice read.
      3. Using the -server option makes no apparent difference.
      4. Modifying or eliminating the -Xms, -Xmx options makes no
      apparent difference.

      EXPECTED VERSUS ACTUAL BEHAVIOR :
      On Win2K, the program takes ~17-40 sec. to read/write each
      islice, about an hour to complete. Performance monitor
      shows available memory decreasing to a few MB within the
      first 50 islices or so, and is stable thereafter, so free'd
      memory is being returned to the system.

      ERROR MESSAGES/STACK TRACES THAT OCCUR :
      Sep 12 17:02:09 mango kernel: Out of Memory: Killed process 32564 (java).
      Sep 12 17:02:11 mango kernel: Out of Memory: Killed process 32565 (java).
      Sep 12 17:02:11 mango kernel: Out of Memory: Killed process 32566 (java).
      Sep 12 17:02:11 mango kernel: Out of Memory: Killed process 32567 (java).
      Sep 12 17:02:11 mango kernel: Out of Memory: Killed process 32568 (java).
      Sep 12 17:02:11 mango kernel: Out of Memory: Killed process 32569 (java).
      Sep 12 17:02:11 mango kernel: Out of Memory: Killed process 32570 (java).
      Sep 12 17:02:11 mango kernel: Out of Memory: Killed process 32571 (java).
      Sep 12 17:02:11 mango kernel: Out of Memory: Killed process 32572 (java).


      REPRODUCIBILITY :
      This bug can be reproduced always.

      ---------- BEGIN SOURCE ----------
      import java.io.*;
      import java.nio.*;
      import java.nio.channels.*;

      /**
       * Describe class <code>ijk2kij</code> here.
       *
       * @author <a href="mailto:###@###.###">Kai Peter Ford</a>
       * @version 1.0
       */
      public class ijk2kij
      {
            /**
             * Transpose file from i,j,k to k,i,j
             *
             * @param argv a <code>String[]</code> value
             */
            public static void main(String[] argv)
            {
      if ( argv.length != 5 ) {
      System.err.println( "Usage:" );
      System.err.println( "java ijk2kij in_file out_file ni nj nk" );
      System.exit(0);
      }
      try {
      System.err.println("ijk2kij: reading dataset "+argv[0]+ " ...");
      long start = System.currentTimeMillis();

      FileChannel source = new FileInputStream(argv[0]).getChannel();
      FileChannel dest = new FileOutputStream(argv[1]).getChannel();
      int ni = Integer.parseInt(argv[2]);
      int nj = Integer.parseInt(argv[3]);
      int nk = Integer.parseInt(argv[4]);

      long nr; // # bytes read (scattered)
      long slot=0; // Current line position in output file
      long skip = nj*ni; // Size of one kslice
      int i,j,k,n;
      ByteBuffer wbuf; // = wbufs[k]
      ByteBuffer wbufs[] = new ByteBuffer[nk];
      // Create nk ByteBuffers of capacity nj for storing an
      // nk x nj islice:
      for ( k=0; k<nk; k++ ) {
      wbufs[k] = ByteBuffer.allocateDirect(nj);
      // Set up to scatter the first line to one sample
      // (byte) per wbuf:
      wbufs[k].limit(1);
      }

      long s2,s1 = System.currentTimeMillis(); // debug
      Runtime rt = Runtime.getRuntime(); // debug
      for ( i=0; i<ni; i++ ) {
      // Get the i'th islice into wbufs:
      for (j=0; j<(nj-1); j++ ) {
      nr = 0;
      // Read nk samples and scatter them, 1/wbuf.
      while ( nr < nk )nr += source.read(wbufs);
      System.err.println("ijk2kij: islice "+i+", jslice= "+j+" read; free = "
      +rt.freeMemory()+"; total = "
      +rt.totalMemory()); // debug
      // Increment each buffer limit by one (#
      // bytes/sample); this is why one can't "scatter"
      // more than a line at a time:
      n = j+2;
      for(k=0; k<nk; k++ ) wbufs[k].limit(n);
      }
      // Do the last j:
      nr = 0;
      while ( nr < nk )nr += source.read(wbufs);
      // The i'th islice is now stored in the wbufs as
      // constant k lines of length nj.... Write
      // them out.
      slot = i*nj;
      for(k=0; k<nk; k++ ) {
      wbuf = wbufs[k];
      wbuf.position(0).limit(nj); // Prepare to write k'th line
      while (wbuf.hasRemaining())
      dest.write(wbuf, slot); // Write line for i'th islice, k'th kslice, to file
      slot+=skip; // Increment file position of next line
      wbuf.position(0).limit(1); // Set up for next islice scatter.
      }
      s2 = System.currentTimeMillis(); // debug
      System.err.println("ijk2kij: Time to read/write islice "+i+
      " = "+((float)(0.001*(s2-s1)))+" seconds."); // debug
      s1 = s2; // debug
      }
      source.close();
      dest.close();

      long stop = System.currentTimeMillis();
      System.err.println("ijk2kij: Time to write file =
      "+((float)0.001*(stop-start))+" seconds.");


      } catch(Exception ex) {
      System.err.println(ex.getMessage());
      ex.printStackTrace(System.err);
      } catch(Error er) {
      System.err.println(er.getMessage());
      er.printStackTrace(System.err);
      }
            }
      }



      ---------- END SOURCE ----------

      CUSTOMER WORKAROUND :
      Don't use ScatterByteChannel for disk io, read entire islice
      and transpose in code.
      Need to do this for performance reasons anyway.
      (Review ID: 164467)
      ======================================================================

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              nthompsosunw Nathanael Thompson (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Imported:
                Indexed: