[Clas12_software] java file size test and jevio

[David Heddle]

The problem with files > 2GB in jevio is nasty. Java can "inherently"
handle files greater than 2GB if they are read traditionally. It is the
memory mapping that causes the problem. In jevio there is, for example, the
call:

mappedByteBuffer = inputChannel.map(FileChannel.MapMode.READ_ONLY, 0L, sz);

This is creating the buffer, in memory, of a region of a file. The 0L
argument is the offset (i.e., in this case the map begins at the beginning
of the file) and the second argument is the length, which in jevio is
essentially the size of the file.

The java documentation gives the weird bad news. The third argument (sz, in
this case)  is a long, but it is restricted to INTEGER.MAXINT (which is 2
GB). That's right: it is a long (64 bit) value that will throw an exception
if it is bigger than the max for a 32 bit int.

To summarize: jevio assumes it can map the entire file to memory--but this
will fail even if you have a 64 bit machine and sufficient (i.e. more than
2GB) of memory.

There is a request for a change at Oracle, but it doesn't seen to be high
priority. They say the only workaround is to work your way through big
files:

offset = 0L
while (!done) {
  long sz = Math.min(remainingSize, INTEGER.MAXINT);
  mappedByteBuffer = inputChannel.map(FileChannel.MapMode.READ_ONLY,
offset, sz);
         {do stuff}
   offset += sz;
  mappedByteBuffer = inputChannel.map(FileChannel.MapMode.READ_ONLY, 0L,
sz);
}

I suspect that this would be a  very nontrivial change to jevio withs lots
of debugging. But I think we need to request it.

Note that we basically have to use memory mapped io--it is about 10x faster
than traditional disk access IO.

-- 
David P. Heddle, Ph.D.
Associate Professor of Physics
Christopher Newport University
Newport News, VA 23606

757.594.8434 (CNU)
757.269.5719 (Jefferson Lab)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.jlab.org/pipermail/clas12_software/attachments/20130711/c0adcdeb/attachment-0002.html>