Any way to persist OutputStream in DiskData?
Wednesday, 14 March 2012 · 5 messages · last reply 16 Mar 2012
DG
Dave Glick14 Mar 2012 20:09 #
Hi,
I've been doing some profiling recently and noticed one area in particular that appears to be using a lot of cycles. In calls to DiskData.write() the final DataOutput.close() call results in 20% - 40% of overall execution time in a variety of cases. The following screen capture shows the aggregated call stack (though it might look a little funny since this is the cross-compiled version used for Nxdb). Admittedly, this is probably an issue that just impacts me. Based on the trace, I suspect the issue is related to the releasing of handles rather than the actual disk writes (though I could be wrong - I'm not entirely sure what the cross-compiler does behind the scenes to support Java file descriptors).
In any case, my question boils down to wondering if there is a way to force the DiskData class to use a persistent DataOutput instance instead of creating and closing a new one for every write? Instead, could a DataOutput be kept open until the database is "unpinned" and simply flushed on each write? Would this cause any problems with the current GUI, server, etc. uses?
Thanks,
Dave
DP
Dimitar Popov14 Mar 2012 21:18 #
Hi Dave,
First, thanks for the analysis!
Quoted text · 19:09:52 · 7 lines
Am Mittwoch, 14. März 2012, 19:09:52 schrieb Dave Glick: > In any case, my question boils down to wondering if there is a way to force > the DiskData class to use a persistent DataOutput instance instead of > creating and closing a new one for every write? Instead, could a DataOutput > be kept open until the database is "unpinned" and simply flushed on each > write? Would this cause any problems with the current GUI, server, etc. > uses?
I think you are right, and it would be, of course, much more efficient not to open/close the DataOutput on each call of write(). I don't see any problem of keeping the file with the meta data open during the whole life-cycle of a DiskData instance (I may be however wrong, so Christian should say his opinion).
The problem which I see is how to make that, because the class org.basex.io.out.DataOutput does not have a method to reset the file offset to the beginning of the file. Therefore, either such a method should be added, or the meta-data should be written using a RandomAccessFile (e.g. org.basex.io.random.DiskAccess).
Regards,
Dimitar
CG
Hi Dave,
thanks for your analysis - profound as usual. You may get much better results by setting the
AUTOFLUSH option to false [1]. Please tell me if you've done that already..
Christian
[1] http://docs.basex.org/wiki/Options#AUTOFLUSHQuoted text · Dave Glick · 16 lines
On Wed, Mar 14, 2012 at 8:09 PM, Dave Glick <dglick at dracorp.com> wrote: > Hi, > > I've been doing some profiling recently and noticed one area in particular that appears to be using a lot of cycles. In calls to DiskData.write() the final DataOutput.close() call results in 20% - 40% of overall execution time in a variety of cases. The following screen capture shows the aggregated call stack (though it might look a little funny since this is the cross-compiled version used for Nxdb). Admittedly, this is probably an issue that just impacts me. Based on the trace, I suspect the issue is related to the releasing of handles rather than the actual disk writes (though I could be wrong - I'm not entirely sure what the cross-compiler does behind the scenes to support Java file descriptors). > > In any case, my question boils down to wondering if there is a way to force the DiskData class to use a persistent DataOutput instance instead of creating and closing a new one for every write? Instead, could a DataOutput be kept open until the database is "unpinned" and simply flushed on each write? Would this cause any problems with the current GUI, server, etc. uses? > > Thanks, > > Dave > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >
DG
Dave Glick16 Mar 2012 14:59 #
Christian,
I had played around with the
AUTOFLUSH option and did notice some improvement, which is great for my own uses when appropriate. If that option is set to false, how often does the database actually get flushed? Only on close? Does it flush before index rebuilds from Optimize?
With the OutputStream question, I was more interested in systematic sources of delay that could potentially be mitigated (thinking about other users of Nxdb). This seemed like an instance where there might be room for improvement - however, I'm not sure the problem even exists outside the cross-compiled case.
Thanks,
DaveQuoted text · Christian Grün · 26 lines
-----Original Message----- From: Christian Grün [mailto:christian.gruen at gmail.com] Sent: Wednesday, March 14, 2012 5:03 PM To: Dave Glick Cc: BaseX Subject: Re: [basex-talk] Any way to persist OutputStream in DiskData? Hi Dave, thanks for your analysis - profound as usual. You may get much better results by setting the AUTOFLUSH option to false [1]. Please tell me if you've done that already.. Christian [1] http://docs.basex.org/wiki/Options#AUTOFLUSH On Wed, Mar 14, 2012 at 8:09 PM, Dave Glick <dglick at dracorp.com> wrote: > Hi, > > I've been doing some profiling recently and noticed one area in particular that appears to be using a lot of cycles. In calls to DiskData.write() the final DataOutput.close() call results in 20% - 40% of overall execution time in a variety of cases. The following screen capture shows the aggregated call stack (though it might look a little funny since this is the cross-compiled version used for Nxdb). Admittedly, this is probably an issue that just impacts me. Based on the trace, I suspect the issue is related to the releasing of handles rather than the actual disk writes (though I could be wrong - I'm not entirely sure what the cross-compiler does behind the scenes to support Java file descriptors). > > In any case, my question boils down to wondering if there is a way to force the DiskData class to use a persistent DataOutput instance instead of creating and closing a new one for every write? Instead, could a DataOutput be kept open until the database is "unpinned" and simply flushed on each write? Would this cause any problems with the current GUI, server, etc. uses? > > Thanks, > > Dave > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >
CG
Christian Grün16 Mar 2012 18:45 #
Hi Dave,
Quoted text · Dave Glick · 1 line
> I had played around with the AUTOFLUSH option and did notice some improvement, which is great for my own uses when appropriate. If that option is set to false, how often does the database actually get flushed? Only on close?
Yes. The number of flushes will be reduced as much as possible. If you want to force flushing, you'll either need to execute the FLUSH command or close the database.
Quoted text · Dave Glick · 1 line
> With the OutputStream question, I was more interested in systematic sources of delay that could potentially be mitigated (thinking about other users of Nxdb). This seemed like an instance where there might be room for improvement - however, I'm not sure the problem even exists outside the cross-compiled case.
To be honest, I'm not sure how this would need to look like in the code.. Do you believe that the existing "autoflush" option will help in that case, or would you like to have it as a potential replacement?
Best,
Christian
Written for BaseX 7.1. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery