BaseX Script memory leak . . ?
Monday, 20 May 2013 · 8 messages · last reply 12 Jun 2013
C
Christopher.R.Ball20 May 2013 06:32 #
I have a BaseX script (.bxs) I am running that does queries in batches (sets of 5k documents), but as it progresses it bogs down in speed, does not release memory between sets even if I force it to close and reopen the db between queries, and eventually runs out of memory.
But, if I break the same BaseX script into separate files still doing the same exact batches it is extremely fast and memory efficient.
Very suggestive of a memory leak . . .
I am running on BaseX 7.6.1 Beta.
Any thoughts?
Is there a way to force the script to do garbage collection?
BC
Ben Companjen20 May 2013 10:33 #
I recognise your problem, and reported it, but never got back to it with more details. I used BaseX client/server 7.5 beta. My first database contained 2.7 million documents, but I created a new one from an exported subset of 700k documents. That helped lower the memory use directly after loading the DB.
Any chance you use the SQL module in your processing?
My guess was that it had been a design choice to keep previously opened documents from a database in use in memory. But running out of memory probably wasn't ;)
Ben
Quoted text · Christopher.R.Ball · 22 lines
On 20 May 2013 04:32, Christopher.R.Ball <christopher.r.ball at gmail.com> wrote: > I have a BaseX script (.bxs) I am running that does queries in batches (sets > of 5k documents), but as it progresses it bogs down in speed, does not > release memory between sets even if I force it to close and reopen the db > between queries, and eventually runs out of memory. > > But, if I break the same BaseX script into separate files still doing the > same exact batches it is extremely fast and memory efficient. > > Very suggestive of a memory leak . . . > > I am running on BaseX 7.6.1 Beta. > > Any thoughts? > > Is there a way to force the script to do garbage collection? > > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
CG
Christian Grün20 May 2013 12:24 #
Hi Christopher, hi Ben,
yes, this sounds like unwanted behavior, and I believe it should be fixable as the commands scripts I’ve been working with didn’t cause memory leaks. I’ll be glad to track down the possible issues. Could (one/both) of you pass me on a script that causes the problems?
Christian
PS: I would be grateful if you could additionally check if the problem persists in the latest stable snapshot.
Quoted text · Ben Companjen · 41 lines
On Mon, May 20, 2013 at 10:33 AM, Ben Companjen <bencompanjen at gmail.com> wrote: > I recognise your problem, and reported it, but never got back to it > with more details. I used BaseX client/server 7.5 beta. My first > database contained 2.7 million documents, but I created a new one from > an exported subset of 700k documents. That helped lower the memory use > directly after loading the DB. > > Any chance you use the SQL module in your processing? > > My guess was that it had been a design choice to keep previously > opened documents from a database in use in memory. But running out of > memory probably wasn't ;) > > Ben > > On 20 May 2013 04:32, Christopher.R.Ball <christopher.r.ball at gmail.com> wrote: >> I have a BaseX script (.bxs) I am running that does queries in batches (sets >> of 5k documents), but as it progresses it bogs down in speed, does not >> release memory between sets even if I force it to close and reopen the db >> between queries, and eventually runs out of memory. >> >> But, if I break the same BaseX script into separate files still doing the >> same exact batches it is extremely fast and memory efficient. >> >> Very suggestive of a memory leak . . . >> >> I am running on BaseX 7.6.1 Beta. >> >> Any thoughts? >> >> Is there a way to force the script to do garbage collection? >> >> >> _______________________________________________ >> BaseX-Talk mailing list >> BaseX-Talk at mailman.uni-konstanz.de >> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
C
Christopher.Ball23 May 2013 23:17 #
Ok. I will 1st upgrade to the latest stable snapshot (http://files.basex.org/releases/latest/). If that does not resolve the issue I will see if I can great a repeatable example with a more portable data sample.
Regards,
Christopher
Quoted text · Christian Grün · 70 lines
On 5/20/2013 6:24 AM, Christian Grün wrote: > Hi Christopher, hi Ben, > > yes, this sounds like unwanted behavior, and I believe it should be > fixable as the commands scripts I’ve been working with didn’t cause > memory leaks. I’ll be glad to track down the possible issues. Could > (one/both) of you pass me on a script that causes the problems? > > Christian > > PS: I would be grateful if you could additionally check if the problem > persists in the latest stable snapshot. > ___________________________ > > On Mon, May 20, 2013 at 10:33 AM, Ben Companjen <bencompanjen at gmail.com> wrote: > > > > > I recognise your problem, and reported it, but never got back to it > > with more details. I used BaseX client/server 7.5 beta. My first > > database contained 2.7 million documents, but I created a new one from > > an exported subset of 700k documents. That helped lower the memory use > > directly after loading the DB. > > > > Any chance you use the SQL module in your processing? > > > > My guess was that it had been a design choice to keep previously > > opened documents from a database in use in memory. But running out of > > memory probably wasn't ;) > > > > Ben > > > > On 20 May 2013 04:32, Christopher.R.Ball <christopher.r.ball at gmail.com> wrote: > > > > > > > > > I have a BaseX script (.bxs) I am running that does queries in batches (sets > > > of 5k documents), but as it progresses it bogs down in speed, does not > > > release memory between sets even if I force it to close and reopen the db > > > between queries, and eventually runs out of memory. > > > > > > But, if I break the same BaseX script into separate files still doing the > > > same exact batches it is extremely fast and memory efficient. > > > > > > Very suggestive of a memory leak . . . > > > > > > I am running on BaseX 7.6.1 Beta. > > > > > > Any thoughts? > > > > > > Is there a way to force the script to do garbage collection? > > > > > > > > > _______________________________________________ > > > BaseX-Talk mailing list > > > BaseX-Talk at mailman.uni-konstanz.de > > > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > > > > > > > > > > > > > _______________________________________________ > > BaseX-Talk mailing list > > BaseX-Talk at mailman.uni-konstanz.de > > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > > > > > > > >
C
Christian,
So I have finally upgraded to BaseX 7.7 and found I am still having the out of memory issue.
Given the size and nature of the data I am working with I am at a loss of how to provide you with a simple example that replicates the problem.
On the flip side, one behavior I am noticing is that breaking the work in to discrete chunks in separate batch scripts gives dramatically faster performance and avoids the memory error. This strongly suggests that something is preventing garbage collection between unrelated tasks in a batch script.
Is there any way I can force garbage collection in a batch script? I tried closing and reopening databases but that had no effect (actually shocked that it did not).
Let me know,
Christopher
Quoted text · Christian Grün · 70 lines
On 5/20/2013 6:24 AM, Christian Grün wrote: > Hi Christopher, hi Ben, > > yes, this sounds like unwanted behavior, and I believe it should be > fixable as the commands scripts I’ve been working with didn’t cause > memory leaks. I’ll be glad to track down the possible issues. Could > (one/both) of you pass me on a script that causes the problems? > > Christian > > PS: I would be grateful if you could additionally check if the problem > persists in the latest stable snapshot. > ___________________________ > > On Mon, May 20, 2013 at 10:33 AM, Ben Companjen <bencompanjen at gmail.com> wrote: > > > > > I recognise your problem, and reported it, but never got back to it > > with more details. I used BaseX client/server 7.5 beta. My first > > database contained 2.7 million documents, but I created a new one from > > an exported subset of 700k documents. That helped lower the memory use > > directly after loading the DB. > > > > Any chance you use the SQL module in your processing? > > > > My guess was that it had been a design choice to keep previously > > opened documents from a database in use in memory. But running out of > > memory probably wasn't ;) > > > > Ben > > > > On 20 May 2013 04:32, Christopher.R.Ball <christopher.r.ball at gmail.com> wrote: > > > > > > > > > I have a BaseX script (.bxs) I am running that does queries in batches (sets > > > of 5k documents), but as it progresses it bogs down in speed, does not > > > release memory between sets even if I force it to close and reopen the db > > > between queries, and eventually runs out of memory. > > > > > > But, if I break the same BaseX script into separate files still doing the > > > same exact batches it is extremely fast and memory efficient. > > > > > > Very suggestive of a memory leak . . . > > > > > > I am running on BaseX 7.6.1 Beta. > > > > > > Any thoughts? > > > > > > Is there a way to force the script to do garbage collection? > > > > > > > > > _______________________________________________ > > > BaseX-Talk mailing list > > > BaseX-Talk at mailman.uni-konstanz.de > > > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > > > > > > > > > > > > > _______________________________________________ > > BaseX-Talk mailing list > > BaseX-Talk at mailman.uni-konstanz.de > > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > > > > > > > >
AH
Alexander Holupirek12 Jun 2013 08:45 #
Christopher,
it may be sufficient if you can pass the script (.bxs) file that you use to process the data.
Would that be possible?
Alex
Quoted text · Christopher.Ball · 82 lines
On 12.06.2013, at 02:46, "Christopher.Ball" <christopher.ball at metaheuristica.com> wrote: > Christian, > So I have finally upgraded to BaseX 7.7 and found I am still having the out of memory issue. > > Given the size and nature of the data I am working with I am at a loss of how to provide you with a simple example that replicates the problem. > > On the flip side, one behavior I am noticing is that breaking the work in to discrete chunks in separate batch scripts gives dramatically faster performance and avoids the memory error. This strongly suggests that something is preventing garbage collection between unrelated tasks in a batch script. > > Is there any way I can force garbage collection in a batch script? I tried closing and reopening databases but that had no effect (actually shocked that it did not). > > Let me know, > > Christopher > > On 5/20/2013 6:24 AM, Christian Grün wrote: >> Hi Christopher, hi Ben, >> >> yes, this sounds like unwanted behavior, and I believe it should be >> fixable as the commands scripts I’ve been working with didn’t cause >> memory leaks. I’ll be glad to track down the possible issues. Could >> (one/both) of you pass me on a script that causes the problems? >> >> Christian >> >> PS: I would be grateful if you could additionally check if the problem >> persists in the latest stable snapshot. >> ___________________________ >> >> On Mon, May 20, 2013 at 10:33 AM, Ben Companjen >> <bencompanjen at gmail.com> >> wrote: >> >>> I recognise your problem, and reported it, but never got back to it >>> with more details. I used BaseX client/server 7.5 beta. My first >>> database contained 2.7 million documents, but I created a new one from >>> an exported subset of 700k documents. That helped lower the memory use >>> directly after loading the DB. >>> >>> Any chance you use the SQL module in your processing? >>> >>> My guess was that it had been a design choice to keep previously >>> opened documents from a database in use in memory. But running out of >>> memory probably wasn't ;) >>> >>> Ben >>> >>> On 20 May 2013 04:32, Christopher.R.Ball >>> <christopher.r.ball at gmail.com> >>> wrote: >>> >>>> I have a BaseX script (.bxs) I am running that does queries in batches (sets >>>> of 5k documents), but as it progresses it bogs down in speed, does not >>>> release memory between sets even if I force it to close and reopen the db >>>> between queries, and eventually runs out of memory. >>>> >>>> But, if I break the same BaseX script into separate files still doing the >>>> same exact batches it is extremely fast and memory efficient. >>>> >>>> Very suggestive of a memory leak . . . >>>> >>>> I am running on BaseX 7.6.1 Beta. >>>> >>>> Any thoughts? >>>> >>>> Is there a way to force the script to do garbage collection? >>>> >>>> >>>> _______________________________________________ >>>> BaseX-Talk mailing list >>>> >>>> BaseX-Talk at mailman.uni-konstanz.de >>>> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >>> _______________________________________________ >>> BaseX-Talk mailing list >>> >>> BaseX-Talk at mailman.uni-konstanz.de >>> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- Dr. Alexander Holupirek
|-- Room E 221, 0049 7531 88 2188 (phone) 3577 (fax)
|-- Database & Information Systems Group, U Konstanz
`-- https://scikon.uni-konstanz.de/personen/alexander.holupirek/
C
Christopher.Ball12 Jun 2013 15:28 #
Alex,
Here is the script (.bxs file) contents in its partitioned form (broken out into 6 seperate scripts rather than one script):
Quoted text · Alexander Holupirek · 45 lines
> SET STRIPNS true > > SET ADDCACHE true > > SET TEXTINDEX false > > SET ATTRINDEX false > > > > OPEN Release-Canonicals-Comparative > > > > XQUERY db:output('
 -- ' || current-time() || ' -- > 
') > > > > XQUERY db:output("
#12") > > SET BINDINGS > $db=Release-Canonicals-Comparative,$containerSetStart=110001,$containerSetCount=10000 > > RUN > ..\webapp\release-identification\xquery\generate-comparison-db.xq > > > > XQUERY db:output("
#13") > > SET BINDINGS > $db=Release-Canonicals-Comparative,$containerSetStart=120001,$containerSetCount=10000 > > RUN > ..\webapp\release-identification\xquery\generate-comparison-db.xq > > > > XQUERY db:output('
 -- ' || current-time() || ' -- > 
') > > > >
If I run it in this partitioned form, it is quite fast, roughly 5 minutes per "RUN" command. If I concatenate them all together, it progressively slows down, consumes all the available memory, starts to swap memory to disc, cpu climbs to 100% and eventually fails with a memory error.
Christopher
Quoted text · Alexander Holupirek · 127 lines
On 6/12/2013 2:45 AM, Alexander Holupirek wrote: > Christopher, > > it may be sufficient if you can pass the script (.bxs) file that you use to process the data. > > Would that be possible? > Alex > > On 12.06.2013, at 02:46, "Christopher.Ball" <christopher.ball at metaheuristica.com> wrote: > > > > > > Christian, > > So I have finally upgraded to BaseX 7.7 and found I am still having the out of memory issue. > > > > Given the size and nature of the data I am working with I am at a loss of how to provide you with a simple example that replicates the problem. > > > > On the flip side, one behavior I am noticing is that breaking the work in to discrete chunks in separate batch scripts gives dramatically faster performance and avoids the memory error. This strongly suggests that something is preventing garbage collection between unrelated tasks in a batch script. > > > > Is there any way I can force garbage collection in a batch script? I tried closing and reopening databases but that had no effect (actually shocked that it did not). > > > > Let me know, > > > > Christopher > > > > On 5/20/2013 6:24 AM, Christian Grün wrote: > > > > > > > > > Hi Christopher, hi Ben, > > > > > > yes, this sounds like unwanted behavior, and I believe it should be > > > fixable as the commands scripts I’ve been working with didn’t cause > > > memory leaks. I’ll be glad to track down the possible issues. Could > > > (one/both) of you pass me on a script that causes the problems? > > > > > > Christian > > > > > > PS: I would be grateful if you could additionally check if the problem > > > persists in the latest stable snapshot. > > > ___________________________ > > > > > > On Mon, May 20, 2013 at 10:33 AM, Ben Companjen > > > <bencompanjen at gmail.com> > > > wrote: > > > > > > > > > > > > > > > > I recognise your problem, and reported it, but never got back to it > > > > with more details. I used BaseX client/server 7.5 beta. My first > > > > database contained 2.7 million documents, but I created a new one from > > > > an exported subset of 700k documents. That helped lower the memory use > > > > directly after loading the DB. > > > > > > > > Any chance you use the SQL module in your processing? > > > > > > > > My guess was that it had been a design choice to keep previously > > > > opened documents from a database in use in memory. But running out of > > > > memory probably wasn't ;) > > > > > > > > Ben > > > > > > > > On 20 May 2013 04:32, Christopher.R.Ball > > > > <christopher.r.ball at gmail.com> > > > > wrote: > > > > > > > > > > > > > > > > > > > > > I have a BaseX script (.bxs) I am running that does queries in batches (sets > > > > > of 5k documents), but as it progresses it bogs down in speed, does not > > > > > release memory between sets even if I force it to close and reopen the db > > > > > between queries, and eventually runs out of memory. > > > > > > > > > > But, if I break the same BaseX script into separate files still doing the > > > > > same exact batches it is extremely fast and memory efficient. > > > > > > > > > > Very suggestive of a memory leak . . . > > > > > > > > > > I am running on BaseX 7.6.1 Beta. > > > > > > > > > > Any thoughts? > > > > > > > > > > Is there a way to force the script to do garbage collection? > > > > > > > > > > > > > > > _______________________________________________ > > > > > BaseX-Talk mailing list > > > > > > > > > > BaseX-Talk at mailman.uni-konstanz.de > > > > > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > BaseX-Talk mailing list > > > > > > > > BaseX-Talk at mailman.uni-konstanz.de > > > > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > BaseX-Talk mailing list > > BaseX-Talk at mailman.uni-konstanz.de > > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > > > > > > > > -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- > Dr. Alexander Holupirek > |-- Room E 221, 0049 7531 88 2188 (phone) 3577 (fax) > |-- Database & Information Systems Group, U Konstanz > `-- https://scikon.uni-konstanz.de/personen/alexander.holupirek/ > > > > > >
CG
Christian Grün12 Jun 2013 21:18 #
Hi Christopher,
thanks for the script. It gives us a first hint on what may go on internally. Nex ,some profiling output could be helpful, so could you please run the complete script with the following Java option..
java -Xrunhprof:cpu=samples,depth=15 .....and send me the java.hprof.txt file, which will be stored in the directory the code is started from? The Java profiler also provides a "heap" option (see -Xrunhprof:help), but I don’t actually know how to reasonably interpret the output..
For testing purposes, it is sometimes helpful to further reduce the amount of memory that’s assigned to the JVM (via -Xmx...m).
Best,
Christian
___________________________
2013/6/12 Christopher.Ball <christopher.ball at metaheuristica.com>:
Quoted text · Christopher.Ball · 150 lines
> Alex, > > Here is the script (.bxs file) contents in its partitioned form (broken out > into 6 seperate scripts rather than one script): > > SET STRIPNS true > SET ADDCACHE true > SET TEXTINDEX false > SET ATTRINDEX false > > OPEN Release-Canonicals-Comparative > > XQUERY db:output('
 -- ' || current-time() || ' -- 
') > > XQUERY db:output("
#12") > SET BINDINGS > $db=Release-Canonicals-Comparative,$containerSetStart=110001,$containerSetCount=10000 > RUN ..\webapp\release-identification\xquery\generate-comparison-db.xq > > XQUERY db:output("
#13") > SET BINDINGS > $db=Release-Canonicals-Comparative,$containerSetStart=120001,$containerSetCount=10000 > RUN ..\webapp\release-identification\xquery\generate-comparison-db.xq > > XQUERY db:output('
 -- ' || current-time() || ' -- 
') > > If I run it in this partitioned form, it is quite fast, roughly 5 minutes > per "RUN" command. If I concatenate them all together, it progressively > slows down, consumes all the available memory, starts to swap memory to > disc, cpu climbs to 100% and eventually fails with a memory error. > > Christopher > > > > > On 6/12/2013 2:45 AM, Alexander Holupirek wrote: > > Christopher, > > it may be sufficient if you can pass the script (.bxs) file that you use to > process the data. > > Would that be possible? > Alex > > On 12.06.2013, at 02:46, "Christopher.Ball" > <christopher.ball at metaheuristica.com> wrote: > > Christian, > So I have finally upgraded to BaseX 7.7 and found I am still having the out > of memory issue. > > Given the size and nature of the data I am working with I am at a loss of > how to provide you with a simple example that replicates the problem. > > On the flip side, one behavior I am noticing is that breaking the work in to > discrete chunks in separate batch scripts gives dramatically faster > performance and avoids the memory error. This strongly suggests that > something is preventing garbage collection between unrelated tasks in a > batch script. > > Is there any way I can force garbage collection in a batch script? I tried > closing and reopening databases but that had no effect (actually shocked > that it did not). > > Let me know, > > Christopher > > On 5/20/2013 6:24 AM, Christian Grün wrote: > > Hi Christopher, hi Ben, > > yes, this sounds like unwanted behavior, and I believe it should be > fixable as the commands scripts I’ve been working with didn’t cause > memory leaks. I’ll be glad to track down the possible issues. Could > (one/both) of you pass me on a script that causes the problems? > > Christian > > PS: I would be grateful if you could additionally check if the problem > persists in the latest stable snapshot. > ___________________________ > > On Mon, May 20, 2013 at 10:33 AM, Ben Companjen > <bencompanjen at gmail.com> > wrote: > > I recognise your problem, and reported it, but never got back to it > with more details. I used BaseX client/server 7.5 beta. My first > database contained 2.7 million documents, but I created a new one from > an exported subset of 700k documents. That helped lower the memory use > directly after loading the DB. > > Any chance you use the SQL module in your processing? > > My guess was that it had been a design choice to keep previously > opened documents from a database in use in memory. But running out of > memory probably wasn't ;) > > Ben > > On 20 May 2013 04:32, Christopher.R.Ball > <christopher.r.ball at gmail.com> > wrote: > > I have a BaseX script (.bxs) I am running that does queries in batches (sets > of 5k documents), but as it progresses it bogs down in speed, does not > release memory between sets even if I force it to close and reopen the db > between queries, and eventually runs out of memory. > > But, if I break the same BaseX script into separate files still doing the > same exact batches it is extremely fast and memory efficient. > > Very suggestive of a memory leak . . . > > I am running on BaseX 7.6.1 Beta. > > Any thoughts? > > Is there a way to force the script to do garbage collection? > > > _______________________________________________ > BaseX-Talk mailing list > > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > > _______________________________________________ > BaseX-Talk mailing list > > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > > -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- > Dr. Alexander Holupirek > |-- Room E 221, 0049 7531 88 2188 (phone) 3577 (fax) > |-- Database & Information Systems Group, U Konstanz > `-- https://scikon.uni-konstanz.de/personen/alexander.holupirek/ > > > >
Written for BaseX 7.6. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery