Apparent "soft" resource leak in job:execute()
Hello,
I have a use case in which queries that used to be run against XML files (in Saxon) need to be run against nodes fetched from the database instead. (This means the query itself does no database access, it simply needs to be evaluated with a database document as its context node.) Since there's no requirement for asynchronous execution, I tried xquery:eval() first; that works, but of course it enforces a global lock (which is wasteful in this instance since the queries don't *happen* to access any database). job:execute() avoids the locking issue, but in turn adds a new problem: jobs seem to "leak" resources through the result cache. I noticed this after switching over to job:execute; some of these queries have fairly large results, and repeated test runs manage to hit the JVM's default heap size limit. Reducing CACHETIMEOUT appears to confirm that this is job results (or some kind of associated resource) sitting in the cache, since running just one of the query jobs allocates extra storage on the heap, which is released roughly CACHETIMEOUT seconds after the job finishes. It appears that setting CACHETIMEOUT to 30, starting the server, and then executing a query like ` job:execute('concat((1 to 10000000) ! "spam")')` is enough to see the issue (while watch(1)ing something to the effect of `jcmd
org.basex.BaseXServer GC.heap_info`) - it'll allocate about 40M of spam on the heap that goes away after about 30 seconds.
I'm not sure why this is happening; job:execute()'s implementation (https://github.com/BaseXdb/basex/blob/72d4c437605a48fb551bd183760d4ca334cc83f0/basex-core/src/main/java/org/basex/query/func/job/JobExecute.java) does hardcode the CACHE option so (if I read this correctly) that it can retrieve the result, but it also removes the result from the cache before returning. I would have thought that should be freeing the bulk of the resources (certainly the 40M of spam!) - maybe there's another reference to it somewhere?
Suggestions welcome,
Peter
Hi Peter,
Thanks for the excellent analysis, which made it easy to find the culprit. You were right: the result was removed from the cache, but the timer task that discards it after CACHETIMEOUT kept a reference to the job, and thus to its result.
Something that is now resolved in the latest snapshot [1,2]. With the official 13.0, a workaround is a lower CACHETIMEOUT value.
Best, Christian
[1] https://github.com/BaseXdb/basex/commit/53aaa396e7c9e5d4cbf0b5f1a007a953f795bd5d [2] https://files.basex.org/releases/latest/
Quoted text · Peter Piwowarski · 37 lines
Peter Piwowarski via BaseX-Talk <basex-talk at mailman.uni-konstanz.de> schrieb am Dienstag, 17. November 2026 um 15:46: > Hello, > > I have a use case in which queries that used to be run against XML files > (in Saxon) need to be run against nodes fetched from the database > instead. (This means the query itself does no database access, it simply > needs to be evaluated with a database document as its context node.) > Since there's no requirement for asynchronous execution, I tried > xquery:eval() first; that works, but of course it enforces a global lock > (which is wasteful in this instance since the queries don't *happen* to > access any database). job:execute() avoids the locking issue, but in > turn adds a new problem: jobs seem to "leak" resources through the > result cache. I noticed this after switching over to job:execute; some > of these queries have fairly large results, and repeated test runs > manage to hit the JVM's default heap size limit. Reducing CACHETIMEOUT > appears to confirm that this is job results (or some kind of associated > resource) sitting in the cache, since running just one of the query jobs > allocates extra storage on the heap, which is released roughly > CACHETIMEOUT seconds after the job finishes. It appears that setting > CACHETIMEOUT to 30, starting the server, and then executing a query like > ` job:execute('concat((1 to 10000000) ! "spam")')` is enough to see the > issue (while watch(1)ing something to the effect of `jcmd > org.basex.BaseXServer GC.heap_info`) - it'll allocate about 40M of spam > on the heap that goes away after about 30 seconds. > > I'm not sure why this is happening; job:execute()'s implementation > (https://github.com/BaseXdb/basex/blob/72d4c437605a48fb551bd183760d4ca334cc83f0/basex-core/src/main/java/org/basex/query/func/job/JobExecute.java) > does hardcode the CACHE option so (if I read this correctly) that it can > retrieve the result, but it also removes the result from the cache > before returning. I would have thought that should be freeing the bulk > of the resources (certainly the 40M of spam!) - maybe there's another > reference to it somewhere? > > Suggestions welcome, > > Peter >
Confirmed: 53aaa396 resolves the issue for me. Thanks for the quick response!
Quoted text · Christian Grün · 12 lines
Christian Grün wrote: > Hi Peter, > > Thanks for the excellent analysis, which made it easy to find the culprit. You were right: the result was removed from the cache, but the timer task that discards it after `CACHETIMEOUT` kept a reference to the job, and thus to its result. > > Something that is now resolved in the latest snapshot [1,2]. With the official 13.0, a workaround is a lower `CACHETIMEOUT` value. > > Best, > Christian > > [1] https://github.com/BaseXdb/basex/commit/53aaa396e7c9e5d4cbf0b5f1a007a953f795bd5d > [2] https://files.basex.org/releases/latest/
Written for BaseX 13.0, the current version.