Deadlock on a subset of RestXQs

> We have BaseX 11.9 here with FAIRLOCK = false and PARALLEL=8 (wonder if this can be a hint).

Von: Marco Lettere via BaseX-Talk <basex-talk at mailman.uni-konstanz.de> Gesendet: Montag, 16. März 2026 12:52 An: basex-talk at mailman.uni-konstanz.de <basex-talk at mailman.uni-konstanz.de> Betreff: [basex-talk] Deadlock on a subset of RestXQs Dear all, I'm currently running into an issue that makes my service stop serving a set of RESTXQ calls indefinitely without any clearly visible cause. When this happens an inspection of the threads returns that there are more than 100 threads locked with the stack trace reported in [1]. The architecture comprises a RestXQ service (without any DB) that receives a POST request, let's call it API. In the implementation of the request handling it "xquery:fork-joins" about 12 operations which mainly consist of http requests to another service or reading of local files, in order to cache (or refresh) information that is kept in a BaseX store locally. During the all of the said operation it also performs some write oprations (mainly to log the outcome of the operation) on a BaseX DB which runs on a different BaseX server (say Persistence) process through db:query(). We notice that many calls to the API continue to be served. Mainly calls that remain on the API itself or access other DBs on Persistence. Only all the calls that access the same store or persistence DB somehow stop being served. In the logs we see the request is received but it never either succeeds nor fails. We have BaseX 11.9 here with FAIRLOCK = false and PARALLEL=8 (wonder if this can be a hint). Because we are rather clueless I ask whether anyone has a hint about where we should restrict our investigations? Thank you very much! Marco. [1] "qtp192881625-748" #748 prio=5 os_prio=0 cpu=11.89ms elapsed=240.61s tid=0x00007abe1a7c4330 nid=0x289f6 in Object.wait() [0x00007abe1a4b2000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(java.base at 17.0.12/Native<mailto:java.base at 17.0.12/Native> Method) - waiting on <no object reference available> at java.lang.Object.wait(java.base at 17.0.12/Object.java:338<mailto:java.base at 17.0.12/Object.java:338>) at org.basex.core.locks.NonfairLockQueue.acquire(NonfairLockQueue.java:37) - locked <0x00000000802a93d8> (a org.basex.core.locks.NonfairLockQueue) at org.basex.core.locks.Locking.acquire(Locking.java:110) at org.basex.core.locks.Locking.acquire(Locking.java:89) at org.basex.core.jobs.Job.register(Job.java:48) at org.basex.http.restxq.RestXqResponse.serialize(RestXqResponse.java:81) at org.basex.http.web.WebResponse.create(WebResponse.java:58) at org.basex.http.restxq.RestXqServlet.run(RestXqServlet.java:70) at org.basex.http.BaseXServlet.service(BaseXServlet.java:68) at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:587) at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1410) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:764) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:529) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:598) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1580) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1381) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1553) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1303) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:822) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) at org.eclipse.jetty.server.Server.handle(Server.java:563) at org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:287) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100) at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53) at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421) at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390) at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277) at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:199) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149) at java.lang.Thread.run(java.base at 17.0.12/Thread.java:840<mailto:java.base at 17.0.12/Thread.java:840>)

On 3/16/26 17:06, Christian Grün wrote: > Hi Marco, > > > We have BaseX 11.9 here with FAIRLOCK = false and PARALLEL=8 (wonder > if this can be a hint). > > > I agree, your stack trace clearly indicates an issue with the disabled > FAIRLOCK mode. > > > In our productive code, we always use fair locking, and we are > considering removing this mode in a future version, so it may well be > that this option has been neglected over the years. Would you be able > to use the default mode, or are there specific reasons why you have > resorted to stricter locking? > > Ciao, > Christian > > ------------------------------------------------------------------------ > *Von:* Marco Lettere via BaseX-Talk <basex-talk at mailman.uni-konstanz.de> > *Gesendet:* Montag, 16. März 2026 12:52 > *An:* basex-talk at mailman.uni-konstanz.de > <basex-talk at mailman.uni-konstanz.de> > *Betreff:* [basex-talk] Deadlock on a subset of RestXQs > > Dear all, > > I'm currently running into an issue that makes my service stop serving > a set of RESTXQ calls indefinitely without any clearly visible cause. > > When this happens an inspection of the threads returns that there are > more than 100 threads locked with the stack trace reported in [1]. > > The architecture comprises a RestXQ service (without any DB) that > receives a POST request, let's call it API. In the implementation of > the request handling it "xquery:fork-joins" about 12 operations which > mainly consist of http requests to another service or reading of local > files, in order to cache (or refresh) information that is kept in a > BaseX store locally. During the all of the said operation it also > performs some write oprations (mainly to log the outcome of the > operation) on a BaseX DB which runs on a different BaseX server (say > Persistence) process through db:query(). > > We notice that many calls to the API continue to be served. Mainly > calls that remain on the API itself or access other DBs on Persistence. > > Only all the calls that access the same store or persistence DB > somehow stop being served. In the logs we see the request is received > but it never either succeeds nor fails. > > We have BaseX 11.9 here with FAIRLOCK = false and PARALLEL=8 (wonder > if this can be a hint). > > Because we are rather clueless I ask whether anyone has a hint about > where we should restrict our investigations? > > Thank you very much! > > Marco. > > [1] > > "qtp192881625-748" #748 prio=5 os_prio=0 cpu=11.89ms elapsed=240.61s > tid=0x00007abe1a7c4330 nid=0x289f6 in Object.wait() > [0x00007abe1a4b2000] java.lang.Thread.State: WAITING (on object > monitor) at java.lang.Object.wait(java.base at 17.0.12/Native > <mailto:java.base at 17.0.12/Native> Method) - waiting on <no object > reference available> at > java.lang.Object.wait(java.base at 17.0.12/Object.java:338 > <mailto:java.base at 17.0.12/Object.java:338>) at > org.basex.core.locks.NonfairLockQueue.acquire(NonfairLockQueue.java:37) > - locked <0x00000000802a93d8> (a > org.basex.core.locks.NonfairLockQueue) at > org.basex.core.locks.Locking.acquire(Locking.java:110) at > org.basex.core.locks.Locking.acquire(Locking.java:89) at > org.basex.core.jobs.Job.register(Job.java:48) at > org.basex.http.restxq.RestXqResponse.serialize(RestXqResponse.java:81) > at org.basex.http.web.WebResponse.create(WebResponse.java:58) at > org.basex.http.restxq.RestXqServlet.run(RestXqServlet.java:70) at > org.basex.http.BaseXServlet.service(BaseXServlet.java:68) at > jakarta.servlet.http.HttpServlet.service(HttpServlet.java:587) at > org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1410) > at > org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:764) > at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:529) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131) > at > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:598) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) > at > org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223) > at > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1580) > at > org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221) > at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1381) > at > org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176) > at > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484) > at > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1553) > at > org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174) > at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1303) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) > at > org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:822) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) > at org.eclipse.jetty.server.Server.handle(Server.java:563) at > org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598) > at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753) > at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501) > at > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:287) > at > org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314) > at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100) > at > org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53) > at > org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421) > at > org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390) > at > org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277) > at > org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:199) > at > org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149) > at java.lang.Thread.run(java.base at 17.0.12/Thread.java:840 > <mailto:java.base at 17.0.12/Thread.java:840>) > >

Von: Marco Lettere <m.lettere at gmail.com> Gesendet: Montag, 16. März 2026 18:05 An: Christian Grün <cg at basex.org>; basex-talk at mailman.uni-konstanz.de <basex-talk at mailman.uni-konstanz.de> Betreff: Re: AW: [basex-talk] Deadlock on a subset of RestXQs Hi Christian, thank you very much. We've not changed anything in the configuration of FAIRLOCK actually I've noticed that FAIRLOCK = false is the default that ships with Basex.zip. So you suggest to put the setting to true in production settings? M. On 3/16/26 17:06, Christian Grün wrote: Hi Marco, > We have BaseX 11.9 here with FAIRLOCK = false and PARALLEL=8 (wonder if this can be a hint). I agree, your stack trace clearly indicates an issue with the disabled FAIRLOCK mode. In our productive code, we always use fair locking, and we are considering removing this mode in a future version, so it may well be that this option has been neglected over the years. Would you be able to use the default mode, or are there specific reasons why you have resorted to stricter locking? Ciao, Christian Von: Marco Lettere via BaseX-Talk <basex-talk at mailman.uni-konstanz.de><mailto:basex-talk at mailman.uni-konstanz.de> Gesendet: Montag, 16. März 2026 12:52 An: basex-talk at mailman.uni-konstanz.de<mailto:basex-talk at mailman.uni-konstanz.de> <basex-talk at mailman.uni-konstanz.de><mailto:basex-talk at mailman.uni-konstanz.de> Betreff: [basex-talk] Deadlock on a subset of RestXQs Dear all, I'm currently running into an issue that makes my service stop serving a set of RESTXQ calls indefinitely without any clearly visible cause. When this happens an inspection of the threads returns that there are more than 100 threads locked with the stack trace reported in [1]. The architecture comprises a RestXQ service (without any DB) that receives a POST request, let's call it API. In the implementation of the request handling it "xquery:fork-joins" about 12 operations which mainly consist of http requests to another service or reading of local files, in order to cache (or refresh) information that is kept in a BaseX store locally. During the all of the said operation it also performs some write oprations (mainly to log the outcome of the operation) on a BaseX DB which runs on a different BaseX server (say Persistence) process through db:query(). We notice that many calls to the API continue to be served. Mainly calls that remain on the API itself or access other DBs on Persistence. Only all the calls that access the same store or persistence DB somehow stop being served. In the logs we see the request is received but it never either succeeds nor fails. We have BaseX 11.9 here with FAIRLOCK = false and PARALLEL=8 (wonder if this can be a hint). Because we are rather clueless I ask whether anyone has a hint about where we should restrict our investigations? Thank you very much! Marco. [1] "qtp192881625-748" #748 prio=5 os_prio=0 cpu=11.89ms elapsed=240.61s tid=0x00007abe1a7c4330 nid=0x289f6 in Object.wait() [0x00007abe1a4b2000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(java.base at 17.0.12/Native<mailto:java.base at 17.0.12/Native> Method) - waiting on <no object reference available> at java.lang.Object.wait(java.base at 17.0.12/Object.java:338<mailto:java.base at 17.0.12/Object.java:338>) at org.basex.core.locks.NonfairLockQueue.acquire(NonfairLockQueue.java:37) - locked <0x00000000802a93d8> (a org.basex.core.locks.NonfairLockQueue) at org.basex.core.locks.Locking.acquire(Locking.java:110) at org.basex.core.locks.Locking.acquire(Locking.java:89) at org.basex.core.jobs.Job.register(Job.java:48) at org.basex.http.restxq.RestXqResponse.serialize(RestXqResponse.java:81) at org.basex.http.web.WebResponse.create(WebResponse.java:58) at org.basex.http.restxq.RestXqServlet.run(RestXqServlet.java:70) at org.basex.http.BaseXServlet.service(BaseXServlet.java:68) at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:587) at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1410) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:764) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:529) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:598) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1580) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1381) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1553) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1303) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:822) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) at org.eclipse.jetty.server.Server.handle(Server.java:563) at org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:287) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100) at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53) at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421) at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390) at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277) at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:199) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149) at java.lang.Thread.run(java.base at 17.0.12/Thread.java:840<mailto:java.base at 17.0.12/Thread.java:840>)

On 3/16/26 18:41, Christian Grün wrote: > Forget my gibberish – FAIRLOCK=false is indeed the setting that I > would recommend. > > That's one more reason why we should understand what’s going on. I > assume it’s hard to make it reproducible – if you manage to do so, it > will simplify things a lot – but you could try to observe what happens > when… > > 1. increasing PARALLEL to 50 or 100 (8 is a fairly conservative > default, which should eventually be changed), > 2. disabling xquery:fork-join, > 3. switching to the latest snapshot. > > Next, > > 4. do you have any operations that could potentially lead to conflicts > when being run concurrently (such as writing files to disk)? > 5. If you have a separate storage layer, can you confirm that none of > the running jobs will lock anything? > > – Christian > > ------------------------------------------------------------------------ > *Von:* Marco Lettere <m.lettere at gmail.com> > *Gesendet:* Montag, 16. März 2026 18:05 > *An:* Christian Grün <cg at basex.org>; > basex-talk at mailman.uni-konstanz.de <basex-talk at mailman.uni-konstanz.de> > *Betreff:* Re: AW: [basex-talk] Deadlock on a subset of RestXQs > > Hi Christian, thank you very much. > > We've not changed anything in the configuration of FAIRLOCK actually > I've noticed that FAIRLOCK = false is the default that ships with > Basex.zip. > > So you suggest to put the setting to true in production settings? > > M. > > > On 3/16/26 17:06, Christian Grün wrote: > > Hi Marco, > > > We have BaseX 11.9 here with FAIRLOCK = false and PARALLEL=8 > (wonder if this can be a hint). > > > I agree, your stack trace clearly indicates an issue with the > disabled FAIRLOCK mode. > > > In our productive code, we always use fair locking, and we are > considering removing this mode in a future version, so it may well > be that this option has been neglected over the years. Would you > be able to use the default mode, or are there specific reasons why > you have resorted to stricter locking? > > Ciao, > Christian > > ------------------------------------------------------------------------ > *Von:* Marco Lettere via BaseX-Talk > <basex-talk at mailman.uni-konstanz.de> > <mailto:basex-talk at mailman.uni-konstanz.de> > *Gesendet:* Montag, 16. März 2026 12:52 > *An:* basex-talk at mailman.uni-konstanz.de > <mailto:basex-talk at mailman.uni-konstanz.de> > <basex-talk at mailman.uni-konstanz.de> > <mailto:basex-talk at mailman.uni-konstanz.de> > *Betreff:* [basex-talk] Deadlock on a subset of RestXQs > > Dear all, > > I'm currently running into an issue that makes my service stop > serving a set of RESTXQ calls indefinitely without any clearly > visible cause. > > When this happens an inspection of the threads returns that there > are more than 100 threads locked with the stack trace reported in [1]. > > The architecture comprises a RestXQ service (without any DB) that > receives a POST request, let's call it API. In the implementation > of the request handling it "xquery:fork-joins" about 12 operations > which mainly consist of http requests to another service or > reading of local files, in order to cache (or refresh) information > that is kept in a BaseX store locally. During the all of the said > operation it also performs some write oprations (mainly to log the > outcome of the operation) on a BaseX DB which runs on a different > BaseX server (say Persistence) process through db:query(). > > We notice that many calls to the API continue to be served. Mainly > calls that remain on the API itself or access other DBs on > Persistence. > > Only all the calls that access the same store or persistence DB > somehow stop being served. In the logs we see the request is > received but it never either succeeds nor fails. > > We have BaseX 11.9 here with FAIRLOCK = false and PARALLEL=8 > (wonder if this can be a hint). > > Because we are rather clueless I ask whether anyone has a hint > about where we should restrict our investigations? > > Thank you very much! > > Marco. > > [1] > > "qtp192881625-748" #748 prio=5 os_prio=0 cpu=11.89ms > elapsed=240.61s tid=0x00007abe1a7c4330 nid=0x289f6 in > Object.wait() [0x00007abe1a4b2000] java.lang.Thread.State: > WAITING (on object monitor) at > java.lang.Object.wait(java.base at 17.0.12/Native > <mailto:java.base at 17.0.12/Native> Method) - waiting on <no object > reference available> at > java.lang.Object.wait(java.base at 17.0.12/Object.java:338 > <mailto:java.base at 17.0.12/Object.java:338>) at > org.basex.core.locks.NonfairLockQueue.acquire(NonfairLockQueue.java:37) > - locked <0x00000000802a93d8> (a > org.basex.core.locks.NonfairLockQueue) at > org.basex.core.locks.Locking.acquire(Locking.java:110) at > org.basex.core.locks.Locking.acquire(Locking.java:89) at > org.basex.core.jobs.Job.register(Job.java:48) at > org.basex.http.restxq.RestXqResponse.serialize(RestXqResponse.java:81) > at org.basex.http.web.WebResponse.create(WebResponse.java:58) at > org.basex.http.restxq.RestXqServlet.run(RestXqServlet.java:70) at > org.basex.http.BaseXServlet.service(BaseXServlet.java:68) at > jakarta.servlet.http.HttpServlet.service(HttpServlet.java:587) at > org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1410) > at > org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:764) > at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:529) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131) > at > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:598) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) > at > org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223) > at > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1580) > at > org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221) > at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1381) > at > org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176) > at > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484) > at > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1553) > at > org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174) > at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1303) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) > at > org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:822) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) > at org.eclipse.jetty.server.Server.handle(Server.java:563) at > org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598) > at > org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753) > at > org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501) > at > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:287) > at > org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314) > at > org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100) > at > org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53) > at > org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421) > at > org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390) > at > org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277) > at > org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:199) > at > org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149) > at java.lang.Thread.run(java.base at 17.0.12/Thread.java:840 > <mailto:java.base at 17.0.12/Thread.java:840>) > >

On 3/17/26 09:35, Marco Lettere wrote: > > Hi Christian! > > > I've started with 2 because it looked to me as the main villain in the > plot. The loading of data takes a bit more time but I can concentrate > the activities in a sort of startup phase in order to reduce the > impact on UX. > > We'll monitor it for a few days ... > > > 1) Seems also a good point and we'll think about it as soon as we > perform the next major deployment update because it requires an in > depth rebuild of our container. > > > For 3) I'm a bit scared of moving to official 12.x because of > https://github.com/BaseXdb/basex/issues/2479. I know the latest > snapshot fixes the issue (already experimented in another project) but > our release cycles are aligned with BaseX official releases so we'll > have to wait for the Spring Edition I suppose. > > > 4) The operations that are performed are actually http:send-request()s > to a remote API, store:put()s on the same process (which I understood > do not imply any locking or serialization) and db:query()s (to another > process in a different container). The last one is performing updating > operations most of the time but they always succeed in few ms. > > QUESTION: I don't think db:query() is able to lock the originating > process by performing inspection of the string passed as query before > sending, is it? > > > 5) as said in 4, once the db:query()s make it through to the > persistence process they always succeed without any issue so the > problem seems really to be constrained to the API process. > > > Thanks again for all the support. > > > M. > > > On 3/16/26 18:41, Christian Grün wrote: >> Forget my gibberish – FAIRLOCK=false is indeed the setting that I >> would recommend. >> >> That's one more reason why we should understand what’s going on. I >> assume it’s hard to make it reproducible – if you manage to do so, it >> will simplify things a lot – but you could try to observe what >> happens when… >> >> 1. increasing PARALLEL to 50 or 100 (8 is a fairly conservative >> default, which should eventually be changed), >> 2. disabling xquery:fork-join, >> 3. switching to the latest snapshot. >> >> Next, >> >> 4. do you have any operations that could potentially lead to >> conflicts when being run concurrently (such as writing files to disk)? >> 5. If you have a separate storage layer, can you confirm that none of >> the running jobs will lock anything? >> >> – Christian >> >> ------------------------------------------------------------------------ >> *Von:* Marco Lettere <m.lettere at gmail.com> >> *Gesendet:* Montag, 16. März 2026 18:05 >> *An:* Christian Grün <cg at basex.org>; >> basex-talk at mailman.uni-konstanz.de <basex-talk at mailman.uni-konstanz.de> >> *Betreff:* Re: AW: [basex-talk] Deadlock on a subset of RestXQs >> >> Hi Christian, thank you very much. >> >> We've not changed anything in the configuration of FAIRLOCK actually >> I've noticed that FAIRLOCK = false is the default that ships with >> Basex.zip. >> >> So you suggest to put the setting to true in production settings? >> >> M. >> >> >> On 3/16/26 17:06, Christian Grün wrote: >> >> Hi Marco, >> >> > We have BaseX 11.9 here with FAIRLOCK = false and PARALLEL=8 >> (wonder if this can be a hint). >> >> >> I agree, your stack trace clearly indicates an issue with the >> disabled FAIRLOCK mode. >> >> >> In our productive code, we always use fair locking, and we are >> considering removing this mode in a future version, so it may >> well be that this option has been neglected over the years. Would >> you be able to use the default mode, or are there specific >> reasons why you have resorted to stricter locking? >> >> Ciao, >> Christian >> >> ------------------------------------------------------------------------ >> *Von:* Marco Lettere via BaseX-Talk >> <basex-talk at mailman.uni-konstanz.de> >> <mailto:basex-talk at mailman.uni-konstanz.de> >> *Gesendet:* Montag, 16. März 2026 12:52 >> *An:* basex-talk at mailman.uni-konstanz.de >> <mailto:basex-talk at mailman.uni-konstanz.de> >> <basex-talk at mailman.uni-konstanz.de> >> <mailto:basex-talk at mailman.uni-konstanz.de> >> *Betreff:* [basex-talk] Deadlock on a subset of RestXQs >> >> Dear all, >> >> I'm currently running into an issue that makes my service stop >> serving a set of RESTXQ calls indefinitely without any clearly >> visible cause. >> >> When this happens an inspection of the threads returns that there >> are more than 100 threads locked with the stack trace reported in >> [1]. >> >> The architecture comprises a RestXQ service (without any DB) that >> receives a POST request, let's call it API. In the implementation >> of the request handling it "xquery:fork-joins" about 12 >> operations which mainly consist of http requests to another >> service or reading of local files, in order to cache (or refresh) >> information that is kept in a BaseX store locally. During the all >> of the said operation it also performs some write oprations >> (mainly to log the outcome of the operation) on a BaseX DB which >> runs on a different BaseX server (say Persistence) process >> through db:query(). >> >> We notice that many calls to the API continue to be >> served. Mainly calls that remain on the API itself or access >> other DBs on Persistence. >> >> Only all the calls that access the same store or persistence DB >> somehow stop being served. In the logs we see the request is >> received but it never either succeeds nor fails. >> >> We have BaseX 11.9 here with FAIRLOCK = false and PARALLEL=8 >> (wonder if this can be a hint). >> >> Because we are rather clueless I ask whether anyone has a hint >> about where we should restrict our investigations? >> >> Thank you very much! >> >> Marco. >> >> [1] >> >> "qtp192881625-748" #748 prio=5 os_prio=0 cpu=11.89ms >> elapsed=240.61s tid=0x00007abe1a7c4330 nid=0x289f6 in >> Object.wait() [0x00007abe1a4b2000] java.lang.Thread.State: >> WAITING (on object monitor) at >> java.lang.Object.wait(java.base at 17.0.12/Native >> <mailto:java.base at 17.0.12/Native> Method) - waiting on <no object >> reference available> at >> java.lang.Object.wait(java.base at 17.0.12/Object.java:338 >> <mailto:java.base at 17.0.12/Object.java:338>) at >> org.basex.core.locks.NonfairLockQueue.acquire(NonfairLockQueue.java:37) >> - locked <0x00000000802a93d8> (a >> org.basex.core.locks.NonfairLockQueue) at >> org.basex.core.locks.Locking.acquire(Locking.java:110) at >> org.basex.core.locks.Locking.acquire(Locking.java:89) at >> org.basex.core.jobs.Job.register(Job.java:48) at >> org.basex.http.restxq.RestXqResponse.serialize(RestXqResponse.java:81) >> at org.basex.http.web.WebResponse.create(WebResponse.java:58) at >> org.basex.http.restxq.RestXqServlet.run(RestXqServlet.java:70) at >> org.basex.http.BaseXServlet.service(BaseXServlet.java:68) at >> jakarta.servlet.http.HttpServlet.service(HttpServlet.java:587) at >> org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1410) >> at >> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:764) >> at >> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:529) >> at >> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131) >> at >> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:598) >> at >> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) >> at >> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223) >> at >> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1580) >> at >> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221) >> at >> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1381) >> at >> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176) >> at >> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484) >> at >> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1553) >> at >> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174) >> at >> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1303) >> at >> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129) >> at >> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) >> at >> org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:822) >> at >> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) >> at org.eclipse.jetty.server.Server.handle(Server.java:563) at >> org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598) >> at >> org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753) >> at >> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501) >> at >> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:287) >> at >> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314) >> at >> org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100) >> at >> org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53) >> at >> org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421) >> at >> org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390) >> at >> org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277) >> at >> org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:199) >> at >> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411) >> at >> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969) >> at >> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194) >> at >> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149) >> at java.lang.Thread.run(java.base at 17.0.12/Thread.java:840 >> <mailto:java.base at 17.0.12/Thread.java:840>) >> >>