SIGPIPE error

Wednesday, 5 August 2015 · 10 messages · last reply 6 Aug 2015
M
mar5 Aug 2015 10:08 #
Hi, I am having difficulties with populating BASEX database. I have plenty of XML files (~ half a million, with various sizes ranging from several kilobytes up to hundred of kilobytes). I use BASEX Java API and finally I call for each file org.basex.core.cmd.Add. I am trying to import them into BASEX database, in fact there are 22 types of files (22 XSD definitions) the files conform to, so I have 22 different databases in a single BASEX server. I have plenty of RAM and CPU power and I monitor the process (both -- the BASEX server and my client program) from within JVisualVM, the JVM reaches the CPU boundaries, but RAM is never exhausted. Before importing, I need to enhance the XML data with some additional information taken from SQL database. I have written a Groovy multithreaded program that uses BASEX Java API with heavy use of GPars library. Simply put, the program: 1. has several producer threads -- each producer reads given portion of the database and provides those additional information 2. has several consumer threads -- each consumer takes the original files, wraps it with additional information and finally calls org.basex.core.cmd.Add command. Various testing with less data (upto ~ several thousands of files) provides good results -- no loss of data, BASEX server and my client program behaves as it should. Unfortunately when trying to import all of the files, the program starts fine, but when it gets "warm" I got SIGPIPE errors in log from time to time (as I said, there is plenty of RAM and CPU available) (see attachment please). Comments to picture: 1. I am adding document with ID ISPOP_166007 -- this ID is indeed missing in the final database 2. just simple call to Add: Closure add = { session -> def cmd = new org.basex.core.cmd.Add(dsn, enhancedXml) session.execute(cmd) } 3. I am reusing the session, the session is bound to current thread and never gets closed until the thread (consumer) finishes There is nothing wrong in BASEX server log, other documents are added just fine, there is no trace about document ISPOP_166007. Just for reference the complete stack trace follows: - - - -
ERROR basex.support.AddResourcesSupport - unable to consume ISPOP_166007
java.net.SocketException: Roura přerušena (SIGPIPE)
        at java.net.SocketOutputStream.socketWrite0(Native Method) ~[na:1.8.0_45]
        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109) ~[na:1.8.0_45]
        at java.net.SocketOutputStream.write(SocketOutputStream.java:153) ~[na:1.8.0_45]
        at org.basex.io.out.BufferOutput.flush(BufferOutput.java:60) ~[basex-8.2.jar!/:8.2]
        at org.basex.io.out.BufferOutput.write(BufferOutput.java:54) ~[basex-8.2.jar!/:8.2]
        at org.basex.io.out.PrintOutput.write(PrintOutput.java:66) ~[basex-8.2.jar!/:8.2]
        at java.io.OutputStream.write(OutputStream.java:116) ~[na:1.8.0_45]
        at java.io.OutputStream.write(OutputStream.java:75) ~[na:1.8.0_45]
        at org.basex.api.client.ClientSession.send(ClientSession.java:238) ~[basex-8.2.jar!/:8.2]
        at org.basex.api.client.ClientSession.execute(ClientSession.java:160) ~[basex-8.2.jar!/:8.2]
        at org.basex.api.client.ClientSession.execute(ClientSession.java:167) ~[basex-8.2.jar!/:8.2]
        at org.basex.api.client.Session.execute(Session.java:36) ~[basex-8.2.jar!/:8.2]
        at org.basex.api.client.Session$execute.call(Unknown Source) ~[na:na]
        at basex.support.AddResourcesSupport$_consume_closure9$_closure17.doCall(AddResourcesSupport.groovy:255) ~[basex-1.0.jar!/:na]
        at sun.reflect.GeneratedMethodAccessor368.invoke(Unknown Source) ~[na:na]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_45]
        at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_45]
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93) [groovy-2.4.4.jar!/:2.4.4]
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325) [groovy-2.4.4.jar!/:2.4.4]
        at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294) [groovy-2.4.4.jar!/:2.4.4]
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1019) [groovy-2.4.4.jar!/:2.4.4]
        at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42) [groovy-2.4.4.jar!/:2.4.4]
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125) [groovy-2.4.4.jar!/:2.4.4]
        at basex.BasexSessionRegistry.withThreadBoundSession(BasexSessionRegistry.groovy:79) ~[basex-1.0.jar!/:na]
        at basex.BasexSessionRegistry$withThreadBoundSession$0.call(Unknown Source) ~[na:na]
        at basex.support.AddResourcesSupport$_consume_closure9.doCall(AddResourcesSupport.groovy:257) ~[basex-1.0.jar!/:na]
        at sun.reflect.GeneratedMethodAccessor327.invoke(Unknown Source) ~[na:na]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_45]
        at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_45]
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93) [groovy-2.4.4.jar!/:2.4.4]
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325) [groovy-2.4.4.jar!/:2.4.4]
        at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294) [groovy-2.4.4.jar!/:2.4.4]
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1019) [groovy-2.4.4.jar!/:2.4.4]
        at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42) [groovy-2.4.4.jar!/:2.4.4]
        at org.codehaus.groovy.runtime.callsite.BooleanReturningMethodInvoker.invoke(BooleanReturningMethodInvoker.java:51) ~[groovy-2.4.4.jar!/:2.4.4]
        at org.codehaus.groovy.runtime.callsite.BooleanClosureWrapper.call(BooleanClosureWrapper.java:53) ~[groovy-2.4.4.jar!/:2.4.4]
        at org.codehaus.groovy.runtime.DefaultGroovyMethods.find(DefaultGroovyMethods.java:3908) ~[groovy-2.4.4.jar!/:2.4.4]
        at org.codehaus.groovy.runtime.dgm$191.invoke(Unknown Source) ~[groovy-2.4.4.jar!/:2.4.4]
        at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:274) ~[groovy-2.4.4.jar!/:2.4.4]
        at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56) ~[groovy-2.4.4.jar!/:2.4.4]
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125) [groovy-2.4.4.jar!/:2.4.4]
        at basex.support.AddResourcesSupport.consume(AddResourcesSupport.groovy:251) ~[basex-1.0.jar!/:na]
- - - - Best Regards, Martin
CG
Christian Grün5 Aug 2015 10:29 #
Hi Martin, I must confess I didn't check all the details in your mail, and I haven't stumbled across something like SIGPIPE errors before, but I would be interested to hear if the problem als occurs... a) with a single thread, or b) if you don't reuse existing sessions? Thanks in advance, Christian
Quoted text · Martin · 111 lines
On Wed, Aug 5, 2015 at 10:08 AM, Martin <mar at centrum.cz> wrote: > Hi, > > I am having difficulties with populating BASEX database. I have plenty of > XML files (~ half a million, with various sizes ranging from several > kilobytes up to hundred of kilobytes). > > I use BASEX Java API and finally I call for each file > org.basex.core.cmd.Add. > > I am trying to import them into BASEX database, in fact there are 22 types > of files (22 XSD definitions) the files conform to, so I have 22 different > databases in a single BASEX server. > > I have plenty of RAM and CPU power and I monitor the process (both -- the > BASEX server and my client program) from within JVisualVM, the JVM reaches > the CPU boundaries, but RAM is never exhausted. > > Before importing, I need to enhance the XML data with some additional > information taken from SQL database. > > I have written a Groovy multithreaded program that uses BASEX Java API with > heavy use of GPars library. Simply put, the program: > > 1. has several producer threads -- each producer reads given portion of the > database and provides those additional information > > 2. has several consumer threads -- each consumer takes the original files, > wraps it with additional information and finally calls > org.basex.core.cmd.Add command. > > Various testing with less data (upto ~ several thousands of files) provides > good results -- no loss of data, BASEX server and my client program behaves > as it should. > > Unfortunately when trying to import all of the files, the program starts > fine, but when it gets "warm" I got SIGPIPE errors in log from time to time > (as I said, there is plenty of RAM and CPU available) (see attachment > please). > > Comments to picture: > > 1. I am adding document with ID ISPOP_166007 -- this ID is indeed missing > in the final database > > 2. just simple call to Add: > > Closure add = { session -> > def cmd = new org.basex.core.cmd.Add(dsn, enhancedXml) > session.execute(cmd) > } > > 3. I am reusing the session, the session is bound to current thread and > never gets closed until the thread (consumer) finishes > > There is nothing wrong in BASEX server log, other documents are added just > fine, there is no trace about document ISPOP_166007. > > Just for reference the complete stack trace follows: > > - - - - > > ERROR basex.support.AddResourcesSupport - unable to consume ISPOP_166007 > java.net.SocketException: Roura přerušena (SIGPIPE) > at java.net.SocketOutputStream.socketWrite0(Native Method) ~[na:1.8.0_45] > at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109) ~[na:1.8.0_45] > at java.net.SocketOutputStream.write(SocketOutputStream.java:153) ~[na:1.8.0_45] > at org.basex.io.out.BufferOutput.flush(BufferOutput.java:60) ~[basex-8.2.jar!/:8.2] > at org.basex.io.out.BufferOutput.write(BufferOutput.java:54) ~[basex-8.2.jar!/:8.2] > at org.basex.io.out.PrintOutput.write(PrintOutput.java:66) ~[basex-8.2.jar!/:8.2] > at java.io.OutputStream.write(OutputStream.java:116) ~[na:1.8.0_45] > at java.io.OutputStream.write(OutputStream.java:75) ~[na:1.8.0_45] > at org.basex.api.client.ClientSession.send(ClientSession.java:238) ~[basex-8.2.jar!/:8.2] > at org.basex.api.client.ClientSession.execute(ClientSession.java:160) ~[basex-8.2.jar!/:8.2] > at org.basex.api.client.ClientSession.execute(ClientSession.java:167) ~[basex-8.2.jar!/:8.2] > at org.basex.api.client.Session.execute(Session.java:36) ~[basex-8.2.jar!/:8.2] > at org.basex.api.client.Session$execute.call(Unknown Source) ~[na:na] > at basex.support.AddResourcesSupport$_consume_closure9$_closure17.doCall(AddResourcesSupport.groovy:255) ~[basex-1.0.jar!/:na] > at sun.reflect.GeneratedMethodAccessor368.invoke(Unknown Source) ~[na:na] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_45] > at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_45] > at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93) [groovy-2.4.4.jar!/:2.4.4] > at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325) [groovy-2.4.4.jar!/:2.4.4] > at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294) [groovy-2.4.4.jar!/:2.4.4] > at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1019) [groovy-2.4.4.jar!/:2.4.4] > at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42) [groovy-2.4.4.jar!/:2.4.4] > at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125) [groovy-2.4.4.jar!/:2.4.4] > at basex.BasexSessionRegistry.withThreadBoundSession(BasexSessionRegistry.groovy:79) ~[basex-1.0.jar!/:na] > at basex.BasexSessionRegistry$withThreadBoundSession$0.call(Unknown Source) ~[na:na] > at basex.support.AddResourcesSupport$_consume_closure9.doCall(AddResourcesSupport.groovy:257) ~[basex-1.0.jar!/:na] > at sun.reflect.GeneratedMethodAccessor327.invoke(Unknown Source) ~[na:na] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_45] > at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_45] > at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93) [groovy-2.4.4.jar!/:2.4.4] > at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325) [groovy-2.4.4.jar!/:2.4.4] > at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294) [groovy-2.4.4.jar!/:2.4.4] > at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1019) [groovy-2.4.4.jar!/:2.4.4] > at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42) [groovy-2.4.4.jar!/:2.4.4] > at org.codehaus.groovy.runtime.callsite.BooleanReturningMethodInvoker.invoke(BooleanReturningMethodInvoker.java:51) ~[groovy-2.4.4.jar!/:2.4.4] > at org.codehaus.groovy.runtime.callsite.BooleanClosureWrapper.call(BooleanClosureWrapper.java:53) ~[groovy-2.4.4.jar!/:2.4.4] > at org.codehaus.groovy.runtime.DefaultGroovyMethods.find(DefaultGroovyMethods.java:3908) ~[groovy-2.4.4.jar!/:2.4.4] > at org.codehaus.groovy.runtime.dgm$191.invoke(Unknown Source) ~[groovy-2.4.4.jar!/:2.4.4] > at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:274) ~[groovy-2.4.4.jar!/:2.4.4] > at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56) ~[groovy-2.4.4.jar!/:2.4.4] > at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125) [groovy-2.4.4.jar!/:2.4.4] > at basex.support.AddResourcesSupport.consume(AddResourcesSupport.groovy:251) ~[basex-1.0.jar!/:na] > > - - - - > > Best Regards, > Martin
M
mar5 Aug 2015 14:06 #
Hi, I will indeed try to investigate it a little more. Reason for being multithreaded producer-consumer si simple -- single thread solution is _very_ slow, even now it takes hours to complete. Now I am not reusing sessions -- each org.basex.core.cmd.Add command opens and closes its own session, still running multithreaded. So far it seems to be behaving better (still running). According to JVM monitoring the thread count of BASEX server is steady ~ 60 live threads. In previous scenario (multithreaded + reusing thread-bound sessions) it peaked to ~ 600 or more live threads -- so reusing sessions is probably a bad idea. Is it a bug in BASEX (not releasing resources properly?) or is reusing sessions (saved in thread local context) a bad idea and should be discouraged? m.
Quoted text · Christian Grün · 126 lines
On Wed, Aug 05, 2015 at 10:29:34AM +0200, Christian Grün wrote: > Hi Martin, > > I must confess I didn't check all the details in your mail, and I > haven't stumbled across something like SIGPIPE errors before, but I > would be interested to hear if the problem als occurs... > > a) with a single thread, or > b) if you don't reuse existing sessions? > > Thanks in advance, > Christian > > > On Wed, Aug 5, 2015 at 10:08 AM, Martin <mar at centrum.cz> wrote: > > Hi, > > > > I am having difficulties with populating BASEX database. I have plenty of > > XML files (~ half a million, with various sizes ranging from several > > kilobytes up to hundred of kilobytes). > > > > I use BASEX Java API and finally I call for each file > > org.basex.core.cmd.Add. > > > > I am trying to import them into BASEX database, in fact there are 22 types > > of files (22 XSD definitions) the files conform to, so I have 22 different > > databases in a single BASEX server. > > > > I have plenty of RAM and CPU power and I monitor the process (both -- the > > BASEX server and my client program) from within JVisualVM, the JVM reaches > > the CPU boundaries, but RAM is never exhausted. > > > > Before importing, I need to enhance the XML data with some additional > > information taken from SQL database. > > > > I have written a Groovy multithreaded program that uses BASEX Java API with > > heavy use of GPars library. Simply put, the program: > > > > 1. has several producer threads -- each producer reads given portion of the > > database and provides those additional information > > > > 2. has several consumer threads -- each consumer takes the original files, > > wraps it with additional information and finally calls > > org.basex.core.cmd.Add command. > > > > Various testing with less data (upto ~ several thousands of files) provides > > good results -- no loss of data, BASEX server and my client program behaves > > as it should. > > > > Unfortunately when trying to import all of the files, the program starts > > fine, but when it gets "warm" I got SIGPIPE errors in log from time to time > > (as I said, there is plenty of RAM and CPU available) (see attachment > > please). > > > > Comments to picture: > > > > 1. I am adding document with ID ISPOP_166007 -- this ID is indeed missing > > in the final database > > > > 2. just simple call to Add: > > > > Closure add = { session -> > > def cmd = new org.basex.core.cmd.Add(dsn, enhancedXml) > > session.execute(cmd) > > } > > > > 3. I am reusing the session, the session is bound to current thread and > > never gets closed until the thread (consumer) finishes > > > > There is nothing wrong in BASEX server log, other documents are added just > > fine, there is no trace about document ISPOP_166007. > > > > Just for reference the complete stack trace follows: > > > > - - - - > > > > ERROR basex.support.AddResourcesSupport - unable to consume ISPOP_166007 > > java.net.SocketException: Roura přerušena (SIGPIPE) > > at java.net.SocketOutputStream.socketWrite0(Native Method) ~[na:1.8.0_45] > > at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109) ~[na:1.8.0_45] > > at java.net.SocketOutputStream.write(SocketOutputStream.java:153) ~[na:1.8.0_45] > > at org.basex.io.out.BufferOutput.flush(BufferOutput.java:60) ~[basex-8.2.jar!/:8.2] > > at org.basex.io.out.BufferOutput.write(BufferOutput.java:54) ~[basex-8.2.jar!/:8.2] > > at org.basex.io.out.PrintOutput.write(PrintOutput.java:66) ~[basex-8.2.jar!/:8.2] > > at java.io.OutputStream.write(OutputStream.java:116) ~[na:1.8.0_45] > > at java.io.OutputStream.write(OutputStream.java:75) ~[na:1.8.0_45] > > at org.basex.api.client.ClientSession.send(ClientSession.java:238) ~[basex-8.2.jar!/:8.2] > > at org.basex.api.client.ClientSession.execute(ClientSession.java:160) ~[basex-8.2.jar!/:8.2] > > at org.basex.api.client.ClientSession.execute(ClientSession.java:167) ~[basex-8.2.jar!/:8.2] > > at org.basex.api.client.Session.execute(Session.java:36) ~[basex-8.2.jar!/:8.2] > > at org.basex.api.client.Session$execute.call(Unknown Source) ~[na:na] > > at basex.support.AddResourcesSupport$_consume_closure9$_closure17.doCall(AddResourcesSupport.groovy:255) ~[basex-1.0.jar!/:na] > > at sun.reflect.GeneratedMethodAccessor368.invoke(Unknown Source) ~[na:na] > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_45] > > at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_45] > > at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93) [groovy-2.4.4.jar!/:2.4.4] > > at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325) [groovy-2.4.4.jar!/:2.4.4] > > at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294) [groovy-2.4.4.jar!/:2.4.4] > > at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1019) [groovy-2.4.4.jar!/:2.4.4] > > at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42) [groovy-2.4.4.jar!/:2.4.4] > > at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125) [groovy-2.4.4.jar!/:2.4.4] > > at basex.BasexSessionRegistry.withThreadBoundSession(BasexSessionRegistry.groovy:79) ~[basex-1.0.jar!/:na] > > at basex.BasexSessionRegistry$withThreadBoundSession$0.call(Unknown Source) ~[na:na] > > at basex.support.AddResourcesSupport$_consume_closure9.doCall(AddResourcesSupport.groovy:257) ~[basex-1.0.jar!/:na] > > at sun.reflect.GeneratedMethodAccessor327.invoke(Unknown Source) ~[na:na] > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_45] > > at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_45] > > at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93) [groovy-2.4.4.jar!/:2.4.4] > > at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325) [groovy-2.4.4.jar!/:2.4.4] > > at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294) [groovy-2.4.4.jar!/:2.4.4] > > at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1019) [groovy-2.4.4.jar!/:2.4.4] > > at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42) [groovy-2.4.4.jar!/:2.4.4] > > at org.codehaus.groovy.runtime.callsite.BooleanReturningMethodInvoker.invoke(BooleanReturningMethodInvoker.java:51) ~[groovy-2.4.4.jar!/:2.4.4] > > at org.codehaus.groovy.runtime.callsite.BooleanClosureWrapper.call(BooleanClosureWrapper.java:53) ~[groovy-2.4.4.jar!/:2.4.4] > > at org.codehaus.groovy.runtime.DefaultGroovyMethods.find(DefaultGroovyMethods.java:3908) ~[groovy-2.4.4.jar!/:2.4.4] > > at org.codehaus.groovy.runtime.dgm$191.invoke(Unknown Source) ~[groovy-2.4.4.jar!/:2.4.4] > > at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:274) ~[groovy-2.4.4.jar!/:2.4.4] > > at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56) ~[groovy-2.4.4.jar!/:2.4.4] > > at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125) [groovy-2.4.4.jar!/:2.4.4] > > at basex.support.AddResourcesSupport.consume(AddResourcesSupport.groovy:251) ~[basex-1.0.jar!/:na] > > > > - - - - > > > > Best Regards, > > Martin >
CG
Christian Grün5 Aug 2015 14:10 #
Quoted text · mar · 2 lines
> Reason for being multithreaded producer-consumer si simple -- single thread > solution is _very_ slow, even now it takes hours to complete.
I see. So just in case it would be interested if the problems also occur with a single thread.
Quoted text · mar · 3 lines
> Is it a bug in BASEX (not releasing resources properly?) or is reusing > sessions (saved in thread local context) a bad idea and should be > discouraged?
I'm not aware of a particular bug in BaseX related to session management, but usually there is no need to reuse existing sessions, as the creation of sessions is pretty light-weight.
SP
Strasser Pabloin reply to mar, 5 Aug 2015, 14:065 Aug 2015 17:55 #
Hi, I'm not really familiar with the formulation of the xquery standard. Should code of this type depending on the short-circuit evaluation of the and operator evaluate correctly?
                                                 declare variable
$version:=1;
                                                 declare variable
$outputdb:='test';
                                                 declare variable
$outputpath:='test';
db:exists($outputdb,$outputpath) and
doc(concat($outputdb,'/',$outputpath))/node()/@version = $version
If the file test/test doesn't exist and receive an error even if the evaluation of the condition is possible. If this is normal does it exist a way to write this type of code without use of if? Pablo Strasser
CG
Christian Grün5 Aug 2015 18:05 #
Hi Pablo,
Quoted text · Strasser Pablo · 2 lines
> Should code of this type depending on the short-circuit evaluation of the > and operator evaluate correctly?
Yes, it does. Here is a mini example (it would raise an error otherwise): 1 or error() Hope this helps, Christian
Quoted text · Strasser Pablo · 16 lines
> > declare variable > $version:=1; > declare variable > $outputdb:='test'; > declare variable > $outputpath:='test'; > db:exists($outputdb,$outputpath) and > doc(concat($outputdb,'/',$outputpath))/node()/@version = $version > > If the file test/test doesn't exist and receive an error even if the > evaluation of the condition is possible. > If this is normal does it exist a way to write this type of code without use > of if? > > Pablo Strasser
SP
Strasser Pablo5 Aug 2015 19:59 #
Hi Christian, Thanks for the answer. It seem that if there is a call to the doc function it doesn't short-circuit. Your exemple work, however this one does not: 1 or doc('non_existent') Which return [FODC0002] Resource '/home/pablo/nonex' does not exist. With respect of your answer I believe this behaviour is a bug. Pablo
Quoted text · Christian Grün · 12 lines
On 05/08/15 18:05, Christian Grün wrote: > Hi Pablo, > >> Should code of this type depending on the short-circuit evaluation of the >> and operator evaluate correctly? > Yes, it does. Here is a mini example (it would raise an error otherwise): > > 1 or error() > > Hope this helps, > Christian >
SP
Strasser Pablo5 Aug 2015 20:15 #
Quoted text · Strasser Pablo · 11 lines
On 05/08/15 19:59, Strasser Pablo wrote: > Hi Christian, > > Thanks for the answer. > > It seem that if there is a call to the doc function it doesn't > short-circuit. > > Your exemple work, however this one does not: > 1 or doc('non_existent') >
However this one work:
declare %basex:lazy variable $doc:=doc('non_existent');
1 or $doc
Pablo
M
marin reply to mar, 5 Aug 2015, 10:085 Aug 2015 23:48 #
Reusing sessions within the same thread resulted in many live threads at BASEX server -- and probably in SIGPIPE errors after a while. Using new session for every Add command solved the problem. Opening and closing a session (as measured) is acceptable performance penalty. Regards, Martin
Quoted text · Martin · 111 lines
On Wed, Aug 05, 2015 at 10:08:05AM +0200, Martin wrote: > Hi, > > I am having difficulties with populating BASEX database. I have plenty of > XML files (~ half a million, with various sizes ranging from several > kilobytes up to hundred of kilobytes). > > I use BASEX Java API and finally I call for each file > org.basex.core.cmd.Add. > > I am trying to import them into BASEX database, in fact there are 22 types > of files (22 XSD definitions) the files conform to, so I have 22 different > databases in a single BASEX server. > > I have plenty of RAM and CPU power and I monitor the process (both -- the > BASEX server and my client program) from within JVisualVM, the JVM reaches > the CPU boundaries, but RAM is never exhausted. > > Before importing, I need to enhance the XML data with some additional > information taken from SQL database. > > I have written a Groovy multithreaded program that uses BASEX Java API with > heavy use of GPars library. Simply put, the program: > > 1. has several producer threads -- each producer reads given portion of the > database and provides those additional information > > 2. has several consumer threads -- each consumer takes the original files, > wraps it with additional information and finally calls > org.basex.core.cmd.Add command. > > Various testing with less data (upto ~ several thousands of files) provides > good results -- no loss of data, BASEX server and my client program behaves > as it should. > > Unfortunately when trying to import all of the files, the program starts > fine, but when it gets "warm" I got SIGPIPE errors in log from time to time > (as I said, there is plenty of RAM and CPU available) (see attachment > please). > > Comments to picture: > > 1. I am adding document with ID ISPOP_166007 -- this ID is indeed missing > in the final database > > 2. just simple call to Add: > > Closure add = { session -> > def cmd = new org.basex.core.cmd.Add(dsn, enhancedXml) > session.execute(cmd) > } > > 3. I am reusing the session, the session is bound to current thread and > never gets closed until the thread (consumer) finishes > > There is nothing wrong in BASEX server log, other documents are added just > fine, there is no trace about document ISPOP_166007. > > Just for reference the complete stack trace follows: > > - - - - > > ERROR basex.support.AddResourcesSupport - unable to consume ISPOP_166007 > java.net.SocketException: Roura přerušena (SIGPIPE) > at java.net.SocketOutputStream.socketWrite0(Native Method) ~[na:1.8.0_45] > at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109) ~[na:1.8.0_45] > at java.net.SocketOutputStream.write(SocketOutputStream.java:153) ~[na:1.8.0_45] > at org.basex.io.out.BufferOutput.flush(BufferOutput.java:60) ~[basex-8.2.jar!/:8.2] > at org.basex.io.out.BufferOutput.write(BufferOutput.java:54) ~[basex-8.2.jar!/:8.2] > at org.basex.io.out.PrintOutput.write(PrintOutput.java:66) ~[basex-8.2.jar!/:8.2] > at java.io.OutputStream.write(OutputStream.java:116) ~[na:1.8.0_45] > at java.io.OutputStream.write(OutputStream.java:75) ~[na:1.8.0_45] > at org.basex.api.client.ClientSession.send(ClientSession.java:238) ~[basex-8.2.jar!/:8.2] > at org.basex.api.client.ClientSession.execute(ClientSession.java:160) ~[basex-8.2.jar!/:8.2] > at org.basex.api.client.ClientSession.execute(ClientSession.java:167) ~[basex-8.2.jar!/:8.2] > at org.basex.api.client.Session.execute(Session.java:36) ~[basex-8.2.jar!/:8.2] > at org.basex.api.client.Session$execute.call(Unknown Source) ~[na:na] > at basex.support.AddResourcesSupport$_consume_closure9$_closure17.doCall(AddResourcesSupport.groovy:255) ~[basex-1.0.jar!/:na] > at sun.reflect.GeneratedMethodAccessor368.invoke(Unknown Source) ~[na:na] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_45] > at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_45] > at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93) [groovy-2.4.4.jar!/:2.4.4] > at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325) [groovy-2.4.4.jar!/:2.4.4] > at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294) [groovy-2.4.4.jar!/:2.4.4] > at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1019) [groovy-2.4.4.jar!/:2.4.4] > at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42) [groovy-2.4.4.jar!/:2.4.4] > at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125) [groovy-2.4.4.jar!/:2.4.4] > at basex.BasexSessionRegistry.withThreadBoundSession(BasexSessionRegistry.groovy:79) ~[basex-1.0.jar!/:na] > at basex.BasexSessionRegistry$withThreadBoundSession$0.call(Unknown Source) ~[na:na] > at basex.support.AddResourcesSupport$_consume_closure9.doCall(AddResourcesSupport.groovy:257) ~[basex-1.0.jar!/:na] > at sun.reflect.GeneratedMethodAccessor327.invoke(Unknown Source) ~[na:na] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_45] > at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_45] > at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93) [groovy-2.4.4.jar!/:2.4.4] > at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325) [groovy-2.4.4.jar!/:2.4.4] > at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294) [groovy-2.4.4.jar!/:2.4.4] > at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1019) [groovy-2.4.4.jar!/:2.4.4] > at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42) [groovy-2.4.4.jar!/:2.4.4] > at org.codehaus.groovy.runtime.callsite.BooleanReturningMethodInvoker.invoke(BooleanReturningMethodInvoker.java:51) ~[groovy-2.4.4.jar!/:2.4.4] > at org.codehaus.groovy.runtime.callsite.BooleanClosureWrapper.call(BooleanClosureWrapper.java:53) ~[groovy-2.4.4.jar!/:2.4.4] > at org.codehaus.groovy.runtime.DefaultGroovyMethods.find(DefaultGroovyMethods.java:3908) ~[groovy-2.4.4.jar!/:2.4.4] > at org.codehaus.groovy.runtime.dgm$191.invoke(Unknown Source) ~[groovy-2.4.4.jar!/:2.4.4] > at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:274) ~[groovy-2.4.4.jar!/:2.4.4] > at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56) ~[groovy-2.4.4.jar!/:2.4.4] > at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125) [groovy-2.4.4.jar!/:2.4.4] > at basex.support.AddResourcesSupport.consume(AddResourcesSupport.groovy:251) ~[basex-1.0.jar!/:na] > > - - - - > > Best Regards, > Martin
CG
Christian Grünin reply to Strasser Pablo, 5 Aug 2015, 19:596 Aug 2015 09:54 #
Hi Pablo, You are right; your expression raised an error, because the existence of documents is checked at compile-time (to allow for subsequent index rewritings, non-existing element names, etc.). I realized that the behavior you observed was compliant with the spec, which does actually not dictate an evaluation order [1]. However, as we are already replacing compile-time errors with an fn:error() function in other expressions – such as if/then/else – and further operands are skipped once the final result is known, it was just consistent to do the same with and/or. I have revised the code and uploaded a new snapshot [2]. Thanks for your report, Christian [1] http://www.w3.org/TR/xquery-31/#id-logical-expressions [2] http://files.basex.org/releases/latest/
Quoted text · Strasser Pablo · 33 lines
On Wed, Aug 5, 2015 at 7:59 PM, Strasser Pablo <strasserpablo at bluewin.ch> wrote: > Hi Christian, > > Thanks for the answer. > > It seem that if there is a call to the doc function it doesn't > short-circuit. > > Your exemple work, however this one does not: > 1 or doc('non_existent') > > Which return [FODC0002] Resource '/home/pablo/nonex' does not exist. > > With respect of your answer I believe this behaviour is a bug. > > Pablo > > > On 05/08/15 18:05, Christian Grün wrote: >> >> Hi Pablo, >> >>> Should code of this type depending on the short-circuit evaluation of the >>> and operator evaluate correctly? >> >> Yes, it does. Here is a mini example (it would raise an error otherwise): >> >> 1 or error() >> >> Hope this helps, >> Christian >> >

⚡Generated with XQuery