Out Of Memory

On Wed, Nov 5, 2014 at 8:48 PM, Mansi Sheth <mansi.sheth at gmail.com> wrote: > Hello, > > I have a use case, where I have to extract lots in information from each XML > in each DB. Something like, attribute values of most of the nodes in an XML. > For such, queries based goes Out Of Memory with below exception. I am giving > it ~12GB of RAM on i7 processor. Well I can't complain here since I am most > definitely asking for loads of data, but is there any way I can get these > kinds of data successfully ? > > mansi-veracode:BigData mansiadmin$ ~/Downloads/basex/bin/basexhttp > BaseX 8.0 beta b45c1e2 [Server] > Server was started (port: 1984) > HTTP Server was started (port: 8984) > Exception in thread "qtp2068921630-18" java.lang.OutOfMemoryError: Java heap > space > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.addConditionWaiter(AbstractQueuedSynchronizer.java:1857) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2073) > at > org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) > at java.lang.Thread.run(Thread.java:744) > > > -- > - Mansi

De : basex-talk-bounces at mailman.uni-konstanz.de [mailto:basex-talk-bounces at mailman.uni-konstanz.de] De la part de Mansi Sheth Envoyé : jeudi 6 novembre 2014 16:33 À : Christian Grün Cc : BaseX Objet : Re: [basex-talk] Out Of Memory This would need a lot of details, so bear with me below: Briefly my XML files look like: <A name=""> <B name=""> <C name=""> <D name=""> <E name=""/> <A> can contain <B>, <C> or <D> and B, C or D can contain E. We have 1000s (currently 3000 in my test data set) of such xml files, of size 50MB on an average. Its tons of data ! Currently, my database is of ~18GB in size. Query: /A/*//E/@name/string() This query, was going OOM, within few mins. I tried a few ways, of whitelisting, with contain clause, to truncate the result set. That didn't help too. So, now I am out of ideas. This is giving JVM 10GB of dedicated memory. Once, above query works and doesn't go Out Of Memory, I also need corresponding file names too: XYZ.xml //E/@name PQR.xml //E/@name Let me know if you would need more details, to appreciate the issue ? - Mansi On Thu, Nov 6, 2014 at 8:48 AM, Christian Grün <christian.gruen at gmail.com<mailto:christian.gruen at gmail.com>> wrote: Hi Mansi, I think we need more information on the queries that are causing the problems. Best, Christian On Wed, Nov 5, 2014 at 8:48 PM, Mansi Sheth <mansi.sheth at gmail.com<mailto:mansi.sheth at gmail.com>> wrote: > Hello, > > I have a use case, where I have to extract lots in information from each XML > in each DB. Something like, attribute values of most of the nodes in an XML. > For such, queries based goes Out Of Memory with below exception. I am giving > it ~12GB of RAM on i7 processor. Well I can't complain here since I am most > definitely asking for loads of data, but is there any way I can get these > kinds of data successfully ? > > mansi-veracode:BigData mansiadmin$ ~/Downloads/basex/bin/basexhttp > BaseX 8.0 beta b45c1e2 [Server] > Server was started (port: 1984) > HTTP Server was started (port: 8984) > Exception in thread "qtp2068921630-18" java.lang.OutOfMemoryError: Java heap > space > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.addConditionWaiter(AbstractQueuedSynchronizer.java:1857) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2073) > at > org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) > at java.lang.Thread.run(Thread.java:744) > > > -- > - Mansi -- - Mansi

> Query: /A/*//E/@name/string()

> > This query, was going OOM, within few mins. > > I tried a few ways, of whitelisting, with contain clause, to truncate the > result set. That didn't help too. So, now I am out of ideas. This is giving > JVM 10GB of dedicated memory. > > Once, above query works and doesn't go Out Of Memory, I also need > corresponding file names too: > > XYZ.xml //E/@name > PQR.xml //E/@name > > Let me know if you would need more details, to appreciate the issue ? > - Mansi > > On Thu, Nov 6, 2014 at 8:48 AM, Christian Grün <christian.gruen at gmail.com> > wrote: >> >> Hi Mansi, >> >> I think we need more information on the queries that are causing the >> problems. >> >> Best, >> Christian >> >> >> >> On Wed, Nov 5, 2014 at 8:48 PM, Mansi Sheth <mansi.sheth at gmail.com> wrote: >> > Hello, >> > >> > I have a use case, where I have to extract lots in information from each >> > XML >> > in each DB. Something like, attribute values of most of the nodes in an >> > XML. >> > For such, queries based goes Out Of Memory with below exception. I am >> > giving >> > it ~12GB of RAM on i7 processor. Well I can't complain here since I am >> > most >> > definitely asking for loads of data, but is there any way I can get >> > these >> > kinds of data successfully ? >> > >> > mansi-veracode:BigData mansiadmin$ ~/Downloads/basex/bin/basexhttp >> > BaseX 8.0 beta b45c1e2 [Server] >> > Server was started (port: 1984) >> > HTTP Server was started (port: 8984) >> > Exception in thread "qtp2068921630-18" java.lang.OutOfMemoryError: Java >> > heap >> > space >> > at >> > >> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.addConditionWaiter(AbstractQueuedSynchronizer.java:1857) >> > at >> > >> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2073) >> > at >> > >> > org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) >> > at >> > >> > org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) >> > at >> > >> > org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) >> > at >> > >> > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) >> > at java.lang.Thread.run(Thread.java:744) >> > >> > >> > -- >> > - Mansi > > > > > -- > - Mansi

De : Mansi Sheth [mailto:mansi.sheth at gmail.com] Envoyé : jeudi 6 novembre 2014 17:11 À : Fabrice Etanchaud Cc : Christian Grün; BaseX Objet : Re: [basex-talk] Out Of Memory Interesting idea, I thought of using db partition, but didn't pursue it further, mainly due to below thought process. Currently, I am ingesting ~3000 xml files, storing ~50 xml files per db, which would be growing quickly. So, below approach would lead to ~3000 more files (which would be increasing), increasing I/O operations considerably for further pre-processing. However, I don't really care if process takes few minutes to few hours (as long as its not day(s) ;)). Given the situation and my options, I would surely try this. Database, is currently indexed at attribute level, as thats what I would be querying the most. Do you think, I should do anything differently ? Thanks, - Mansi On Thu, Nov 6, 2014 at 10:48 AM, Fabrice Etanchaud <fetanchaud at questel.com<mailto:fetanchaud at questel.com>> wrote: Hi Mansi, Here you have a natural partition of your data : the files you ingested. So my first suggestion would be to query your data on a file basis: for $doc in db:open(‘your_collection_name’) let $file-name := db:path($doc) return file:write( $file-name, <names> { for $name in $doc//E/@name/data() return <name>{$name}</name> } </names> ) Is it for indexing ? Hope it helps, Best regards, Fabrice Etanchaud Questel/Orbit De : basex-talk-bounces at mailman.uni-konstanz.de<mailto:basex-talk-bounces at mailman.uni-konstanz.de> [mailto:basex-talk-bounces at mailman.uni-konstanz.de<mailto:basex-talk-bounces at mailman.uni-konstanz.de>] De la part de Mansi Sheth Envoyé : jeudi 6 novembre 2014 16:33 À : Christian Grün Cc : BaseX Objet : Re: [basex-talk] Out Of Memory This would need a lot of details, so bear with me below: Briefly my XML files look like: <A name=""> <B name=""> <C name=""> <D name=""> <E name=""/> <A> can contain <B>, <C> or <D> and B, C or D can contain E. We have 1000s (currently 3000 in my test data set) of such xml files, of size 50MB on an average. Its tons of data ! Currently, my database is of ~18GB in size. Query: /A/*//E/@name/string() This query, was going OOM, within few mins. I tried a few ways, of whitelisting, with contain clause, to truncate the result set. That didn't help too. So, now I am out of ideas. This is giving JVM 10GB of dedicated memory. Once, above query works and doesn't go Out Of Memory, I also need corresponding file names too: XYZ.xml //E/@name PQR.xml //E/@name Let me know if you would need more details, to appreciate the issue ? - Mansi On Thu, Nov 6, 2014 at 8:48 AM, Christian Grün <christian.gruen at gmail.com<mailto:christian.gruen at gmail.com>> wrote: Hi Mansi, I think we need more information on the queries that are causing the problems. Best, Christian On Wed, Nov 5, 2014 at 8:48 PM, Mansi Sheth <mansi.sheth at gmail.com<mailto:mansi.sheth at gmail.com>> wrote: > Hello, > > I have a use case, where I have to extract lots in information from each XML > in each DB. Something like, attribute values of most of the nodes in an XML. > For such, queries based goes Out Of Memory with below exception. I am giving > it ~12GB of RAM on i7 processor. Well I can't complain here since I am most > definitely asking for loads of data, but is there any way I can get these > kinds of data successfully ? > > mansi-veracode:BigData mansiadmin$ ~/Downloads/basex/bin/basexhttp > BaseX 8.0 beta b45c1e2 [Server] > Server was started (port: 1984) > HTTP Server was started (port: 8984) > Exception in thread "qtp2068921630-18" java.lang.OutOfMemoryError: Java heap > space > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.addConditionWaiter(AbstractQueuedSynchronizer.java:1857) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2073) > at > org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) > at java.lang.Thread.run(Thread.java:744) > > > -- > - Mansi -- - Mansi -- - Mansi

On Thu, Nov 6, 2014 at 8:48 AM, Christian Grün <christian.gruen at gmail.com> wrote: > Hi Mansi, > > I think we need more information on the queries that are causing the > problems. > > Best, > Christian > > > > On Wed, Nov 5, 2014 at 8:48 PM, Mansi Sheth <mansi.sheth at gmail.com> wrote: > > Hello, > > > > I have a use case, where I have to extract lots in information from each > XML > > in each DB. Something like, attribute values of most of the nodes in an > XML. > > For such, queries based goes Out Of Memory with below exception. I am > giving > > it ~12GB of RAM on i7 processor. Well I can't complain here since I am > most > > definitely asking for loads of data, but is there any way I can get these > > kinds of data successfully ? > > > > mansi-veracode:BigData mansiadmin$ ~/Downloads/basex/bin/basexhttp > > BaseX 8.0 beta b45c1e2 [Server] > > Server was started (port: 1984) > > HTTP Server was started (port: 8984) > > Exception in thread "qtp2068921630-18" java.lang.OutOfMemoryError: Java > heap > > space > > at > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.addConditionWaiter(AbstractQueuedSynchronizer.java:1857) > > at > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2073) > > at > > > org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) > > at > > > org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) > > at > > > org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) > > at > > > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) > > at java.lang.Thread.run(Thread.java:744) > > > > > > -- > > - Mansi >

On Thu, Nov 6, 2014 at 10:48 AM, Fabrice Etanchaud <fetanchaud at questel.com> wrote: > Hi Mansi, > > > > Here you have a natural partition of your data : the files you ingested. > > So my first suggestion would be to query your data on a file basis: > > > > for $doc in db:open(‘your_collection_name’) > > let $file-name := db:path($doc) > > return > > file:write( > > $file-name, > > <names> > > { > > for $name in > $doc//E/@name/data() > > return > > > <name>{$name}</name> > > } > > </names> > > ) > > > > Is it for indexing ? > > > > Hope it helps, > > > > Best regards, > > > > Fabrice Etanchaud > > Questel/Orbit > > > > *De :* basex-talk-bounces at mailman.uni-konstanz.de [mailto: > basex-talk-bounces at mailman.uni-konstanz.de] *De la part de* Mansi Sheth > *Envoyé :* jeudi 6 novembre 2014 16:33 > *À :* Christian Grün > *Cc :* BaseX > *Objet :* Re: [basex-talk] Out Of Memory > > > > This would need a lot of details, so bear with me below: > > > > Briefly my XML files look like: > > > > <A name=""> > > <B name=""> > > <C name=""> > > <D name=""> > > <E name=""/> > > > > <A> can contain <B>, <C> or <D> and B, C or D can contain E. We have 1000s > (currently 3000 in my test data set) of such xml files, of size 50MB on an > average. Its tons of data ! Currently, my database is of ~18GB in size. > > > > Query: /A/*//E/@name/string() > > > > This query, was going OOM, within few mins. > > > > I tried a few ways, of whitelisting, with contain clause, to truncate the > result set. That didn't help too. So, now I am out of ideas. This is giving > JVM 10GB of dedicated memory. > > > > Once, above query works and doesn't go Out Of Memory, I also need > corresponding file names too: > > > > XYZ.xml //E/@name > > PQR.xml //E/@name > > > > Let me know if you would need more details, to appreciate the issue ? > > - Mansi > > > > On Thu, Nov 6, 2014 at 8:48 AM, Christian Grün <christian.gruen at gmail.com> > wrote: > > Hi Mansi, > > I think we need more information on the queries that are causing the > problems. > > Best, > Christian > > > > > On Wed, Nov 5, 2014 at 8:48 PM, Mansi Sheth <mansi.sheth at gmail.com> wrote: > > Hello, > > > > I have a use case, where I have to extract lots in information from each > XML > > in each DB. Something like, attribute values of most of the nodes in an > XML. > > For such, queries based goes Out Of Memory with below exception. I am > giving > > it ~12GB of RAM on i7 processor. Well I can't complain here since I am > most > > definitely asking for loads of data, but is there any way I can get these > > kinds of data successfully ? > > > > mansi-veracode:BigData mansiadmin$ ~/Downloads/basex/bin/basexhttp > > BaseX 8.0 beta b45c1e2 [Server] > > Server was started (port: 1984) > > HTTP Server was started (port: 8984) > > Exception in thread "qtp2068921630-18" java.lang.OutOfMemoryError: Java > heap > > space > > at > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.addConditionWaiter(AbstractQueuedSynchronizer.java:1857) > > at > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2073) > > at > > > org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) > > at > > > org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) > > at > > > org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) > > at > > > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) > > at java.lang.Thread.run(Thread.java:744) > > > > > > -- > > - Mansi > > > > > > -- > > - Mansi >

On Thu, Nov 6, 2014 at 11:11 AM, Mansi Sheth <mansi.sheth at gmail.com> wrote: > Interesting idea, I thought of using db partition, but didn't pursue it > further, mainly due to below thought process. > > Currently, I am ingesting ~3000 xml files, storing ~50 xml files per db, > which would be growing quickly. So, below approach would lead to ~3000 more > files (which would be increasing), increasing I/O operations considerably > for further pre-processing. > > However, I don't really care if process takes few minutes to few hours (as > long as its not day(s) ;)). Given the situation and my options, I would > surely try this. > > Database, is currently indexed at attribute level, as thats what I would be > querying the most. Do you think, I should do anything differently ? > > Thanks, > - Mansi > > On Thu, Nov 6, 2014 at 10:48 AM, Fabrice Etanchaud <fetanchaud at questel.com> > wrote: >> >> Hi Mansi, >> >> >> >> Here you have a natural partition of your data : the files you ingested. >> >> So my first suggestion would be to query your data on a file basis: >> >> >> >> for $doc in db:open(‘your_collection_name’) >> >> let $file-name := db:path($doc) >> >> return >> >> file:write( >> >> $file-name, >> >> <names> >> >> { >> >> for $name in >> $doc//E/@name/data() >> >> return >> >> >> <name>{$name}</name> >> >> } >> >> </names> >> >> ) >> >> >> >> Is it for indexing ? >> >> >> >> Hope it helps, >> >> >> >> Best regards, >> >> >> >> Fabrice Etanchaud >> >> Questel/Orbit >> >> >> >> De : basex-talk-bounces at mailman.uni-konstanz.de >> [mailto:basex-talk-bounces at mailman.uni-konstanz.de] De la part de Mansi >> Sheth >> Envoyé : jeudi 6 novembre 2014 16:33 >> À : Christian Grün >> Cc : BaseX >> Objet : Re: [basex-talk] Out Of Memory >> >> >> >> This would need a lot of details, so bear with me below: >> >> >> >> Briefly my XML files look like: >> >> >> >> <A name=""> >> >> <B name=""> >> >> <C name=""> >> >> <D name=""> >> >> <E name=""/> >> >> >> >> <A> can contain <B>, <C> or <D> and B, C or D can contain E. We have 1000s >> (currently 3000 in my test data set) of such xml files, of size 50MB on an >> average. Its tons of data ! Currently, my database is of ~18GB in size. >> >> >> >> Query: /A/*//E/@name/string() >> >> >> >> This query, was going OOM, within few mins. >> >> >> >> I tried a few ways, of whitelisting, with contain clause, to truncate the >> result set. That didn't help too. So, now I am out of ideas. This is giving >> JVM 10GB of dedicated memory. >> >> >> >> Once, above query works and doesn't go Out Of Memory, I also need >> corresponding file names too: >> >> >> >> XYZ.xml //E/@name >> >> PQR.xml //E/@name >> >> >> >> Let me know if you would need more details, to appreciate the issue ? >> >> - Mansi >> >> >> >> On Thu, Nov 6, 2014 at 8:48 AM, Christian Grün <christian.gruen at gmail.com> >> wrote: >> >> Hi Mansi, >> >> I think we need more information on the queries that are causing the >> problems. >> >> Best, >> Christian >> >> >> >> >> On Wed, Nov 5, 2014 at 8:48 PM, Mansi Sheth <mansi.sheth at gmail.com> wrote: >> > Hello, >> > >> > I have a use case, where I have to extract lots in information from each >> > XML >> > in each DB. Something like, attribute values of most of the nodes in an >> > XML. >> > For such, queries based goes Out Of Memory with below exception. I am >> > giving >> > it ~12GB of RAM on i7 processor. Well I can't complain here since I am >> > most >> > definitely asking for loads of data, but is there any way I can get >> > these >> > kinds of data successfully ? >> > >> > mansi-veracode:BigData mansiadmin$ ~/Downloads/basex/bin/basexhttp >> > BaseX 8.0 beta b45c1e2 [Server] >> > Server was started (port: 1984) >> > HTTP Server was started (port: 8984) >> > Exception in thread "qtp2068921630-18" java.lang.OutOfMemoryError: Java >> > heap >> > space >> > at >> > >> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.addConditionWaiter(AbstractQueuedSynchronizer.java:1857) >> > at >> > >> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2073) >> > at >> > >> > org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) >> > at >> > >> > org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) >> > at >> > >> > org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) >> > at >> > >> > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) >> > at java.lang.Thread.run(Thread.java:744) >> > >> > >> > -- >> > - Mansi >> >> >> >> >> >> -- >> >> - Mansi > > > > > -- > - Mansi

On Thu, Nov 6, 2014 at 11:19 AM, Christian Grün <christian.gruen at gmail.com> wrote: > > Query: /A/*//E/@name/string() > > In the GUI, all results will be cached, so you could think about > switching to command line. > > Do you really need to output all results, or do you do some further > processing with the intermediate results? > > For example, the query "count(/A/*//E/@name/string())" will probably > run without getting stuck. > > > > > > This query, was going OOM, within few mins. > > > > I tried a few ways, of whitelisting, with contain clause, to truncate the > > result set. That didn't help too. So, now I am out of ideas. This is > giving > > JVM 10GB of dedicated memory. > > > > Once, above query works and doesn't go Out Of Memory, I also need > > corresponding file names too: > > > > XYZ.xml //E/@name > > PQR.xml //E/@name > > > > Let me know if you would need more details, to appreciate the issue ? > > - Mansi > > > > On Thu, Nov 6, 2014 at 8:48 AM, Christian Grün < > christian.gruen at gmail.com> > > wrote: > >> > >> Hi Mansi, > >> > >> I think we need more information on the queries that are causing the > >> problems. > >> > >> Best, > >> Christian > >> > >> > >> > >> On Wed, Nov 5, 2014 at 8:48 PM, Mansi Sheth <mansi.sheth at gmail.com> > wrote: > >> > Hello, > >> > > >> > I have a use case, where I have to extract lots in information from > each > >> > XML > >> > in each DB. Something like, attribute values of most of the nodes in > an > >> > XML. > >> > For such, queries based goes Out Of Memory with below exception. I am > >> > giving > >> > it ~12GB of RAM on i7 processor. Well I can't complain here since I am > >> > most > >> > definitely asking for loads of data, but is there any way I can get > >> > these > >> > kinds of data successfully ? > >> > > >> > mansi-veracode:BigData mansiadmin$ ~/Downloads/basex/bin/basexhttp > >> > BaseX 8.0 beta b45c1e2 [Server] > >> > Server was started (port: 1984) > >> > HTTP Server was started (port: 8984) > >> > Exception in thread "qtp2068921630-18" java.lang.OutOfMemoryError: > Java > >> > heap > >> > space > >> > at > >> > > >> > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.addConditionWaiter(AbstractQueuedSynchronizer.java:1857) > >> > at > >> > > >> > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2073) > >> > at > >> > > >> > > org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) > >> > at > >> > > >> > > org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) > >> > at > >> > > >> > > org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) > >> > at > >> > > >> > > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) > >> > at java.lang.Thread.run(Thread.java:744) > >> > > >> > > >> > -- > >> > - Mansi > > > > > > > > > > -- > > - Mansi >

On Thu, Nov 6, 2014 at 11:23 AM, Fabrice Etanchaud <fetanchaud at questel.com> wrote: > The solution depends on the usage you will have of your extraction. > > May I ask you what is your extraction for ? > > > > Best regards, > > Fabrice > > > > *De :* Mansi Sheth [mailto:mansi.sheth at gmail.com] > *Envoyé :* jeudi 6 novembre 2014 17:11 > *À :* Fabrice Etanchaud > *Cc :* Christian Grün; BaseX > > *Objet :* Re: [basex-talk] Out Of Memory > > > > Interesting idea, I thought of using db partition, but didn't pursue it > further, mainly due to below thought process. > > > > Currently, I am ingesting ~3000 xml files, storing ~50 xml files per db, > which would be growing quickly. So, below approach would lead to ~3000 more > files (which would be increasing), increasing I/O operations considerably > for further pre-processing. > > > > However, I don't really care if process takes few minutes to few hours (as > long as its not day(s) ;)). Given the situation and my options, I would > surely try this. > > > > Database, is currently indexed at attribute level, as thats what I would > be querying the most. Do you think, I should do anything differently ? > > > > Thanks, > > - Mansi > > > > On Thu, Nov 6, 2014 at 10:48 AM, Fabrice Etanchaud <fetanchaud at questel.com> > wrote: > > Hi Mansi, > > > > Here you have a natural partition of your data : the files you ingested. > > So my first suggestion would be to query your data on a file basis: > > > > for $doc in db:open(‘your_collection_name’) > > let $file-name := db:path($doc) > > return > > file:write( > > $file-name, > > <names> > > { > > for $name in > $doc//E/@name/data() > > return > > > <name>{$name}</name> > > } > > </names> > > ) > > > > Is it for indexing ? > > > > Hope it helps, > > > > Best regards, > > > > Fabrice Etanchaud > > Questel/Orbit > > > > *De :* basex-talk-bounces at mailman.uni-konstanz.de [mailto: > basex-talk-bounces at mailman.uni-konstanz.de] *De la part de* Mansi Sheth > *Envoyé :* jeudi 6 novembre 2014 16:33 > *À :* Christian Grün > *Cc :* BaseX > *Objet :* Re: [basex-talk] Out Of Memory > > > > This would need a lot of details, so bear with me below: > > > > Briefly my XML files look like: > > > > <A name=""> > > <B name=""> > > <C name=""> > > <D name=""> > > <E name=""/> > > > > <A> can contain <B>, <C> or <D> and B, C or D can contain E. We have 1000s > (currently 3000 in my test data set) of such xml files, of size 50MB on an > average. Its tons of data ! Currently, my database is of ~18GB in size. > > > > Query: /A/*//E/@name/string() > > > > This query, was going OOM, within few mins. > > > > I tried a few ways, of whitelisting, with contain clause, to truncate the > result set. That didn't help too. So, now I am out of ideas. This is giving > JVM 10GB of dedicated memory. > > > > Once, above query works and doesn't go Out Of Memory, I also need > corresponding file names too: > > > > XYZ.xml //E/@name > > PQR.xml //E/@name > > > > Let me know if you would need more details, to appreciate the issue ? > > - Mansi > > > > On Thu, Nov 6, 2014 at 8:48 AM, Christian Grün <christian.gruen at gmail.com> > wrote: > > Hi Mansi, > > I think we need more information on the queries that are causing the > problems. > > Best, > Christian > > > > > On Wed, Nov 5, 2014 at 8:48 PM, Mansi Sheth <mansi.sheth at gmail.com> wrote: > > Hello, > > > > I have a use case, where I have to extract lots in information from each > XML > > in each DB. Something like, attribute values of most of the nodes in an > XML. > > For such, queries based goes Out Of Memory with below exception. I am > giving > > it ~12GB of RAM on i7 processor. Well I can't complain here since I am > most > > definitely asking for loads of data, but is there any way I can get these > > kinds of data successfully ? > > > > mansi-veracode:BigData mansiadmin$ ~/Downloads/basex/bin/basexhttp > > BaseX 8.0 beta b45c1e2 [Server] > > Server was started (port: 1984) > > HTTP Server was started (port: 8984) > > Exception in thread "qtp2068921630-18" java.lang.OutOfMemoryError: Java > heap > > space > > at > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.addConditionWaiter(AbstractQueuedSynchronizer.java:1857) > > at > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2073) > > at > > > org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) > > at > > > org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) > > at > > > org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) > > at > > > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) > > at java.lang.Thread.run(Thread.java:744) > > > > > > -- > > - Mansi > > > > > > -- > > - Mansi > > > > > > -- > > - Mansi >

On Thu, Nov 6, 2014 at 2:58 PM, Mansi Sheth <mansi.sheth at gmail.com> wrote: > Briefly explaining, trying to extract these values/per xml file (where .xml > files are ID), to map it to its corresponding values. > > Imagine, you have 100s of customers, and each customer uses/needs 1000s of > different "@name". These "@name" would be similar across customer, but few > would be using some values, few customer some other. Trying to collect all > this information and find, which "@name" is used by most customer and so on > and so forth. There are few such use cases, this one being most generic. > > > On Thu, Nov 6, 2014 at 11:23 AM, Fabrice Etanchaud <fetanchaud at questel.com> > wrote: >> >> The solution depends on the usage you will have of your extraction. >> >> May I ask you what is your extraction for ? >> >> >> >> Best regards, >> >> Fabrice >> >> >> >> De : Mansi Sheth [mailto:mansi.sheth at gmail.com] >> Envoyé : jeudi 6 novembre 2014 17:11 >> À : Fabrice Etanchaud >> Cc : Christian Grün; BaseX >> >> >> Objet : Re: [basex-talk] Out Of Memory >> >> >> >> Interesting idea, I thought of using db partition, but didn't pursue it >> further, mainly due to below thought process. >> >> >> >> Currently, I am ingesting ~3000 xml files, storing ~50 xml files per db, >> which would be growing quickly. So, below approach would lead to ~3000 more >> files (which would be increasing), increasing I/O operations considerably >> for further pre-processing. >> >> >> >> However, I don't really care if process takes few minutes to few hours (as >> long as its not day(s) ;)). Given the situation and my options, I would >> surely try this. >> >> >> >> Database, is currently indexed at attribute level, as thats what I would >> be querying the most. Do you think, I should do anything differently ? >> >> >> >> Thanks, >> >> - Mansi >> >> >> >> On Thu, Nov 6, 2014 at 10:48 AM, Fabrice Etanchaud >> <fetanchaud at questel.com> wrote: >> >> Hi Mansi, >> >> >> >> Here you have a natural partition of your data : the files you ingested. >> >> So my first suggestion would be to query your data on a file basis: >> >> >> >> for $doc in db:open(‘your_collection_name’) >> >> let $file-name := db:path($doc) >> >> return >> >> file:write( >> >> $file-name, >> >> <names> >> >> { >> >> for $name in >> $doc//E/@name/data() >> >> return >> >> >> <name>{$name}</name> >> >> } >> >> </names> >> >> ) >> >> >> >> Is it for indexing ? >> >> >> >> Hope it helps, >> >> >> >> Best regards, >> >> >> >> Fabrice Etanchaud >> >> Questel/Orbit >> >> >> >> De : basex-talk-bounces at mailman.uni-konstanz.de >> [mailto:basex-talk-bounces at mailman.uni-konstanz.de] De la part de Mansi >> Sheth >> Envoyé : jeudi 6 novembre 2014 16:33 >> À : Christian Grün >> Cc : BaseX >> Objet : Re: [basex-talk] Out Of Memory >> >> >> >> This would need a lot of details, so bear with me below: >> >> >> >> Briefly my XML files look like: >> >> >> >> <A name=""> >> >> <B name=""> >> >> <C name=""> >> >> <D name=""> >> >> <E name=""/> >> >> >> >> <A> can contain <B>, <C> or <D> and B, C or D can contain E. We have 1000s >> (currently 3000 in my test data set) of such xml files, of size 50MB on an >> average. Its tons of data ! Currently, my database is of ~18GB in size. >> >> >> >> Query: /A/*//E/@name/string() >> >> >> >> This query, was going OOM, within few mins. >> >> >> >> I tried a few ways, of whitelisting, with contain clause, to truncate the >> result set. That didn't help too. So, now I am out of ideas. This is giving >> JVM 10GB of dedicated memory. >> >> >> >> Once, above query works and doesn't go Out Of Memory, I also need >> corresponding file names too: >> >> >> >> XYZ.xml //E/@name >> >> PQR.xml //E/@name >> >> >> >> Let me know if you would need more details, to appreciate the issue ? >> >> - Mansi >> >> >> >> On Thu, Nov 6, 2014 at 8:48 AM, Christian Grün <christian.gruen at gmail.com> >> wrote: >> >> Hi Mansi, >> >> I think we need more information on the queries that are causing the >> problems. >> >> Best, >> Christian >> >> >> >> >> On Wed, Nov 5, 2014 at 8:48 PM, Mansi Sheth <mansi.sheth at gmail.com> wrote: >> > Hello, >> > >> > I have a use case, where I have to extract lots in information from each >> > XML >> > in each DB. Something like, attribute values of most of the nodes in an >> > XML. >> > For such, queries based goes Out Of Memory with below exception. I am >> > giving >> > it ~12GB of RAM on i7 processor. Well I can't complain here since I am >> > most >> > definitely asking for loads of data, but is there any way I can get >> > these >> > kinds of data successfully ? >> > >> > mansi-veracode:BigData mansiadmin$ ~/Downloads/basex/bin/basexhttp >> > BaseX 8.0 beta b45c1e2 [Server] >> > Server was started (port: 1984) >> > HTTP Server was started (port: 8984) >> > Exception in thread "qtp2068921630-18" java.lang.OutOfMemoryError: Java >> > heap >> > space >> > at >> > >> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.addConditionWaiter(AbstractQueuedSynchronizer.java:1857) >> > at >> > >> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2073) >> > at >> > >> > org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) >> > at >> > >> > org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) >> > at >> > >> > org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) >> > at >> > >> > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) >> > at java.lang.Thread.run(Thread.java:744) >> > >> > >> > -- >> > - Mansi >> >> >> >> >> >> -- >> >> - Mansi >> >> >> >> >> >> -- >> >> - Mansi > > > > > -- > - Mansi

De : basex-talk-bounces at mailman.uni-konstanz.de [mailto:basex-talk-bounces at mailman.uni-konstanz.de] De la part de Mansi Sheth Envoyé : jeudi 6 novembre 2014 20:55 À : Christian Grün Cc : BaseX Objet : Re: [basex-talk] Out Of Memory I would be doing tons of post processing. I never use UI. I either use REST thru cURL or command line. I would basically need data in below format: XML File Name, @name I am trying to whitelist picking up values for only "starts-with(@name,"pqr"). where "pqr" is a list of 150 odd values. My file names, are essentially some ID/keys, which I would need to map it further using sqlite to some values and may be group by it.. etc. So, basically I am trying to visualize some data, based on its existence in which xml files. So, yes count(<query>) would be fine, but won't solve much purpose, since I still need value "pqr". - Mansi On Thu, Nov 6, 2014 at 11:19 AM, Christian Grün <christian.gruen at gmail.com<mailto:christian.gruen at gmail.com>> wrote: > Query: /A/*//E/@name/string() In the GUI, all results will be cached, so you could think about switching to command line. Do you really need to output all results, or do you do some further processing with the intermediate results? For example, the query "count(/A/*//E/@name/string())" will probably run without getting stuck. > > This query, was going OOM, within few mins. > > I tried a few ways, of whitelisting, with contain clause, to truncate the > result set. That didn't help too. So, now I am out of ideas. This is giving > JVM 10GB of dedicated memory. > > Once, above query works and doesn't go Out Of Memory, I also need > corresponding file names too: > > XYZ.xml //E/@name > PQR.xml //E/@name > > Let me know if you would need more details, to appreciate the issue ? > - Mansi > > On Thu, Nov 6, 2014 at 8:48 AM, Christian Grün <christian.gruen at gmail.com<mailto:christian.gruen at gmail.com>> > wrote: >> >> Hi Mansi, >> >> I think we need more information on the queries that are causing the >> problems. >> >> Best, >> Christian >> >> >> >> On Wed, Nov 5, 2014 at 8:48 PM, Mansi Sheth <mansi.sheth at gmail.com<mailto:mansi.sheth at gmail.com>> wrote: >> > Hello, >> > >> > I have a use case, where I have to extract lots in information from each >> > XML >> > in each DB. Something like, attribute values of most of the nodes in an >> > XML. >> > For such, queries based goes Out Of Memory with below exception. I am >> > giving >> > it ~12GB of RAM on i7 processor. Well I can't complain here since I am >> > most >> > definitely asking for loads of data, but is there any way I can get >> > these >> > kinds of data successfully ? >> > >> > mansi-veracode:BigData mansiadmin$ ~/Downloads/basex/bin/basexhttp >> > BaseX 8.0 beta b45c1e2 [Server] >> > Server was started (port: 1984) >> > HTTP Server was started (port: 8984) >> > Exception in thread "qtp2068921630-18" java.lang.OutOfMemoryError: Java >> > heap >> > space >> > at >> > >> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.addConditionWaiter(AbstractQueuedSynchronizer.java:1857) >> > at >> > >> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2073) >> > at >> > >> > org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) >> > at >> > >> > org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) >> > at >> > >> > org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) >> > at >> > >> > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) >> > at java.lang.Thread.run(Thread.java:744) >> > >> > >> > -- >> > - Mansi > > > > > -- > - Mansi -- - Mansi

> Once, above query works and doesn't go Out Of Memory, I also need corresponding file names too:

On Thu, Nov 6, 2014 at 8:58 PM, Mansi Sheth <mansi.sheth at gmail.com> wrote: > Briefly explaining, trying to extract these values/per xml file (where .xml > files are ID), to map it to its corresponding values. > > Imagine, you have 100s of customers, and each customer uses/needs 1000s of > different "@name". These "@name" would be similar across customer, but few > would be using some values, few customer some other. Trying to collect all > this information and find, which "@name" is used by most customer and so on > and so forth. There are few such use cases, this one being most generic. > > > On Thu, Nov 6, 2014 at 11:23 AM, Fabrice Etanchaud <fetanchaud at questel.com> > wrote: >> >> The solution depends on the usage you will have of your extraction. >> >> May I ask you what is your extraction for ? >> >> >> >> Best regards, >> >> Fabrice >> >> >> >> De : Mansi Sheth [mailto:mansi.sheth at gmail.com] >> Envoyé : jeudi 6 novembre 2014 17:11 >> À : Fabrice Etanchaud >> Cc : Christian Grün; BaseX >> >> >> Objet : Re: [basex-talk] Out Of Memory >> >> >> >> Interesting idea, I thought of using db partition, but didn't pursue it >> further, mainly due to below thought process. >> >> >> >> Currently, I am ingesting ~3000 xml files, storing ~50 xml files per db, >> which would be growing quickly. So, below approach would lead to ~3000 more >> files (which would be increasing), increasing I/O operations considerably >> for further pre-processing. >> >> >> >> However, I don't really care if process takes few minutes to few hours (as >> long as its not day(s) ;)). Given the situation and my options, I would >> surely try this. >> >> >> >> Database, is currently indexed at attribute level, as thats what I would >> be querying the most. Do you think, I should do anything differently ? >> >> >> >> Thanks, >> >> - Mansi >> >> >> >> On Thu, Nov 6, 2014 at 10:48 AM, Fabrice Etanchaud >> <fetanchaud at questel.com> wrote: >> >> Hi Mansi, >> >> >> >> Here you have a natural partition of your data : the files you ingested. >> >> So my first suggestion would be to query your data on a file basis: >> >> >> >> for $doc in db:open(‘your_collection_name’) >> >> let $file-name := db:path($doc) >> >> return >> >> file:write( >> >> $file-name, >> >> <names> >> >> { >> >> for $name in >> $doc//E/@name/data() >> >> return >> >> >> <name>{$name}</name> >> >> } >> >> </names> >> >> ) >> >> >> >> Is it for indexing ? >> >> >> >> Hope it helps, >> >> >> >> Best regards, >> >> >> >> Fabrice Etanchaud >> >> Questel/Orbit >> >> >> >> De : basex-talk-bounces at mailman.uni-konstanz.de >> [mailto:basex-talk-bounces at mailman.uni-konstanz.de] De la part de Mansi >> Sheth >> Envoyé : jeudi 6 novembre 2014 16:33 >> À : Christian Grün >> Cc : BaseX >> Objet : Re: [basex-talk] Out Of Memory >> >> >> >> This would need a lot of details, so bear with me below: >> >> >> >> Briefly my XML files look like: >> >> >> >> <A name=""> >> >> <B name=""> >> >> <C name=""> >> >> <D name=""> >> >> <E name=""/> >> >> >> >> <A> can contain <B>, <C> or <D> and B, C or D can contain E. We have 1000s >> (currently 3000 in my test data set) of such xml files, of size 50MB on an >> average. Its tons of data ! Currently, my database is of ~18GB in size. >> >> >> >> Query: /A/*//E/@name/string() >> >> >> >> This query, was going OOM, within few mins. >> >> >> >> I tried a few ways, of whitelisting, with contain clause, to truncate the >> result set. That didn't help too. So, now I am out of ideas. This is giving >> JVM 10GB of dedicated memory. >> >> >> >> Once, above query works and doesn't go Out Of Memory, I also need >> corresponding file names too: >> >> >> >> XYZ.xml //E/@name >> >> PQR.xml //E/@name >> >> >> >> Let me know if you would need more details, to appreciate the issue ? >> >> - Mansi >> >> >> >> On Thu, Nov 6, 2014 at 8:48 AM, Christian Grün <christian.gruen at gmail.com> >> wrote: >> >> Hi Mansi, >> >> I think we need more information on the queries that are causing the >> problems. >> >> Best, >> Christian >> >> >> >> >> On Wed, Nov 5, 2014 at 8:48 PM, Mansi Sheth <mansi.sheth at gmail.com> wrote: >> > Hello, >> > >> > I have a use case, where I have to extract lots in information from each >> > XML >> > in each DB. Something like, attribute values of most of the nodes in an >> > XML. >> > For such, queries based goes Out Of Memory with below exception. I am >> > giving >> > it ~12GB of RAM on i7 processor. Well I can't complain here since I am >> > most >> > definitely asking for loads of data, but is there any way I can get >> > these >> > kinds of data successfully ? >> > >> > mansi-veracode:BigData mansiadmin$ ~/Downloads/basex/bin/basexhttp >> > BaseX 8.0 beta b45c1e2 [Server] >> > Server was started (port: 1984) >> > HTTP Server was started (port: 8984) >> > Exception in thread "qtp2068921630-18" java.lang.OutOfMemoryError: Java >> > heap >> > space >> > at >> > >> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.addConditionWaiter(AbstractQueuedSynchronizer.java:1857) >> > at >> > >> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2073) >> > at >> > >> > org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) >> > at >> > >> > org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) >> > at >> > >> > org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) >> > at >> > >> > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) >> > at java.lang.Thread.run(Thread.java:744) >> > >> > >> > -- >> > - Mansi >> >> >> >> >> >> -- >> >> - Mansi >> >> >> >> >> >> -- >> >> - Mansi > > > > > -- > - Mansi

> do you need more flexibility?

On Fri, Nov 7, 2014 at 10:48 AM, Christian Grün <christian.gruen at gmail.com> wrote: > declare option output:item-separator "&#xa;"; > for $db in db:open('....') > let $path := db:path($db) > for $name in $db//E/@name > return $path || out:tab() || $name

On Fri, Nov 7, 2014 at 3:48 AM, Fabrice Etanchaud <fetanchaud at questel.com> wrote: > Hi Mansi, > > > > From what I can see, > > for each pqr value, you could use db:attribute-range to retrieve all the > file names, group by/count to obtain statistics. > > You could also create a new collection from an extraction of only the data > you need, changing @name into element and use full text fuzzy match. > > > > Hoping it helps > > > > Cordialement > > Fabrice > > > > *De :* basex-talk-bounces at mailman.uni-konstanz.de [mailto: > basex-talk-bounces at mailman.uni-konstanz.de] *De la part de* Mansi Sheth > *Envoyé :* jeudi 6 novembre 2014 20:55 > *À :* Christian Grün > *Cc :* BaseX > *Objet :* Re: [basex-talk] Out Of Memory > > > > I would be doing tons of post processing. I never use UI. I either use > REST thru cURL or command line. > > > > I would basically need data in below format: > > > > XML File Name, @name > > > > I am trying to whitelist picking up values for only > "starts-with(@name,"pqr"). where "pqr" is a list of 150 odd values. > > > > My file names, are essentially some ID/keys, which I would need to map it > further using sqlite to some values and may be group by it.. etc. > > > > So, basically I am trying to visualize some data, based on its existence > in which xml files. So, yes count(<query>) would be fine, but won't solve > much purpose, since I still need value "pqr". > > > > - Mansi > > > > > > On Thu, Nov 6, 2014 at 11:19 AM, Christian Grün <christian.gruen at gmail.com> > wrote: > > > Query: /A/*//E/@name/string() > > In the GUI, all results will be cached, so you could think about > switching to command line. > > Do you really need to output all results, or do you do some further > processing with the intermediate results? > > For example, the query "count(/A/*//E/@name/string())" will probably > run without getting stuck. > > > > > > > This query, was going OOM, within few mins. > > > > I tried a few ways, of whitelisting, with contain clause, to truncate the > > result set. That didn't help too. So, now I am out of ideas. This is > giving > > JVM 10GB of dedicated memory. > > > > Once, above query works and doesn't go Out Of Memory, I also need > > corresponding file names too: > > > > XYZ.xml //E/@name > > PQR.xml //E/@name > > > > Let me know if you would need more details, to appreciate the issue ? > > - Mansi > > > > On Thu, Nov 6, 2014 at 8:48 AM, Christian Grün < > christian.gruen at gmail.com> > > wrote: > >> > >> Hi Mansi, > >> > >> I think we need more information on the queries that are causing the > >> problems. > >> > >> Best, > >> Christian > >> > >> > >> > >> On Wed, Nov 5, 2014 at 8:48 PM, Mansi Sheth <mansi.sheth at gmail.com> > wrote: > >> > Hello, > >> > > >> > I have a use case, where I have to extract lots in information from > each > >> > XML > >> > in each DB. Something like, attribute values of most of the nodes in > an > >> > XML. > >> > For such, queries based goes Out Of Memory with below exception. I am > >> > giving > >> > it ~12GB of RAM on i7 processor. Well I can't complain here since I am > >> > most > >> > definitely asking for loads of data, but is there any way I can get > >> > these > >> > kinds of data successfully ? > >> > > >> > mansi-veracode:BigData mansiadmin$ ~/Downloads/basex/bin/basexhttp > >> > BaseX 8.0 beta b45c1e2 [Server] > >> > Server was started (port: 1984) > >> > HTTP Server was started (port: 8984) > >> > Exception in thread "qtp2068921630-18" java.lang.OutOfMemoryError: > Java > >> > heap > >> > space > >> > at > >> > > >> > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.addConditionWaiter(AbstractQueuedSynchronizer.java:1857) > >> > at > >> > > >> > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2073) > >> > at > >> > > >> > > org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) > >> > at > >> > > >> > > org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) > >> > at > >> > > >> > > org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) > >> > at > >> > > >> > > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) > >> > at java.lang.Thread.run(Thread.java:744) > >> > > >> > > >> > -- > >> > - Mansi > > > > > > > > > > -- > > - Mansi > > > > > > -- > > - Mansi >

On Fri, Nov 7, 2014 at 12:25 PM, Mansi Sheth <mansi.sheth at gmail.com> wrote: > This email chain, is extremely helpful. Thanks a ton guys. Certainly one > of the most helpful folks here :) > > I have to try a lot of these suggestions but currently I am being pulled > into something else, so I have to pause for the time being. > > Will get back to this email thread, after trying a few things and my > relevant observations. > > - Mansi > > On Fri, Nov 7, 2014 at 3:48 AM, Fabrice Etanchaud <fetanchaud at questel.com> > wrote: > >> Hi Mansi, >> >> >> >> From what I can see, >> >> for each pqr value, you could use db:attribute-range to retrieve all the >> file names, group by/count to obtain statistics. >> >> You could also create a new collection from an extraction of only the >> data you need, changing @name into element and use full text fuzzy match. >> >> >> >> Hoping it helps >> >> >> >> Cordialement >> >> Fabrice >> >> >> >> *De :* basex-talk-bounces at mailman.uni-konstanz.de [mailto: >> basex-talk-bounces at mailman.uni-konstanz.de] *De la part de* Mansi Sheth >> *Envoyé :* jeudi 6 novembre 2014 20:55 >> *À :* Christian Grün >> *Cc :* BaseX >> *Objet :* Re: [basex-talk] Out Of Memory >> >> >> >> I would be doing tons of post processing. I never use UI. I either use >> REST thru cURL or command line. >> >> >> >> I would basically need data in below format: >> >> >> >> XML File Name, @name >> >> >> >> I am trying to whitelist picking up values for only >> "starts-with(@name,"pqr"). where "pqr" is a list of 150 odd values. >> >> >> >> My file names, are essentially some ID/keys, which I would need to map it >> further using sqlite to some values and may be group by it.. etc. >> >> >> >> So, basically I am trying to visualize some data, based on its existence >> in which xml files. So, yes count(<query>) would be fine, but won't solve >> much purpose, since I still need value "pqr". >> >> >> >> - Mansi >> >> >> >> >> >> On Thu, Nov 6, 2014 at 11:19 AM, Christian Grün < >> christian.gruen at gmail.com> wrote: >> >> > Query: /A/*//E/@name/string() >> >> In the GUI, all results will be cached, so you could think about >> switching to command line. >> >> Do you really need to output all results, or do you do some further >> processing with the intermediate results? >> >> For example, the query "count(/A/*//E/@name/string())" will probably >> run without getting stuck. >> >> >> >> > >> > This query, was going OOM, within few mins. >> > >> > I tried a few ways, of whitelisting, with contain clause, to truncate >> the >> > result set. That didn't help too. So, now I am out of ideas. This is >> giving >> > JVM 10GB of dedicated memory. >> > >> > Once, above query works and doesn't go Out Of Memory, I also need >> > corresponding file names too: >> > >> > XYZ.xml //E/@name >> > PQR.xml //E/@name >> > >> > Let me know if you would need more details, to appreciate the issue ? >> > - Mansi >> > >> > On Thu, Nov 6, 2014 at 8:48 AM, Christian Grün < >> christian.gruen at gmail.com> >> > wrote: >> >> >> >> Hi Mansi, >> >> >> >> I think we need more information on the queries that are causing the >> >> problems. >> >> >> >> Best, >> >> Christian >> >> >> >> >> >> >> >> On Wed, Nov 5, 2014 at 8:48 PM, Mansi Sheth <mansi.sheth at gmail.com> >> wrote: >> >> > Hello, >> >> > >> >> > I have a use case, where I have to extract lots in information from >> each >> >> > XML >> >> > in each DB. Something like, attribute values of most of the nodes in >> an >> >> > XML. >> >> > For such, queries based goes Out Of Memory with below exception. I am >> >> > giving >> >> > it ~12GB of RAM on i7 processor. Well I can't complain here since I >> am >> >> > most >> >> > definitely asking for loads of data, but is there any way I can get >> >> > these >> >> > kinds of data successfully ? >> >> > >> >> > mansi-veracode:BigData mansiadmin$ ~/Downloads/basex/bin/basexhttp >> >> > BaseX 8.0 beta b45c1e2 [Server] >> >> > Server was started (port: 1984) >> >> > HTTP Server was started (port: 8984) >> >> > Exception in thread "qtp2068921630-18" java.lang.OutOfMemoryError: >> Java >> >> > heap >> >> > space >> >> > at >> >> > >> >> > >> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.addConditionWaiter(AbstractQueuedSynchronizer.java:1857) >> >> > at >> >> > >> >> > >> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2073) >> >> > at >> >> > >> >> > >> org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) >> >> > at >> >> > >> >> > >> org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) >> >> > at >> >> > >> >> > >> org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) >> >> > at >> >> > >> >> > >> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) >> >> > at java.lang.Thread.run(Thread.java:744) >> >> > >> >> > >> >> > -- >> >> > - Mansi >> > >> > >> > >> > >> > -- >> > - Mansi >> >> >> >> >> >> -- >> >> - Mansi >> > > > > -- > - Mansi >

On Tue, Dec 30, 2014 at 2:38 PM, Mansi Sheth <mansi.sheth at gmail.com> wrote: > Hello, > > Wanted to get back to this email chain and share my experience. > > I got this running beautifully (including all post processing of results), > using the below command: > > curl -ig ' > http://localhost:8984/rest?run=get_query.xq&n=/Archives/*/descendant::D/@name/string()' > | cut -d: -f1 | cut -d. -f1-3 | sort | uniq -c | sort -n -r > > I am using Basex 8.0 beta 763cc93 build. Running this on i7 2.7GHZ MBP, > giving 8GB to basexhttp process. it took around 34 min on a 41 GB data. I > think, lot of time went in post processing (sorting) the result set, rather > than actually extracting the results from BaseX DB. > > When tried a similar query on a much smaller database(3GB) on a much > powerful amazon instance, giving 20GB RAM to basex http process, got me > results with post processing within 4 mins. > > Thanks for all your inputs guys, > > Keep BaseXing... !!! > - Mansi > > On Fri, Nov 7, 2014 at 12:25 PM, Mansi Sheth <mansi.sheth at gmail.com> > wrote: > >> This email chain, is extremely helpful. Thanks a ton guys. Certainly one >> of the most helpful folks here :) >> >> I have to try a lot of these suggestions but currently I am being pulled >> into something else, so I have to pause for the time being. >> >> Will get back to this email thread, after trying a few things and my >> relevant observations. >> >> - Mansi >> >> On Fri, Nov 7, 2014 at 3:48 AM, Fabrice Etanchaud <fetanchaud at questel.com >> > wrote: >> >>> Hi Mansi, >>> >>> >>> >>> From what I can see, >>> >>> for each pqr value, you could use db:attribute-range to retrieve all the >>> file names, group by/count to obtain statistics. >>> >>> You could also create a new collection from an extraction of only the >>> data you need, changing @name into element and use full text fuzzy match. >>> >>> >>> >>> Hoping it helps >>> >>> >>> >>> Cordialement >>> >>> Fabrice >>> >>> >>> >>> *De :* basex-talk-bounces at mailman.uni-konstanz.de [mailto: >>> basex-talk-bounces at mailman.uni-konstanz.de] *De la part de* Mansi Sheth >>> *Envoyé :* jeudi 6 novembre 2014 20:55 >>> *À :* Christian Grün >>> *Cc :* BaseX >>> *Objet :* Re: [basex-talk] Out Of Memory >>> >>> >>> >>> I would be doing tons of post processing. I never use UI. I either use >>> REST thru cURL or command line. >>> >>> >>> >>> I would basically need data in below format: >>> >>> >>> >>> XML File Name, @name >>> >>> >>> >>> I am trying to whitelist picking up values for only >>> "starts-with(@name,"pqr"). where "pqr" is a list of 150 odd values. >>> >>> >>> >>> My file names, are essentially some ID/keys, which I would need to map >>> it further using sqlite to some values and may be group by it.. etc. >>> >>> >>> >>> So, basically I am trying to visualize some data, based on its existence >>> in which xml files. So, yes count(<query>) would be fine, but won't solve >>> much purpose, since I still need value "pqr". >>> >>> >>> >>> - Mansi >>> >>> >>> >>> >>> >>> On Thu, Nov 6, 2014 at 11:19 AM, Christian Grün < >>> christian.gruen at gmail.com> wrote: >>> >>> > Query: /A/*//E/@name/string() >>> >>> In the GUI, all results will be cached, so you could think about >>> switching to command line. >>> >>> Do you really need to output all results, or do you do some further >>> processing with the intermediate results? >>> >>> For example, the query "count(/A/*//E/@name/string())" will probably >>> run without getting stuck. >>> >>> >>> >>> > >>> > This query, was going OOM, within few mins. >>> > >>> > I tried a few ways, of whitelisting, with contain clause, to truncate >>> the >>> > result set. That didn't help too. So, now I am out of ideas. This is >>> giving >>> > JVM 10GB of dedicated memory. >>> > >>> > Once, above query works and doesn't go Out Of Memory, I also need >>> > corresponding file names too: >>> > >>> > XYZ.xml //E/@name >>> > PQR.xml //E/@name >>> > >>> > Let me know if you would need more details, to appreciate the issue ? >>> > - Mansi >>> > >>> > On Thu, Nov 6, 2014 at 8:48 AM, Christian Grün < >>> christian.gruen at gmail.com> >>> > wrote: >>> >> >>> >> Hi Mansi, >>> >> >>> >> I think we need more information on the queries that are causing the >>> >> problems. >>> >> >>> >> Best, >>> >> Christian >>> >> >>> >> >>> >> >>> >> On Wed, Nov 5, 2014 at 8:48 PM, Mansi Sheth <mansi.sheth at gmail.com> >>> wrote: >>> >> > Hello, >>> >> > >>> >> > I have a use case, where I have to extract lots in information from >>> each >>> >> > XML >>> >> > in each DB. Something like, attribute values of most of the nodes >>> in an >>> >> > XML. >>> >> > For such, queries based goes Out Of Memory with below exception. I >>> am >>> >> > giving >>> >> > it ~12GB of RAM on i7 processor. Well I can't complain here since I >>> am >>> >> > most >>> >> > definitely asking for loads of data, but is there any way I can get >>> >> > these >>> >> > kinds of data successfully ? >>> >> > >>> >> > mansi-veracode:BigData mansiadmin$ ~/Downloads/basex/bin/basexhttp >>> >> > BaseX 8.0 beta b45c1e2 [Server] >>> >> > Server was started (port: 1984) >>> >> > HTTP Server was started (port: 8984) >>> >> > Exception in thread "qtp2068921630-18" java.lang.OutOfMemoryError: >>> Java >>> >> > heap >>> >> > space >>> >> > at >>> >> > >>> >> > >>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.addConditionWaiter(AbstractQueuedSynchronizer.java:1857) >>> >> > at >>> >> > >>> >> > >>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2073) >>> >> > at >>> >> > >>> >> > >>> org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) >>> >> > at >>> >> > >>> >> > >>> org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) >>> >> > at >>> >> > >>> >> > >>> org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) >>> >> > at >>> >> > >>> >> > >>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) >>> >> > at java.lang.Thread.run(Thread.java:744) >>> >> > >>> >> > >>> >> > -- >>> >> > - Mansi >>> > >>> > >>> > >>> > >>> > -- >>> > - Mansi >>> >>> >>> >>> >>> >>> -- >>> >>> - Mansi >>> >> >> >> >> -- >> - Mansi >> > > > > -- > - Mansi >

> curl -ig > 'http://localhost:8984/rest?run=get_query.xq&n=/Archives/*/descendant::D/@name/string()' > | cut -d: -f1 | cut -d. -f1-3 | sort | uniq -c | sort -n -r

> > I am using Basex 8.0 beta 763cc93 build. Running this on i7 2.7GHZ MBP, > giving 8GB to basexhttp process. it took around 34 min on a 41 GB data. I > think, lot of time went in post processing (sorting) the result set, rather > than actually extracting the results from BaseX DB. > > When tried a similar query on a much smaller database(3GB) on a much > powerful amazon instance, giving 20GB RAM to basex http process, got me > results with post processing within 4 mins. > > Thanks for all your inputs guys, > > Keep BaseXing... !!! > - Mansi > > On Fri, Nov 7, 2014 at 12:25 PM, Mansi Sheth <mansi.sheth at gmail.com> wrote: >> >> This email chain, is extremely helpful. Thanks a ton guys. Certainly one >> of the most helpful folks here :) >> >> I have to try a lot of these suggestions but currently I am being pulled >> into something else, so I have to pause for the time being. >> >> Will get back to this email thread, after trying a few things and my >> relevant observations. >> >> - Mansi >> >> On Fri, Nov 7, 2014 at 3:48 AM, Fabrice Etanchaud <fetanchaud at questel.com> >> wrote: >>> >>> Hi Mansi, >>> >>> >>> >>> From what I can see, >>> >>> for each pqr value, you could use db:attribute-range to retrieve all the >>> file names, group by/count to obtain statistics. >>> >>> You could also create a new collection from an extraction of only the >>> data you need, changing @name into element and use full text fuzzy match. >>> >>> >>> >>> Hoping it helps >>> >>> >>> >>> Cordialement >>> >>> Fabrice >>> >>> >>> >>> De : basex-talk-bounces at mailman.uni-konstanz.de >>> [mailto:basex-talk-bounces at mailman.uni-konstanz.de] De la part de Mansi >>> Sheth >>> Envoyé : jeudi 6 novembre 2014 20:55 >>> À : Christian Grün >>> Cc : BaseX >>> Objet : Re: [basex-talk] Out Of Memory >>> >>> >>> >>> I would be doing tons of post processing. I never use UI. I either use >>> REST thru cURL or command line. >>> >>> >>> >>> I would basically need data in below format: >>> >>> >>> >>> XML File Name, @name >>> >>> >>> >>> I am trying to whitelist picking up values for only >>> "starts-with(@name,"pqr"). where "pqr" is a list of 150 odd values. >>> >>> >>> >>> My file names, are essentially some ID/keys, which I would need to map it >>> further using sqlite to some values and may be group by it.. etc. >>> >>> >>> >>> So, basically I am trying to visualize some data, based on its existence >>> in which xml files. So, yes count(<query>) would be fine, but won't solve >>> much purpose, since I still need value "pqr". >>> >>> >>> >>> - Mansi >>> >>> >>> >>> >>> >>> On Thu, Nov 6, 2014 at 11:19 AM, Christian Grün >>> <christian.gruen at gmail.com> wrote: >>> >>> > Query: /A/*//E/@name/string() >>> >>> In the GUI, all results will be cached, so you could think about >>> switching to command line. >>> >>> Do you really need to output all results, or do you do some further >>> processing with the intermediate results? >>> >>> For example, the query "count(/A/*//E/@name/string())" will probably >>> run without getting stuck. >>> >>> >>> >>> > >>> > This query, was going OOM, within few mins. >>> > >>> > I tried a few ways, of whitelisting, with contain clause, to truncate >>> > the >>> > result set. That didn't help too. So, now I am out of ideas. This is >>> > giving >>> > JVM 10GB of dedicated memory. >>> > >>> > Once, above query works and doesn't go Out Of Memory, I also need >>> > corresponding file names too: >>> > >>> > XYZ.xml //E/@name >>> > PQR.xml //E/@name >>> > >>> > Let me know if you would need more details, to appreciate the issue ? >>> > - Mansi >>> > >>> > On Thu, Nov 6, 2014 at 8:48 AM, Christian Grün >>> > <christian.gruen at gmail.com> >>> > wrote: >>> >> >>> >> Hi Mansi, >>> >> >>> >> I think we need more information on the queries that are causing the >>> >> problems. >>> >> >>> >> Best, >>> >> Christian >>> >> >>> >> >>> >> >>> >> On Wed, Nov 5, 2014 at 8:48 PM, Mansi Sheth <mansi.sheth at gmail.com> >>> >> wrote: >>> >> > Hello, >>> >> > >>> >> > I have a use case, where I have to extract lots in information from >>> >> > each >>> >> > XML >>> >> > in each DB. Something like, attribute values of most of the nodes in >>> >> > an >>> >> > XML. >>> >> > For such, queries based goes Out Of Memory with below exception. I >>> >> > am >>> >> > giving >>> >> > it ~12GB of RAM on i7 processor. Well I can't complain here since I >>> >> > am >>> >> > most >>> >> > definitely asking for loads of data, but is there any way I can get >>> >> > these >>> >> > kinds of data successfully ? >>> >> > >>> >> > mansi-veracode:BigData mansiadmin$ ~/Downloads/basex/bin/basexhttp >>> >> > BaseX 8.0 beta b45c1e2 [Server] >>> >> > Server was started (port: 1984) >>> >> > HTTP Server was started (port: 8984) >>> >> > Exception in thread "qtp2068921630-18" java.lang.OutOfMemoryError: >>> >> > Java >>> >> > heap >>> >> > space >>> >> > at >>> >> > >>> >> > >>> >> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.addConditionWaiter(AbstractQueuedSynchronizer.java:1857) >>> >> > at >>> >> > >>> >> > >>> >> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2073) >>> >> > at >>> >> > >>> >> > >>> >> > org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) >>> >> > at >>> >> > >>> >> > >>> >> > org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) >>> >> > at >>> >> > >>> >> > >>> >> > org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) >>> >> > at >>> >> > >>> >> > >>> >> > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) >>> >> > at java.lang.Thread.run(Thread.java:744) >>> >> > >>> >> > >>> >> > -- >>> >> > - Mansi >>> > >>> > >>> > >>> > >>> > -- >>> > - Mansi >>> >>> >>> >>> >>> >>> -- >>> >>> - Mansi >> >> >> >> >> -- >> - Mansi > > > > > -- > - Mansi