Multi threading doing slow process of Xquery Execution
Wednesday, 13 November 2013 · 5 messages
PS
Pushpendra Singh Sengar13 Nov 2013 07:38 #
Hi All,
I create a collection of 4.5 gb xml document.
and process some Xquery in sequencial manner then it is taking adequate time to process all queries.
But when I converting process to multi threading then it is taking more than 10 times of time.
Please someone suggest what is the best way to process Xquey in multithreading model.
MY code is following :
Thread thread = new Thread(new XqueryProcessor( new Context(),"db1"));
Thread thread1 = new Thread(new XqueryProcessor( new Context(),"db2"));
Thread thread2 = new Thread(new XqueryProcessor( new Context(),"db3"));
thread.start();
thread1.start();
thread2.start();Xquery Processor has all queries which need to process.
and db1, db2 ,db3 are basically same xml files three collection.
--
Thanks & Regards ,
Pushpendra Singh
JE
Jens Erat13 Nov 2013 09:47 #
Hi Pushpendra Singh,
more information on [transaction management] can be found in the documentation. Especially have a look at the chapter Limitations. Are you running any updates? If you have problems with your specific use case, please give us more information on the queries used.
Kind regards from Lake Constance, Germany,
Jens Erat
[transaction management]: http://docs.basex.org/wiki/Transaction_Management
--
Jens Erat
[phone]: tel:+49-151-56961126
[mail]: mailto:email at jenserat.de
[jabber]: xmpp:jabber at jenserat.de
[web]: http://www.jenserat.de
PGP: 350E D9B6 9ADC 2DED F5F2 8549 CBC2 613C D745 722B
Quoted text · Pushpendra Singh Sengar · 27 lines
Am 13.11.2013 um 07:38 schrieb Pushpendra Singh Sengar <pushpendra1412 at gmail.com>: > Hi All, > I create a collection of 4.5 gb xml document. > and process some Xquery in sequencial manner then it is taking adequate time to process all queries. > But when I converting process to multi threading then it is taking more than 10 times of time. > Please someone suggest what is the best way to process Xquey in multithreading model. > > MY code is following : > Thread thread = new Thread(new XqueryProcessor( new Context(),"db1")); > Thread thread1 = new Thread(new XqueryProcessor( new Context(),"db2")); > Thread thread2 = new Thread(new XqueryProcessor( new Context(),"db3")); > > thread.start(); > thread1.start(); > thread2.start(); > > Xquery Processor has all queries which need to process. > and db1, db2 ,db3 are basically same xml files three collection. > > -- > Thanks & Regards , > Pushpendra Singh > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
CG
Hi Pushpendra,
first of all, it’s highly advisable to only create one instance of the Context class and pass it on as reference. Regarding the slow down, I assume that your queries read a lot of data sequentially, and the concurrent query execution leads to competing disk operations, leading to random access patterns. This problem can to some extent be alleviated by switching to SSDs, but a cheaper and more
straightforward alternative could be to optimize your queries first.
Best,
Christian
Quoted text · Pushpendra Singh Sengar · 33 lines
On Wed, Nov 13, 2013 at 7:38 AM, Pushpendra Singh Sengar <pushpendra1412 at gmail.com> wrote: > Hi All, > I create a collection of 4.5 gb xml document. > and process some Xquery in sequencial manner then it is taking adequate time > to process all queries. > But when I converting process to multi threading then it is taking more than > 10 times of time. > Please someone suggest what is the best way to process Xquey in > multithreading model. > > MY code is following : > Thread thread = new Thread(new XqueryProcessor( new Context(),"db1")); > Thread thread1 = new Thread(new XqueryProcessor( new Context(),"db2")); > Thread thread2 = new Thread(new XqueryProcessor( new Context(),"db3")); > > thread.start(); > thread1.start(); > thread2.start(); > > Xquery Processor has all queries which need to process. > and db1, db2 ,db3 are basically same xml files three collection. > > -- > Thanks & Regards , > Pushpendra Singh > > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >
PS
Pushpendra Singh Sengar13 Nov 2013 11:56 #
Hi Christian,
I tried it with single Context also still it is taking same time as with different context.
Queries written in optimized manner.Disk operation is also high taking 10 to 20 mb reading per second.
I will try with SSD.
Could you please guide me to sharing data between queries.
On Wed, Nov 13, 2013 at 4:11 PM, Christian Grün
<christian.gruen at gmail.com>wrote:
Quoted text · Christian Grün · 52 lines
> Hi Pushpendra, > > first of all, it’s highly advisable to only create one instance of the > Context class and pass it on as reference. Regarding the slow down, I > assume that your queries read a lot of data sequentially, and the > concurrent query execution leads to competing disk operations, leading > to random access patterns. This problem can to some extent be > alleviated by switching to SSDs, but a cheaper and more > straightforward alternative could be to optimize your queries first. > > Best, > Christian > ___________________________ > > On Wed, Nov 13, 2013 at 7:38 AM, Pushpendra Singh Sengar > <pushpendra1412 at gmail.com> wrote: > > Hi All, > > I create a collection of 4.5 gb xml document. > > and process some Xquery in sequencial manner then it is taking adequate > time > > to process all queries. > > But when I converting process to multi threading then it is taking more > than > > 10 times of time. > > Please someone suggest what is the best way to process Xquey in > > multithreading model. > > > > MY code is following : > > Thread thread = new Thread(new XqueryProcessor( new Context(),"db1")); > > Thread thread1 = new Thread(new XqueryProcessor( new > Context(),"db2")); > > Thread thread2 = new Thread(new XqueryProcessor( new > Context(),"db3")); > > > > thread.start(); > > thread1.start(); > > thread2.start(); > > > > Xquery Processor has all queries which need to process. > > and db1, db2 ,db3 are basically same xml files three collection. > > > > -- > > Thanks & Regards , > > Pushpendra Singh > > > > > > _______________________________________________ > > BaseX-Talk mailing list > > BaseX-Talk at mailman.uni-konstanz.de > > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > > >
--
Thanks & Regards ,
Pushpendra Singh
CG
Christian Grün13 Nov 2013 12:05 #
Quoted text · Pushpendra Singh Sengar · 2 lines
> I tried it with single Context also still it is taking same time as with > different context.
Well, this is not about performance. Several context instances may lead to the corruption of your data if you e.g. decide to perform write operations. The JavaDoc may give you some hints [1]. Beside that, the Wiki article on transactions tell you how concurrent operations are handled by BaseX [2].
Quoted text · Pushpendra Singh Sengar · 2 lines
> Queries written in optimized manner.Disk operation is also high taking 10 to > 20 mb reading per second.
So this means that all the data you are scanning really needs to be looked at? And that there is no way to benefit from index structures?
Quoted text · Pushpendra Singh Sengar · 1 line
> Could you please guide me to sharing data between queries.
I’m not sure why you want to do that, or if it would really help. As XQuery is a rather complex language, you’d first need to define what parts of the data are relevant for all queries, etc. Maybe you could give us a minimized query example instead, which shows us what your queries do?
Christian
[1] http://docs.basex.org/javadoc/org/basex/core/Context.html [2] http://docs.basex.org/wiki/Transaction_Management
Written for BaseX 7.7. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery