Distributing queries to several on several processors
Tuesday, 21 April 2015 · 7 messages · last reply 22 Apr 2015
GH
Goetz Heller21 Apr 2015 20:23 #
So far I did not find any information on how BaseX can be advised how to use computing resources. The use case here is as follows:
I get several megabytes of xml files each day, usually between 50 and 100 MB. These are organized in one database per day. Since most queries run on a daily base this works perfectly fine. However, there are situations when I need to run a query over a larger time span, say three or six months. (Note that I'm speaking of read-only queries here, not of transactions.) Of course I can do this in a loop (for $db in $db-list) but since the data in each database is completely independent from that in the other databases it would make perfect sense to parallelize the query. Is there a way to achieve this using xQuery? I'm aware of the possibility to split the sequence into several ones and run them in different threads on different connections using Java, for instance. But even then I still don't know what the server does (my queries run in a client-server configuration): will it occupy just one processor, or will it distribute the workload?
CG
Christian Grün22 Apr 2015 10:28 #
Hi Götz,
Quoted text · Goetz Heller · 3 lines
> it would > make perfect sense to parallelize the query. Is there a way to achieve this > using xQuery?
Our initial attempts to integrate low-level support for
parallelization in XQuery turned out not to be as successful as we hoped they would be. One reason for that is that you can basically do everything with XQuery, and it's pretty hard to detect patterns in the code that are simple enough to be parallelized. Next to that, Java does not give us enough facilities to control CPU caching behavior.
As you already indicated, you can simply run multiple queries in parallel by e.g. using Java threads or the BaseX client/server architecture (which by default allows 8 transactions in parallel [1]). If your queries do a lot of I/O, you will often get better performance by only allowing one transaction at a time, though. This is due to the random access patterns on your external drives (and in my experience, it also applies to SSDs). However, if you work with main-memory instances of databases, parallelization might give you some
performance gains (albeit not as big as you might expect).
Hope this helps,
Christian
[1] http://docs.basex.org/wiki/Options#PARALLEL
CG
Any volunteers out there? ;)
Quoted text · Erol Akarsu · 44 lines
On Wed, Apr 22, 2015 at 11:05 AM, Erol Akarsu <eakarsu at gmail.com> wrote: > Christian, > > I think we should be able to attach BaseX to Apache spark. But integration > code need to be written. > Everybody is able to read from Hadoop,SOLR, ElasticSearch etc. to Spark and > process there. > Why not for BaseX? > > Erol Akarsu > > On Wed, Apr 22, 2015 at 4:28 AM, Christian Grün <christian.gruen at gmail.com> > wrote: >> >> Hi Götz, >> >> > it would >> > make perfect sense to parallelize the query. Is there a way to achieve >> > this >> > using xQuery? >> >> Our initial attempts to integrate low-level support for >> parallelization in XQuery turned out not to be as successful as we >> hoped they would be. One reason for that is that you can basically do >> everything with XQuery, and it's pretty hard to detect patterns in the >> code that are simple enough to be parallelized. Next to that, Java >> does not give us enough facilities to control CPU caching behavior. >> >> As you already indicated, you can simply run multiple queries in >> parallel by e.g. using Java threads or the BaseX client/server >> architecture (which by default allows 8 transactions in parallel [1]). >> If your queries do a lot of I/O, you will often get better performance >> by only allowing one transaction at a time, though. This is due to the >> random access patterns on your external drives (and in my experience, >> it also applies to SSDs). However, if you work with main-memory >> instances of databases, parallelization might give you some >> performance gains (albeit not as big as you might expect). >> >> Hope this helps, >> Christian >> >> [1] http://docs.basex.org/wiki/Options#PARALLEL > >
CG
Christian Grün22 Apr 2015 11:15 #
Hi Götz (cc @ basex-talk),
Quoted text · Christian Grün · 1 line
> OK, I think I understand. However, I think there should be some
possibilities to allow the user to give hints. In my opinion, FOR-loops would be first-class candidates to use parallel streams, in particular in the use case I described in my previous posting:
Quoted text · Christian Grün · 3 lines
> > FOR $var IN (collection) > PARALLEL RETURN (expression-list)
Makes sense, in general.. XQuery pragmas could be solution:
(# basex: parallel #) { ... }
Higher-order functions provide functions like hof:parallel-map(...).
However, it has many effects on the architecture of BaseX in terms of performance, because we'd need to create new contexts for each parallelized query, which takes additional time. See the following query as example:
$x[. = "123"]The dot applies to the "current context item". If we parallelize a query, we'd have multiple current context items. The same
multiplication would apply to the stack frame and other runtime variables, and the time lost for duplicating these instances is in most cases more expensive than doing stuff in a single thread.
At least that's our experience so far. Once again, we are happy to see people jump into our code and show us that it can be done better..
Christian
GH
Goetz Heller22 Apr 2015 11:41 #
OK. Let me do my stuff first. Then I will see if I'm able to dive deep enough into the BaseX code to come up with some meaningful contribution!
Kind regards,
Goetz
Quoted text · Christian Grün · 18 lines
-----Ursprüngliche Nachricht----- Von: Christian Grün [mailto:christian.gruen at gmail.com] Gesendet: Mittwoch, 22. April 2015 11:15 An: Goetz Heller; BaseX Betreff: Re: Distributing queries to several on several processors Hi Götz (cc @ basex-talk), > OK, I think I understand. However, I think there should be some possibilities to allow the user to give hints. In my opinion, FOR-loops would be first-class candidates to use parallel streams, in particular in the use case I described in my previous posting: > > FOR $var IN (collection) > PARALLEL RETURN (expression-list) Makes sense, in general.. XQuery pragmas could be solution: (# basex: parallel #) { ... } Higher-order functions provide functions like hof:parallel-map(...). However, it has many effects on the architecture of BaseX in terms of performance, because we'd need to create new contexts for each parallelized query, which takes additional time. See the following query as example: $x[. = "123"] The dot applies to the "current context item". If we parallelize a query, we'd have multiple current context items. The same multiplication would apply to the stack frame and other runtime variables, and the time lost for duplicating these instances is in most cases more expensive than doing stuff in a single thread. At least that's our experience so far. Once again, we are happy to see people jump into our code and show us that it can be done better.. Christian
AB
Hi Erol,
I am not volunteering :-) but if somebody wants to take this route this code might give some pointers [1].
It uses Apache Spark to run Saxon-HE, an XQuery example [2], and more info [3].
/Andy
[1] https://github.com/elsevierlabs/spark-xml-utils
[2] https://github.com/elsevierlabs/spark-xml-utils/wiki/xquery [3]
http://mail-archives.apache.org/mod_mbox/spark-user/201408.mbox/%3C1407936616.34624.YahooMailNeo at web141003.mail.bf1.yahoo.com%3E
Quoted text · Erol Akarsu · 45 lines
On 22 April 2015 at 10:05, Erol Akarsu <eakarsu at gmail.com> wrote: > Christian, > > I think we should be able to attach BaseX to Apache spark. But integration > code need to be written. > Everybody is able to read from Hadoop,SOLR, ElasticSearch etc. to Spark > and process there. > Why not for BaseX? > > Erol Akarsu > > On Wed, Apr 22, 2015 at 4:28 AM, Christian Grün <christian.gruen at gmail.com > > wrote: > >> Hi Götz, >> >> > it would >> > make perfect sense to parallelize the query. Is there a way to achieve >> this >> > using xQuery? >> >> Our initial attempts to integrate low-level support for >> parallelization in XQuery turned out not to be as successful as we >> hoped they would be. One reason for that is that you can basically do >> everything with XQuery, and it's pretty hard to detect patterns in the >> code that are simple enough to be parallelized. Next to that, Java >> does not give us enough facilities to control CPU caching behavior. >> >> As you already indicated, you can simply run multiple queries in >> parallel by e.g. using Java threads or the BaseX client/server >> architecture (which by default allows 8 transactions in parallel [1]). >> If your queries do a lot of I/O, you will often get better performance >> by only allowing one transaction at a time, though. This is due to the >> random access patterns on your external drives (and in my experience, >> it also applies to SSDs). However, if you work with main-memory >> instances of databases, parallelization might give you some >> performance gains (albeit not as big as you might expect). >> >> Hope this helps, >> Christian >> >> [1] http://docs.basex.org/wiki/Options#PARALLEL >> > >
EA
Christian,
I think we should be able to attach BaseX to Apache spark. But integration code need to be written.
Everybody is able to read from Hadoop,SOLR, ElasticSearch etc. to Spark and process there.
Why not for BaseX?
Erol Akarsu
Quoted text · Christian Grün · 31 lines
On Wed, Apr 22, 2015 at 4:28 AM, Christian Grün <christian.gruen at gmail.com> wrote: > Hi Götz, > > > it would > > make perfect sense to parallelize the query. Is there a way to achieve > this > > using xQuery? > > Our initial attempts to integrate low-level support for > parallelization in XQuery turned out not to be as successful as we > hoped they would be. One reason for that is that you can basically do > everything with XQuery, and it's pretty hard to detect patterns in the > code that are simple enough to be parallelized. Next to that, Java > does not give us enough facilities to control CPU caching behavior. > > As you already indicated, you can simply run multiple queries in > parallel by e.g. using Java threads or the BaseX client/server > architecture (which by default allows 8 transactions in parallel [1]). > If your queries do a lot of I/O, you will often get better performance > by only allowing one transaction at a time, though. This is due to the > random access patterns on your external drives (and in my experience, > it also applies to SSDs). However, if you work with main-memory > instances of databases, parallelization might give you some > performance gains (albeit not as big as you might expect). > > Hope this helps, > Christian > > [1] http://docs.basex.org/wiki/Options#PARALLEL >
Written for BaseX 8.1. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery