Main Page » Mailing List » March 2014

multi-threaded XQueries

Sunday, 23 March 2014 · 6 messages · last reply 25 Mar 2014
YS
Ying Shanshan23 Mar 2014 15:19 #
Hi All, I created one database of 1GB in size, and have a batch of *read-only*queries to be performed. Each query is quite simple, so there is not much room to improve. All the queries are executed through *XQJ api.* If these queries are sequentially executed, it costs 5~8 seconds per query. To speed it up, I tried to executed it in a multi-thread way. But turns out to be much slower than the sequential one.
Quoted text · 1 line
This is what I wrote for each thread:
public void run() {
try {
 XQConnection conn = ds.getConnection();
XQSequence rs = conn.prepareExpression(input)
.executeQuery();
 //process the query result
conn.close();
}
 } catch (Throwable e) {
e.printStackTrace();
}
 }
Anyone has the idea on how to improve the batch of queries. Thanks,
-- Regards Shanshan School of Computing National University of Singapore
CG
Christian Grün23 Mar 2014 19:41 #
Hi Shanshan, thanks for writing to the list.
Quoted text · Ying Shanshan · 3 lines
> I created one database of 1GB in size, and have a batch of read-only queries > to be performed. > Each query is quite simple, so there is not much room to improve.
Does your query return large results? Could you please give us an example of one of your queries?
Quoted text · Ying Shanshan · 2 lines
> All the queries are executed through XQJ api. > If these queries are sequentially executed, it costs 5~8 seconds per query.
This sounds pretty slow. Have you compared the performance with the time required by the BaseX GUI or the command-line?
Quoted text · Ying Shanshan · 2 lines
> To speed it up, I tried to executed it in a multi-thread way. But turns out > to be much slower than the sequential one.
This could be due to the fact that concurrent read operations lead to random disk access patterns. I assume that your queries lead to much disk I/O? Christian
CG
Christian Grün24 Mar 2014 10:06 #
Hi Ying,
Quoted text · Christian Grün · 5 lines
> e.g. the dataset I use is DBLP, one of the example query is > > for $x in db:open('dblp_2013')/dblp/article[child::pages and child::title] > let $y := count($x/author) > return concat($y, "/t", db:node-id($x))
Yes, I agree there is not much that can be done to speed up this query.
Quoted text · Christian Grün · 2 lines
> It is true that these queries are competing for the same resource. > So you mean that each query causes a lot of disk I/O?
Exactly. All DBLP articles need to be parsed by this query, and > 1 million result strings will be generated, resulting in a query time of appr. 0.005 ms per result. Hope this helps, Christian
Quoted text · Christian Grün · 8 lines
> However all these queries are read-only. It there any room to improve this? > -- > Regards > > Shanshan > > School of Computing > National University of Singapore
YS
Ying Shanshan24 Mar 2014 14:46 #
Hi Christian, Thanks for your reply. First, I am a layman of XML DB, do not have much experience of it. I am just very curious why for read-only queries, they will compete for disk I/O. As in relational database management systems, we may control the granularity of lock. Is it possible to allow multiple read thread on the same I/O, for read only queries. It there anyway we can brutely load the whole index and then do the queries. Thanks, On Mon, Mar 24, 2014 at 5:06 PM, Christian Grün <christian.gruen at gmail.com>wrote:
Quoted text · Christian Grün · 34 lines
> Hi Ying, > > > e.g. the dataset I use is DBLP, one of the example query is > > > > for $x in db:open('dblp_2013')/dblp/article[child::pages and > child::title] > > let $y := count($x/author) > > return concat($y, "/t", db:node-id($x)) > > Yes, I agree there is not much that can be done to speed up this query. > > > It is true that these queries are competing for the same resource. > > So you mean that each query causes a lot of disk I/O? > > Exactly. All DBLP articles need to be parsed by this query, and > 1 > million result strings will be generated, resulting in a query time of > appr. 0.005 ms per result. > > Hope this helps, > Christian > > > > > However all these queries are read-only. It there any room to improve > this? > > > -- > > Regards > > > > Shanshan > > > > School of Computing > > National University of Singapore >
-- Regards Shanshan School of Computing National University of Singapore
CG
Christian Grün24 Mar 2014 15:34 #
Hi Ying,
Quoted text · Ying Shanshan · 2 lines
> I am just very curious why for read-only queries, they will compete for disk > I/O.
The reason is that your query requires a sequential scan of the full XML structure. If you run several requests in parallel, all requests will try to read the same blocks in a slightly delayed order. As a result, the pointer, which is reading a block on disk, will be moved again and again, and the resulting pattern is somewhat random.
Quoted text · Ying Shanshan · 1 line
> It there anyway we can brutely load the whole index and then do the queries.
Due to the nature of your query, there is now way to benefit from index structures. Instead, I would suggest to create additional databases that contain all the information you will be frequently accessing. Christian
YS
Ying Shanshan25 Mar 2014 02:02 #
Hi Christian, Thanks for your response. On Mon, Mar 24, 2014 at 10:34 PM, Christian Grün <christian.gruen at gmail.com>wrote:
Quoted text · Christian Grün · 22 lines
> Hi Ying, > > > I am just very curious why for read-only queries, they will compete for > disk > > I/O. > > The reason is that your query requires a sequential scan of the full > XML structure. If you run several requests in parallel, all requests > will try to read the same blocks in a slightly delayed order. As a > result, the pointer, which is reading a block on disk, will be moved > again and again, and the resulting pattern is somewhat random. > > > It there anyway we can brutely load the whole index and then do the > queries. > > Due to the nature of your query, there is now way to benefit from > index structures. Instead, I would suggest to create additional > databases that contain all the information you will be frequently > accessing. > > Christian >
-- Regards Shanshan School of Computing National University of Singapore

Written for BaseX 7.8. BaseX may have changed since: please check the current documentation.

« Confusing error using GROUP BYXML clob w/ SQL module »
⚡Generated with XQuery