Main Page » Mailing List » June 2015

Performing queries with a local function takes too much time

Wednesday, 17 June 2015 · 5 messages · last reply 20 Jun 2015
SC
serdar ceran17 Jun 2015 11:18 #
Hi, BaseX team. I have a problem; Performing queries with a local function takes too much time to get the result. Please have a look at the Stackoverflow <http://stackoverflow.com/questions/30865291/basex-query-optimization-on-join> for more details. Best regards; Serdar CERAN
CG
Christian Grün17 Jun 2015 11:57 #
Hi Serdar, Thanks for the hint. We have added a new GitHub issue to remember your request. Best, Christian [1] https://github.com/BaseXdb/basex/issues/1160
Quoted text · serdar ceran · 13 lines
On Wed, Jun 17, 2015 at 11:18 AM, serdar ceran <bsceran at gmail.com> wrote: > Hi, BaseX team. I have a problem; Performing queries with a local function > takes too much time to get the result. > > Please have a look at the Stackoverflow for more details. > > > Best regards; > > > Serdar CERAN > >
CG
Christian Grün17 Jun 2015 14:15 #
Hi Serdar (cc to the list), Your query amounts to more than thousands of index requests, and this eventually makes the query faster than when being executed sequentially. You can enforce sequential execution for the last for clause by using "eq" instead of "=". Next, you should factor out the content id path; otherwise, it will be evaluated thousands of times as well. The following query will be executed a bit faster:
  let $PlCts := //PlCt[environment = 'AIR']/@id
  for $PlGeTy in //PlGeTy[isOfPlCt/@href = $PlCts]
  for $PlSpTy in //PlSpTy[isOfPlGeTy/@href = $PlGeTy/@id]
  let $ContId := $PlSpTy/isOfCont/@href/string()
  for $Cont in //Cont[@id eq $ContId]
  return <done>{ $PlGeTy, $PlSpTy, $Cont }</done>
Ideally, the query optimizer would detect that it's counterproductive to rewrite the last for clause to an index expression. However, decisions like this very much depend on the input data, and there are no simple heuristics for that. Hope this helps, Christian
Quoted text · serdar ceran · 54 lines
On Wed, Jun 17, 2015 at 1:39 PM, serdar ceran <bsceran at gmail.com> wrote: > Hi Christian, > > I have also another optimization issue on a query as I mentioned in Update > three section. > Could you help on this one? > > Thanks, > > Serdar > > > On Wed, Jun 17, 2015 at 12:57 PM, Christian Grün <christian.gruen at gmail.com> > wrote: >> >> Hi Serdar, >> >> Thanks for the hint. We have added a new GitHub issue to remember your >> request. >> >> Best, >> Christian >> >> [1] https://github.com/BaseXdb/basex/issues/1160 >> >> >> On Wed, Jun 17, 2015 at 11:18 AM, serdar ceran <bsceran at gmail.com> wrote: >> > Hi, BaseX team. I have a problem; Performing queries with a local >> > function >> > takes too much time to get the result. >> > >> > Please have a look at the Stackoverflow for more details. >> > >> > >> > Best regards; >> > >> > >> > Serdar CERAN >> > >> > > > > > > -- > Best regards; > > -- > B. Serdar CERAN > > Yaltes Company > Specialist Software Engineer > Tel: +905423993073 >
CG
Christian Grün17 Jun 2015 14:49 #
Hi Serdar, I forgot one more optimization: You can take advantage of maps in XQuery 3.1 and cache repeatedly accessed values in advance:
  let $Conts := map:merge(//Cont/map { @id/string(): . })
  let $PlCts := //PlCt[environment = 'AIR']/@id
  for $PlGeTy in //PlGeTy[isOfPlCt/@href = $PlCts]
  for $PlSpTy in //PlSpTy[isOfPlGeTy/@href = $PlGeTy/@id]
  for $Cont in $Conts($PlSpTy/isOfCont/@href)
  return <done>{ $PlGeTy, $PlSpTy, $Cont }</done>
If you are wondering what's going on in the first line of code.. Here is a more verbose writing:
  map:merge(
    for $Cont in //Cont
    return map { $Cont/@id/string(): $Cont }
  )
You can even take this further and create maps for all accessed ids in advance, as shown in the query below. However, please note that... * this may take a lot of memory if your document contains a large number of entries * it may get slower than the original query if you only a few of the cached entries * as you see below, you'll have to group entries if there is more than one entry with the same key. Maps in XQuery are a powerfull feature. It may take some time to get to know them, but it's really worth the trouble.. Have fun, Christian
  let $Conts := map:merge(//Cont/map { @id/string(): . })
  let $PlGeTys := map:merge(
    for $value in //PlGeTy
    group by $key := $value/isOfPlCt/@href/string()
    return map { $key: $value }
  )
  let $PlSpTys := map:merge(
    for $value in //PlSpTy
    group by $key := $value/isOfPlGeTy/@href/string()
    return map { $key: $value }
  )

  for $PlCts in //PlCt[environment = 'AIR']/@id
  for $PlGeTy in $PlGeTys($PlCts)
  for $PlSpTy in $PlSpTys($PlGeTy/@id)
  for $Cont in $Conts($PlSpTy/isOfCont/@href)
  return <done>{ $PlGeTy, $PlSpTy, $Cont }</done>
CG
Christian Grünin reply to serdar ceran, 17 Jun 2015, 11:1820 Jun 2015 09:58 #
Hi Serdard, This is just a notificiation that the initial performance flaw you reported in your Stackoverflow issue has been tackled in the latest snapshot [1,2]. Your feedback is welcome. Best, Christian [1] https://github.com/BaseXdb/basex/issues/1160 [2] http://files.basex.org/releases/latest/
Quoted text · serdar ceran · 13 lines
On Wed, Jun 17, 2015 at 11:18 AM, serdar ceran <bsceran at gmail.com> wrote: > Hi, BaseX team. I have a problem; Performing queries with a local function > takes too much time to get the result. > > Please have a look at the Stackoverflow for more details. > > > Best regards; > > > Serdar CERAN > >

Written for BaseX 8.2. BaseX may have changed since: please check the current documentation.

« Low/high caps...thinking outloud »
⚡Generated with XQuery