Main Page » Mailing List » July 2022

Possible to Speed Up This Lookup?

Friday, 15 July 2022 · 12 messages · last reply 16 Jul 2022
EK
Eliot Kimber15 Jul 2022 20:04 #
I generate a “where used” index as an XML doc in a separate database. I’m then doing lookups against this index like this where $node is the element I want to find the index entry for: collection($rkDatabase)/doc-where-used-index/where-used-entry[string(@key) eq string(db:node-id($node))] Where the markup is shown in [1] below. Using prof:track() I’m measuring a consistent 0.036 seconds per lookup, which seems very slow. Db:node-id() is instantaneous so the cost must be in the XPath itself. I’ve tried different variations on the predicate but nothing makes a measurable difference. There must be a more efficient way to do this kind of lookup or a better way to structure my index XML so that the lookup is optimized. Or is this something that could be affected by my index settings at the database level? Thanks, Eliot [1] Where-used index markup:
<doc-where-used-index>
<where-used-entry key="9151416" tagname="map" class="map/map" id="">
  <title>
    <ph keyref="workplace-safety-management"/>
  </title>
  <conrefs/>
  <topicrefs>
    <noderef node-id="672471" database="tokyo" tagname="mapref" baseuri="/tokyo/qa-sushma-sundareswaran.ditamap" href="workplace-service-delivery-core.ditamap"/>
    <noderef node-id="898684" database="tokyo" tagname="mapref" baseuri="/tokyo/qa-ankit-jaiswal.ditamap" href="workplace-service-delivery-core.ditamap"/>
    <noderef node-id="9524996" database="tokyo" tagname="mapref" baseuri="/tokyo/workplace-service-delivery-suite.ditamap" href="workplace-service-delivery-core.ditamap"/>
  </topicrefs>
  <doc>
    <noderef node-id="9151416" database="tokyo" tagname="map" baseuri="/tokyo/workplace-service-delivery-core.ditamap"/>
  </doc>
  <xrefs/>
  <referencing-maps>
    <noderef node-id="672452" database="tokyo" tagname="map" baseuri="/tokyo/qa-sushma-sundareswaran.ditamap"/>
    <noderef node-id="898667" database="tokyo" tagname="map" baseuri="/tokyo/qa-ankit-jaiswal.ditamap"/>
    <noderef node-id="9524980" database="tokyo" tagname="map" baseuri="/tokyo/workplace-service-delivery-suite.ditamap"/>
  </referencing-maps>
</where-used-entry>
…
</doc-where-used-index>
_____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>
EK
Eliot Kimberin reply to Liam R. E. Quin, 15 Jul 2022, 22:2215 Jul 2022 22:01 #
There are on the order of 50K index entry items. I think the attribute index is probably the answer—for some reason I assumed that would be indexed by default but I see that it is not looking at the database details. I’ll try enabling the index and see what happens. Cheers, E. _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>
Quoted text · Liam R. E. Quin · 21 lines
From: Liam R. E. Quin <liam at fromoldbooks.org> Date: Friday, July 15, 2022 at 1:22 PM To: Eliot Kimber <eliot.kimber at servicenow.com>, basex-talk at mailman.uni-konstanz.de <basex-talk at mailman.uni-konstanz.de> Subject: Re: [basex-talk] Possible to Speed Up This Lookup? [External Email] On Fri, 2022-07-15 at 18:04 +0000, Eliot Kimber wrote: > collection($rkDatabase)/doc-where-used-index/where-used- > entry[string(@key) eq string(db:node-id($node))] > > Where the markup is shown in [1] below. > > Using prof:track() I’m measuring a consistent 0.036 seconds per > lookup, could it be opening the collection that takes time? Or are there hundreds of thousands of where-used-entry elements, and, if so, do you have an attribute value index? -- Liam Quin, https://urldefense.com/v3/__https://www.delightfulcomputing.com/__;!!N4vogdjhuJM!C1fj636Ib2y0kENz2AwiUJZj6IyBT5mrb5YE_kSoMeqI0fFs95OpyWpldUalcS47Lg7Dt-1d0gNC7xyQXLy9rZQ$<https://urldefense.com/v3/__https:/www.delightfulcomputing.com/__;!!N4vogdjhuJM!C1fj636Ib2y0kENz2AwiUJZj6IyBT5mrb5YE_kSoMeqI0fFs95OpyWpldUalcS47Lg7Dt-1d0gNC7xyQXLy9rZQ$> Available for XML/Document/Information Architecture/XSLT/ XSL/XQuery/Web/Text Processing/A11Y training, work & consulting. Barefoot Web-slave, antique illustrations: https://urldefense.com/v3/__http://www.fromoldbooks.org__;!!N4vogdjhuJM!C1fj636Ib2y0kENz2AwiUJZj6IyBT5mrb5YE_kSoMeqI0fFs95OpyWpldUalcS47Lg7Dt-1d0gNC7xyQWGj0qnM$<https://urldefense.com/v3/__http:/www.fromoldbooks.org__;!!N4vogdjhuJM!C1fj636Ib2y0kENz2AwiUJZj6IyBT5mrb5YE_kSoMeqI0fFs95OpyWpldUalcS47Lg7Dt-1d0gNC7xyQWGj0qnM$>
LQ
Liam R. E. Quinin reply to Eliot Kimber, 15 Jul 2022, 20:0415 Jul 2022 22:22 #
Quoted text · Eliot Kimber · 8 lines
On Fri, 2022-07-15 at 18:04 +0000, Eliot Kimber wrote: > collection($rkDatabase)/doc-where-used-index/where-used- > entry[string(@key) eq string(db:node-id($node))] > > Where the markup is shown in [1] below. > > Using prof:track() I’m measuring a consistent 0.036 seconds per > lookup,
could it be opening the collection that takes time? Or are there hundreds of thousands of where-used-entry elements, and, if so, do you have an attribute value index?
-- Liam Quin, https://www.delightfulcomputing.com/ Available for XML/Document/Information Architecture/XSLT/ XSL/XQuery/Web/Text Processing/A11Y training, work & consulting. Barefoot Web-slave, antique illustrations: http://www.fromoldbooks.org
EK
Eliot Kimberin reply to Eliot Kimber, 15 Jul 2022, 22:0115 Jul 2022 22:39 #
Yes, turning on the attribute index definitely improves performa nce dramatically, which is no surprise. Not sure how I missed the need to expressly turn on these indexes. Doing that now. Cheers, E. _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>
Quoted text · Eliot Kimber · 36 lines
From: Eliot Kimber <eliot.kimber at servicenow.com> Date: Friday, July 15, 2022 at 3:01 PM To: Liam R. E. Quin <liam at fromoldbooks.org>, basex-talk at mailman.uni-konstanz.de <basex-talk at mailman.uni-konstanz.de> Subject: Re: [basex-talk] Possible to Speed Up This Lookup? There are on the order of 50K index entry items. I think the attribute index is probably the answer—for some reason I assumed that would be indexed by default but I see that it is not looking at the database details. I’ll try enabling the index and see what happens. Cheers, E. Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow> From: Liam R. E. Quin <liam at fromoldbooks.org> Date: Friday, July 15, 2022 at 1:22 PM To: Eliot Kimber <eliot.kimber at servicenow.com>, basex-talk at mailman.uni-konstanz.de <basex-talk at mailman.uni-konstanz.de> Subject: Re: [basex-talk] Possible to Speed Up This Lookup? [External Email] On Fri, 2022-07-15 at 18:04 +0000, Eliot Kimber wrote: > collection($rkDatabase)/doc-where-used-index/where-used- > entry[string(@key) eq string(db:node-id($node))] > > Where the markup is shown in [1] below. > > Using prof:track() I’m measuring a consistent 0.036 seconds per > lookup, could it be opening the collection that takes time? Or are there hundreds of thousands of where-used-entry elements, and, if so, do you have an attribute value index? -- Liam Quin, https://urldefense.com/v3/__https://www.delightfulcomputing.com/__;!!N4vogdjhuJM!C1fj636Ib2y0kENz2AwiUJZj6IyBT5mrb5YE_kSoMeqI0fFs95OpyWpldUalcS47Lg7Dt-1d0gNC7xyQXLy9rZQ$<https://urldefense.com/v3/__https:/www.delightfulcomputing.com/__;!!N4vogdjhuJM!C1fj636Ib2y0kENz2AwiUJZj6IyBT5mrb5YE_kSoMeqI0fFs95OpyWpldUalcS47Lg7Dt-1d0gNC7xyQXLy9rZQ$> Available for XML/Document/Information Architecture/XSLT/ XSL/XQuery/Web/Text Processing/A11Y training, work & consulting. Barefoot Web-slave, antique illustrations: https://urldefense.com/v3/__http://www.fromoldbooks.org__;!!N4vogdjhuJM!C1fj636Ib2y0kENz2AwiUJZj6IyBT5mrb5YE_kSoMeqI0fFs95OpyWpldUalcS47Lg7Dt-1d0gNC7xyQWGj0qnM$<https://urldefense.com/v3/__http:/www.fromoldbooks.org__;!!N4vogdjhuJM!C1fj636Ib2y0kENz2AwiUJZj6IyBT5mrb5YE_kSoMeqI0fFs95OpyWpldUalcS47Lg7Dt-1d0gNC7xyQWGj0qnM$>
EK
Eliot Kimber15 Jul 2022 23:20 #
Actually, I think I spoke too soon. This test:
return (db:option('attrindex'),
      for $node in $map
      return prof:track(db:open($rkDatabase)/doc-where-used-index/where-used-entry/@key[xs:integer(.) eq 9151416])?time ! util:formatTime(.)
Returns: true 00.030 seconds Where my current database context is the database named in $rkDatabase. A small improvement (0.036 to 0.030) but not dramatic. So I think I’m still missing some detail, probably a configuration issue. Cheers, E, _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>
Quoted text · Eliot Kimber · 49 lines
From: Eliot Kimber <eliot.kimber at servicenow.com> Date: Friday, July 15, 2022 at 3:39 PM To: Liam R. E. Quin <liam at fromoldbooks.org>, basex-talk at mailman.uni-konstanz.de <basex-talk at mailman.uni-konstanz.de> Subject: Re: [basex-talk] Possible to Speed Up This Lookup? Yes, turning on the attribute index definitely improves performa nce dramatically, which is no surprise. Not sure how I missed the need to expressly turn on these indexes. Doing that now. Cheers, E. Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow> From: Eliot Kimber <eliot.kimber at servicenow.com> Date: Friday, July 15, 2022 at 3:01 PM To: Liam R. E. Quin <liam at fromoldbooks.org>, basex-talk at mailman.uni-konstanz.de <basex-talk at mailman.uni-konstanz.de> Subject: Re: [basex-talk] Possible to Speed Up This Lookup? There are on the order of 50K index entry items. I think the attribute index is probably the answer—for some reason I assumed that would be indexed by default but I see that it is not looking at the database details. I’ll try enabling the index and see what happens. Cheers, E. Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow> From: Liam R. E. Quin <liam at fromoldbooks.org> Date: Friday, July 15, 2022 at 1:22 PM To: Eliot Kimber <eliot.kimber at servicenow.com>, basex-talk at mailman.uni-konstanz.de <basex-talk at mailman.uni-konstanz.de> Subject: Re: [basex-talk] Possible to Speed Up This Lookup? [External Email] On Fri, 2022-07-15 at 18:04 +0000, Eliot Kimber wrote: > collection($rkDatabase)/doc-where-used-index/where-used- > entry[string(@key) eq string(db:node-id($node))] > > Where the markup is shown in [1] below. > > Using prof:track() I’m measuring a consistent 0.036 seconds per > lookup, could it be opening the collection that takes time? Or are there hundreds of thousands of where-used-entry elements, and, if so, do you have an attribute value index? -- Liam Quin, https://urldefense.com/v3/__https://www.delightfulcomputing.com/__;!!N4vogdjhuJM!C1fj636Ib2y0kENz2AwiUJZj6IyBT5mrb5YE_kSoMeqI0fFs95OpyWpldUalcS47Lg7Dt-1d0gNC7xyQXLy9rZQ$<https://urldefense.com/v3/__https:/www.delightfulcomputing.com/__;!!N4vogdjhuJM!C1fj636Ib2y0kENz2AwiUJZj6IyBT5mrb5YE_kSoMeqI0fFs95OpyWpldUalcS47Lg7Dt-1d0gNC7xyQXLy9rZQ$> Available for XML/Document/Information Architecture/XSLT/ XSL/XQuery/Web/Text Processing/A11Y training, work & consulting. Barefoot Web-slave, antique illustrations: https://urldefense.com/v3/__http://www.fromoldbooks.org__;!!N4vogdjhuJM!C1fj636Ib2y0kENz2AwiUJZj6IyBT5mrb5YE_kSoMeqI0fFs95OpyWpldUalcS47Lg7Dt-1d0gNC7xyQWGj0qnM$<https://urldefense.com/v3/__http:/www.fromoldbooks.org__;!!N4vogdjhuJM!C1fj636Ib2y0kENz2AwiUJZj6IyBT5mrb5YE_kSoMeqI0fFs95OpyWpldUalcS47Lg7Dt-1d0gNC7xyQWGj0qnM$>
EK
Eliot Kimberin reply to Tim Thompson, 16 Jul 2022, 01:4015 Jul 2022 23:48 #
Yes, that is 100x faster. Clearly I need to slow down and RTFM a bit more closely. Thanks, E. _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>
Quoted text · Tim Thompson · 82 lines
From: Tim Thompson <timathom at gmail.com> Date: Friday, July 15, 2022 at 4:40 PM To: Eliot Kimber <eliot.kimber at servicenow.com> Cc: Liam R. E. Quin <liam at fromoldbooks.org>, basex-talk at mailman.uni-konstanz.de <basex-talk at mailman.uni-konstanz.de> Subject: Re: [basex-talk] Possible to Speed Up This Lookup? [External Email] Is there a performance difference when using db:attribute() instead[1]? Tim [1] https://docs.basex.org/wiki/Database_Module#db:attribute<https://urldefense.com/v3/__https:/docs.basex.org/wiki/Database_Module*db:attribute__;Iw!!N4vogdjhuJM!Ca2zI2049S4xTcwSnRICwcUHLqboyNh8rMFfVglLT9-A39rtWpP0ztq55CticIRYCsegnqwW0D7StcFb-Z4lHQ$> -- Tim A. Thompson (he, him) Librarian for Applied Metadata Research Yale University Library On Fri, Jul 15, 2022 at 5:21 PM Eliot Kimber <eliot.kimber at servicenow.com<mailto:eliot.kimber at servicenow.com>> wrote: Actually, I think I spoke too soon. This test: return (db:option('attrindex'), for $node in $map return prof:track(db:open($rkDatabase)/doc-where-used-index/where-used-entry/@key[xs:integer(.) eq 9151416])?time ! util:formatTime(.) Returns: true 00.030 seconds Where my current database context is the database named in $rkDatabase. A small improvement (0.036 to 0.030) but not dramatic. So I think I’m still missing some detail, probably a configuration issue. Cheers, E, Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://urldefense.com/v3/__https:/www.linkedin.com/company/servicenow__;!!N4vogdjhuJM!Ca2zI2049S4xTcwSnRICwcUHLqboyNh8rMFfVglLT9-A39rtWpP0ztq55CticIRYCsegnqwW0D7StcFPeRPeww$> | Twitter<https://urldefense.com/v3/__https:/twitter.com/servicenow__;!!N4vogdjhuJM!Ca2zI2049S4xTcwSnRICwcUHLqboyNh8rMFfVglLT9-A39rtWpP0ztq55CticIRYCsegnqwW0D7StcGPMd9U7w$> | YouTube<https://urldefense.com/v3/__https:/www.youtube.com/user/servicenowinc__;!!N4vogdjhuJM!Ca2zI2049S4xTcwSnRICwcUHLqboyNh8rMFfVglLT9-A39rtWpP0ztq55CticIRYCsegnqwW0D7StcHUl5mSEQ$> | Facebook<https://urldefense.com/v3/__https:/www.facebook.com/servicenow__;!!N4vogdjhuJM!Ca2zI2049S4xTcwSnRICwcUHLqboyNh8rMFfVglLT9-A39rtWpP0ztq55CticIRYCsegnqwW0D7StcEdjcThVg$> From: Eliot Kimber <eliot.kimber at servicenow.com<mailto:eliot.kimber at servicenow.com>> Date: Friday, July 15, 2022 at 3:39 PM To: Liam R. E. Quin <liam at fromoldbooks.org<mailto:liam at fromoldbooks.org>>, basex-talk at mailman.uni-konstanz.de<mailto:basex-talk at mailman.uni-konstanz.de> <basex-talk at mailman.uni-konstanz.de<mailto:basex-talk at mailman.uni-konstanz.de>> Subject: Re: [basex-talk] Possible to Speed Up This Lookup? Yes, turning on the attribute index definitely improves performa nce dramatically, which is no surprise. Not sure how I missed the need to expressly turn on these indexes. Doing that now. Cheers, E. Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://urldefense.com/v3/__https:/www.linkedin.com/company/servicenow__;!!N4vogdjhuJM!Ca2zI2049S4xTcwSnRICwcUHLqboyNh8rMFfVglLT9-A39rtWpP0ztq55CticIRYCsegnqwW0D7StcFPeRPeww$> | Twitter<https://urldefense.com/v3/__https:/twitter.com/servicenow__;!!N4vogdjhuJM!Ca2zI2049S4xTcwSnRICwcUHLqboyNh8rMFfVglLT9-A39rtWpP0ztq55CticIRYCsegnqwW0D7StcGPMd9U7w$> | YouTube<https://urldefense.com/v3/__https:/www.youtube.com/user/servicenowinc__;!!N4vogdjhuJM!Ca2zI2049S4xTcwSnRICwcUHLqboyNh8rMFfVglLT9-A39rtWpP0ztq55CticIRYCsegnqwW0D7StcHUl5mSEQ$> | Facebook<https://urldefense.com/v3/__https:/www.facebook.com/servicenow__;!!N4vogdjhuJM!Ca2zI2049S4xTcwSnRICwcUHLqboyNh8rMFfVglLT9-A39rtWpP0ztq55CticIRYCsegnqwW0D7StcEdjcThVg$> From: Eliot Kimber <eliot.kimber at servicenow.com<mailto:eliot.kimber at servicenow.com>> Date: Friday, July 15, 2022 at 3:01 PM To: Liam R. E. Quin <liam at fromoldbooks.org<mailto:liam at fromoldbooks.org>>, basex-talk at mailman.uni-konstanz.de<mailto:basex-talk at mailman.uni-konstanz.de> <basex-talk at mailman.uni-konstanz.de<mailto:basex-talk at mailman.uni-konstanz.de>> Subject: Re: [basex-talk] Possible to Speed Up This Lookup? There are on the order of 50K index entry items. I think the attribute index is probably the answer—for some reason I assumed that would be indexed by default but I see that it is not looking at the database details. I’ll try enabling the index and see what happens. Cheers, E. Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://urldefense.com/v3/__https:/www.linkedin.com/company/servicenow__;!!N4vogdjhuJM!Ca2zI2049S4xTcwSnRICwcUHLqboyNh8rMFfVglLT9-A39rtWpP0ztq55CticIRYCsegnqwW0D7StcFPeRPeww$> | Twitter<https://urldefense.com/v3/__https:/twitter.com/servicenow__;!!N4vogdjhuJM!Ca2zI2049S4xTcwSnRICwcUHLqboyNh8rMFfVglLT9-A39rtWpP0ztq55CticIRYCsegnqwW0D7StcGPMd9U7w$> | YouTube<https://urldefense.com/v3/__https:/www.youtube.com/user/servicenowinc__;!!N4vogdjhuJM!Ca2zI2049S4xTcwSnRICwcUHLqboyNh8rMFfVglLT9-A39rtWpP0ztq55CticIRYCsegnqwW0D7StcHUl5mSEQ$> | Facebook<https://urldefense.com/v3/__https:/www.facebook.com/servicenow__;!!N4vogdjhuJM!Ca2zI2049S4xTcwSnRICwcUHLqboyNh8rMFfVglLT9-A39rtWpP0ztq55CticIRYCsegnqwW0D7StcEdjcThVg$> From: Liam R. E. Quin <liam at fromoldbooks.org<mailto:liam at fromoldbooks.org>> Date: Friday, July 15, 2022 at 1:22 PM To: Eliot Kimber <eliot.kimber at servicenow.com<mailto:eliot.kimber at servicenow.com>>, basex-talk at mailman.uni-konstanz.de<mailto:basex-talk at mailman.uni-konstanz.de> <basex-talk at mailman.uni-konstanz.de<mailto:basex-talk at mailman.uni-konstanz.de>> Subject: Re: [basex-talk] Possible to Speed Up This Lookup? [External Email] On Fri, 2022-07-15 at 18:04 +0000, Eliot Kimber wrote: > collection($rkDatabase)/doc-where-used-index/where-used- > entry[string(@key) eq string(db:node-id($node))] > > Where the markup is shown in [1] below. > > Using prof:track() I’m measuring a consistent 0.036 seconds per > lookup, could it be opening the collection that takes time? Or are there hundreds of thousands of where-used-entry elements, and, if so, do you have an attribute value index? -- Liam Quin, https://urldefense.com/v3/__https://www.delightfulcomputing.com/__;!!N4vogdjhuJM!C1fj636Ib2y0kENz2AwiUJZj6IyBT5mrb5YE_kSoMeqI0fFs95OpyWpldUalcS47Lg7Dt-1d0gNC7xyQXLy9rZQ$<https://urldefense.com/v3/__https:/www.delightfulcomputing.com/__;!!N4vogdjhuJM!C1fj636Ib2y0kENz2AwiUJZj6IyBT5mrb5YE_kSoMeqI0fFs95OpyWpldUalcS47Lg7Dt-1d0gNC7xyQXLy9rZQ$> Available for XML/Document/Information Architecture/XSLT/ XSL/XQuery/Web/Text Processing/A11Y training, work & consulting. Barefoot Web-slave, antique illustrations: https://urldefense.com/v3/__http://www.fromoldbooks.org__;!!N4vogdjhuJM!C1fj636Ib2y0kENz2AwiUJZj6IyBT5mrb5YE_kSoMeqI0fFs95OpyWpldUalcS47Lg7Dt-1d0gNC7xyQWGj0qnM$<https://urldefense.com/v3/__http:/www.fromoldbooks.org__;!!N4vogdjhuJM!C1fj636Ib2y0kENz2AwiUJZj6IyBT5mrb5YE_kSoMeqI0fFs95OpyWpldUalcS47Lg7Dt-1d0gNC7xyQWGj0qnM$>
EK
Eliot Kimberin reply to Liam R. E. Quin, 16 Jul 2022, 01:5416 Jul 2022 00:00 #
I profiled the db:open() and collection() calls and they were basically instantaneous. Moving to using db:attribute() and the index makes my process 100x times faster. Now I feel like a bit of an idiot for not having figured this out sooner. Somebody needs to write the Optimizing BaseX book…. Cheers, E. _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>
Quoted text · BaseX-Talk · 27 lines
From: BaseX-Talk <basex-talk-bounces at mailman.uni-konstanz.de> on behalf of Liam R. E. Quin <liam at fromoldbooks.org> Date: Friday, July 15, 2022 at 4:55 PM To: basex-talk at mailman.uni-konstanz.de <basex-talk at mailman.uni-konstanz.de> Subject: Re: [basex-talk] Possible to Speed Up This Lookup? [External Email] On Fri, 2022-07-15 at 21:20 +0000, Eliot Kimber wrote: > > return (db:option('attrindex'), > for $node in $map > return prof:track(db:open($rkDatabase)/doc-where-used- > index/where-used-entry/@key[xs:integer(.) eq 9151416])?time ! > util:formatTime(.) can you take the db:open() call out and see how much faster the expression runs without it? return (db:option('attrindex'), let $db := db:open($rkDatabase) return for $node in $map return prof:track($db/doc-where-used-index/where-used-entry /@key[xs:integer(.) eq 9151416])?time ! util:formatTime(.) (if i have that right) -- Liam Quin, https://urldefense.com/v3/__https://www.delightfulcomputing.com/__;!!N4vogdjhuJM!Ge6_HxIH-uWYEol5ImGje0BVsQ_FKkbp1x9czcNl7d0t6BKvoQ1v0sENn6_kHvOQI0dfFmEig6LEfg3dDcIyNVA$<https://urldefense.com/v3/__https:/www.delightfulcomputing.com/__;!!N4vogdjhuJM!Ge6_HxIH-uWYEol5ImGje0BVsQ_FKkbp1x9czcNl7d0t6BKvoQ1v0sENn6_kHvOQI0dfFmEig6LEfg3dDcIyNVA$> Available for XML/Document/Information Architecture/XSLT/ XSL/XQuery/Web/Text Processing/A11Y training, work & consulting. Barefoot Web-slave, antique illustrations: https://urldefense.com/v3/__http://www.fromoldbooks.org__;!!N4vogdjhuJM!Ge6_HxIH-uWYEol5ImGje0BVsQ_FKkbp1x9czcNl7d0t6BKvoQ1v0sENn6_kHvOQI0dfFmEig6LEfg3dk_syZfc$<https://urldefense.com/v3/__http:/www.fromoldbooks.org__;!!N4vogdjhuJM!Ge6_HxIH-uWYEol5ImGje0BVsQ_FKkbp1x9czcNl7d0t6BKvoQ1v0sENn6_kHvOQI0dfFmEig6LEfg3dk_syZfc$>
TT
Tim Thompsonin reply to Eliot Kimber, 15 Jul 2022, 23:2016 Jul 2022 01:40 #
Is there a performance difference when using db:attribute() instead[1]? Tim [1] https://docs.basex.org/wiki/Database_Module#db:attribute
-- Tim A. Thompson (he, him) Librarian for Applied Metadata Research Yale University Library
Quoted text · Eliot Kimber · 183 lines
On Fri, Jul 15, 2022 at 5:21 PM Eliot Kimber <eliot.kimber at servicenow.com> wrote: > Actually, I think I spoke too soon. > > > > This test: > > return (db:option('attrindex'), > > for $node in $map > > return > prof:track(db:open($rkDatabase)/doc-where-used-index/where-used-entry/@key[xs:integer(.) > eq 9151416])?time ! util:formatTime(.) > > > > Returns: > > > > true > > 00.030 seconds > > > > Where my current database context is the database named in $rkDatabase. > > > > A small improvement (0.036 to 0.030) but not dramatic. > > > > So I think I’m still missing some detail, probably a configuration issue. > > > > Cheers, > > > > E, > > _____________________________________________ > > *Eliot Kimber* > > Sr Staff Content Engineer > > O: 512 554 9368 > > M: 512 554 9368 > > servicenow.com <https://www.servicenow.com> > > LinkedIn <https://www.linkedin.com/company/servicenow> | Twitter > <https://twitter.com/servicenow> | YouTube > <https://www.youtube.com/user/servicenowinc> | Facebook > <https://www.facebook.com/servicenow> > > > > *From: *Eliot Kimber <eliot.kimber at servicenow.com> > *Date: *Friday, July 15, 2022 at 3:39 PM > *To: *Liam R. E. Quin <liam at fromoldbooks.org>, > basex-talk at mailman.uni-konstanz.de <basex-talk at mailman.uni-konstanz.de> > *Subject: *Re: [basex-talk] Possible to Speed Up This Lookup? > > Yes, turning on the attribute index definitely improves performa nce > dramatically, which is no surprise. Not sure how I missed the need to > expressly turn on these indexes. Doing that now. > > > > Cheers, > > > > E. > > > > _____________________________________________ > > *Eliot Kimber* > > Sr Staff Content Engineer > > O: 512 554 9368 > > M: 512 554 9368 > > servicenow.com <https://www.servicenow.com> > > LinkedIn <https://www.linkedin.com/company/servicenow> | Twitter > <https://twitter.com/servicenow> | YouTube > <https://www.youtube.com/user/servicenowinc> | Facebook > <https://www.facebook.com/servicenow> > > > > *From: *Eliot Kimber <eliot.kimber at servicenow.com> > *Date: *Friday, July 15, 2022 at 3:01 PM > *To: *Liam R. E. Quin <liam at fromoldbooks.org>, > basex-talk at mailman.uni-konstanz.de <basex-talk at mailman.uni-konstanz.de> > *Subject: *Re: [basex-talk] Possible to Speed Up This Lookup? > > There are on the order of 50K index entry items. > > > > I think the attribute index is probably the answer—for some reason I > assumed that would be indexed by default but I see that it is not looking > at the database details. > > > > I’ll try enabling the index and see what happens. > > > > Cheers, > > > > E. > > > > _____________________________________________ > > *Eliot Kimber* > > Sr Staff Content Engineer > > O: 512 554 9368 > > M: 512 554 9368 > > servicenow.com <https://www.servicenow.com> > > LinkedIn <https://www.linkedin.com/company/servicenow> | Twitter > <https://twitter.com/servicenow> | YouTube > <https://www.youtube.com/user/servicenowinc> | Facebook > <https://www.facebook.com/servicenow> > > > > *From: *Liam R. E. Quin <liam at fromoldbooks.org> > *Date: *Friday, July 15, 2022 at 1:22 PM > *To: *Eliot Kimber <eliot.kimber at servicenow.com>, > basex-talk at mailman.uni-konstanz.de <basex-talk at mailman.uni-konstanz.de> > *Subject: *Re: [basex-talk] Possible to Speed Up This Lookup? > > [External Email] > > > On Fri, 2022-07-15 at 18:04 +0000, Eliot Kimber wrote: > > collection($rkDatabase)/doc-where-used-index/where-used- > > entry[string(@key) eq string(db:node-id($node))] > > > > Where the markup is shown in [1] below. > > > > Using prof:track() I’m measuring a consistent 0.036 seconds per > > lookup, > > could it be opening the collection that takes time? Or are there > hundreds of thousands of where-used-entry elements, and, if so, do you > have an attribute value index? > > -- > Liam Quin, > https://urldefense.com/v3/__https://www.delightfulcomputing.com/__;!!N4vogdjhuJM!C1fj636Ib2y0kENz2AwiUJZj6IyBT5mrb5YE_kSoMeqI0fFs95OpyWpldUalcS47Lg7Dt-1d0gNC7xyQXLy9rZQ$ > <https://urldefense.com/v3/__https:/www.delightfulcomputing.com/__;!!N4vogdjhuJM!C1fj636Ib2y0kENz2AwiUJZj6IyBT5mrb5YE_kSoMeqI0fFs95OpyWpldUalcS47Lg7Dt-1d0gNC7xyQXLy9rZQ$> > Available for XML/Document/Information Architecture/XSLT/ > XSL/XQuery/Web/Text Processing/A11Y training, work & consulting. > Barefoot Web-slave, antique illustrations: > https://urldefense.com/v3/__http://www.fromoldbooks.org__;!!N4vogdjhuJM!C1fj636Ib2y0kENz2AwiUJZj6IyBT5mrb5YE_kSoMeqI0fFs95OpyWpldUalcS47Lg7Dt-1d0gNC7xyQWGj0qnM$ > <https://urldefense.com/v3/__http:/www.fromoldbooks.org__;!!N4vogdjhuJM!C1fj636Ib2y0kENz2AwiUJZj6IyBT5mrb5YE_kSoMeqI0fFs95OpyWpldUalcS47Lg7Dt-1d0gNC7xyQWGj0qnM$> >
LQ
Liam R. E. Quinin reply to Eliot Kimber, 15 Jul 2022, 23:2016 Jul 2022 01:54 #
Quoted text · Eliot Kimber · 7 lines
On Fri, 2022-07-15 at 21:20 +0000, Eliot Kimber wrote: > > return (db:option('attrindex'), > for $node in $map > return prof:track(db:open($rkDatabase)/doc-where-used- > index/where-used-entry/@key[xs:integer(.) eq 9151416])?time ! > util:formatTime(.)
can you take the db:open() call out and see how much faster the expression runs without it?
return (db:option('attrindex'),
      let $db := db:open($rkDatabase)
      return
      for $node in $map

      return
prof:track($db/doc-where-used-index/where-used-entry
/@key[xs:integer(.) eq 9151416])?time ! util:formatTime(.)
(if i have that right)
-- Liam Quin, https://www.delightfulcomputing.com/ Available for XML/Document/Information Architecture/XSLT/ XSL/XQuery/Web/Text Processing/A11Y training, work & consulting. Barefoot Web-slave, antique illustrations: http://www.fromoldbooks.org
BD
Bridger Dyson-Smithin reply to Eliot Kimber, 16 Jul 2022, 00:0016 Jul 2022 02:27 #
Hi Eliot - Others are much more proficient and knowledgeable, but I try the following: 1. ABI := Always Be Indexing :) 2. I think that Christian has expressed that, with indexes in play, many queries can experience significant optimizations by reversing the query pattern; eg starting with the attribute value and working back up from there, etc. 3. The optimized query view can be super helpful in this regard, too. I'm sure there are other, better habits and strategies, I know (I just don't have them ingrained! 🙂). Other users will share, I'm sure. Have a pleasant weekend! Best, Bridger
Quoted text · Eliot Kimber · 86 lines
On Fri, Jul 15, 2022, 6:01 PM Eliot Kimber <eliot.kimber at servicenow.com> wrote: > I profiled the db:open() and collection() calls and they were basically > instantaneous. > > > > Moving to using db:attribute() and the index makes my process 100x times > faster. Now I feel like a bit of an idiot for not having figured this out > sooner. > > > > Somebody needs to write the Optimizing BaseX book…. > > > > Cheers, > > > > E. > > > > _____________________________________________ > > *Eliot Kimber* > > Sr Staff Content Engineer > > O: 512 554 9368 > > M: 512 554 9368 > > servicenow.com <https://www.servicenow.com> > > LinkedIn <https://www.linkedin.com/company/servicenow> | Twitter > <https://twitter.com/servicenow> | YouTube > <https://www.youtube.com/user/servicenowinc> | Facebook > <https://www.facebook.com/servicenow> > > > > *From: *BaseX-Talk <basex-talk-bounces at mailman.uni-konstanz.de> on behalf > of Liam R. E. Quin <liam at fromoldbooks.org> > *Date: *Friday, July 15, 2022 at 4:55 PM > *To: *basex-talk at mailman.uni-konstanz.de < > basex-talk at mailman.uni-konstanz.de> > *Subject: *Re: [basex-talk] Possible to Speed Up This Lookup? > > [External Email] > > > On Fri, 2022-07-15 at 21:20 +0000, Eliot Kimber wrote: > > > > return (db:option('attrindex'), > > for $node in $map > > return prof:track(db:open($rkDatabase)/doc-where-used- > > index/where-used-entry/@key[xs:integer(.) eq 9151416])?time ! > > util:formatTime(.) > > can you take the db:open() call out and see how much faster the > expression runs without it? > > return (db:option('attrindex'), > let $db := db:open($rkDatabase) > return > for $node in $map > > return > prof:track($db/doc-where-used-index/where-used-entry > /@key[xs:integer(.) eq 9151416])?time ! util:formatTime(.) > > (if i have that right) > > -- > Liam Quin, > https://urldefense.com/v3/__https://www.delightfulcomputing.com/__;!!N4vogdjhuJM!Ge6_HxIH-uWYEol5ImGje0BVsQ_FKkbp1x9czcNl7d0t6BKvoQ1v0sENn6_kHvOQI0dfFmEig6LEfg3dDcIyNVA$ > <https://urldefense.com/v3/__https:/www.delightfulcomputing.com/__;!!N4vogdjhuJM!Ge6_HxIH-uWYEol5ImGje0BVsQ_FKkbp1x9czcNl7d0t6BKvoQ1v0sENn6_kHvOQI0dfFmEig6LEfg3dDcIyNVA$> > Available for XML/Document/Information Architecture/XSLT/ > XSL/XQuery/Web/Text Processing/A11Y training, work & consulting. > Barefoot Web-slave, antique illustrations: > https://urldefense.com/v3/__http://www.fromoldbooks.org__;!!N4vogdjhuJM!Ge6_HxIH-uWYEol5ImGje0BVsQ_FKkbp1x9czcNl7d0t6BKvoQ1v0sENn6_kHvOQI0dfFmEig6LEfg3dk_syZfc$ > <https://urldefense.com/v3/__http:/www.fromoldbooks.org__;!!N4vogdjhuJM!Ge6_HxIH-uWYEol5ImGje0BVsQ_FKkbp1x9czcNl7d0t6BKvoQ1v0sENn6_kHvOQI0dfFmEig6LEfg3dk_syZfc$> >
LQ
Liam R. E. Quinin reply to Eliot Kimber, 16 Jul 2022, 00:0016 Jul 2022 03:15 #
Quoted text · Eliot Kimber · 7 lines
On Fri, 2022-07-15 at 22:00 +0000, Eliot Kimber wrote: > I profiled the db:open() and collection() calls and they were > basically instantaneous. > > Moving to using db:attribute() and the index makes my process 100x > times faster. Now I feel like a bit of an idiot for not having > figured this out sooner.
Nah, optimization is tricksy. Glad it's working!
Quoted text · Eliot Kimber · 63 lines
> > Somebody needs to write the Optimizing BaseX book…. > > Cheers, > > E. > > _____________________________________________ > Eliot Kimber > Sr Staff Content Engineer > O: 512 554 9368 > M: 512 554 9368 > servicenow.com<https://www.servicenow.com> > LinkedIn<https://www.linkedin.com/company/servicenow> | > Twitter<https://twitter.com/servicenow> | > YouTube<https://www.youtube.com/user/servicenowinc> | > Facebook<https://www.facebook.com/servicenow> > > From: BaseX-Talk <basex-talk-bounces at mailman.uni-konstanz.de> on > behalf of Liam R. E. Quin <liam at fromoldbooks.org> > Date: Friday, July 15, 2022 at 4:55 PM > To: > basex-talk at mailman.uni-konstanz.de <basex-talk at mailman.uni-konstanz.de > > > Subject: Re: [basex-talk] Possible to Speed Up This Lookup? > [External Email] > > > On Fri, 2022-07-15 at 21:20 +0000, Eliot Kimber wrote: > > > > return (db:option('attrindex'), > > for $node in $map > > return prof:track(db:open($rkDatabase)/doc-where-used- > > index/where-used-entry/@key[xs:integer(.) eq 9151416])?time ! > > util:formatTime(.) > > can you take the db:open() call out and see how much faster the > expression runs without it? > > return (db:option('attrindex'), > let $db := db:open($rkDatabase) > return > for $node in $map > > return > prof:track($db/doc-where-used-index/where-used-entry > /@key[xs:integer(.) eq 9151416])?time ! util:formatTime(.) > > (if i have that right) > > -- > Liam Quin, > https://urldefense.com/v3/__https://www.delightfulcomputing.com/__;!!N4vogdjhuJM!Ge6_HxIH-uWYEol5ImGje0BVsQ_FKkbp1x9czcNl7d0t6BKvoQ1v0sENn6_kHvOQI0dfFmEig6LEfg3dDcIyNVA$ > < > https://urldefense.com/v3/__https:/www.delightfulcomputing.com/__;!!N4vogdjhuJM!Ge6_HxIH-uWYEol5ImGje0BVsQ_FKkbp1x9czcNl7d0t6BKvoQ1v0sENn6_kHvOQI0dfFmEig6LEfg3dDcIyNVA$ > > > Available for XML/Document/Information Architecture/XSLT/ > XSL/XQuery/Web/Text Processing/A11Y training, work & consulting. > Barefoot Web-slave, antique illustrations: > https://urldefense.com/v3/__http://www.fromoldbooks.org__;!!N4vogdjhuJM!Ge6_HxIH-uWYEol5ImGje0BVsQ_FKkbp1x9czcNl7d0t6BKvoQ1v0sENn6_kHvOQI0dfFmEig6LEfg3dk_syZfc$ > < > https://urldefense.com/v3/__http:/www.fromoldbooks.org__;!!N4vogdjhuJM!Ge6_HxIH-uWYEol5ImGje0BVsQ_FKkbp1x9czcNl7d0t6BKvoQ1v0sENn6_kHvOQI0dfFmEig6LEfg3dk_syZfc$ > >
-- Liam Quin, https://www.delightfulcomputing.com/ Available for XML/Document/Information Architecture/XSLT/ XSL/XQuery/Web/Text Processing/A11Y training, work & consulting. Barefoot Web-slave, antique illustrations: http://www.fromoldbooks.org
CG
Christian Grün16 Jul 2022 09:03 #
Quoted text · Liam R. E. Quin · 1 line
> Nah, optimization is tricksy.
True. Both for the optimizer and the developer. I can spot two indicators for …
Quoted text · Liam R. E. Quin · 1 line
> db:open($rkDatabase)/doc-where-used-index/where-used-entry/@key[xs:integer(.) eq 9151416]
… not being rewritten for index access: • If $rkDatabase is not static, it won’t be possible at compile time to check if the database contains the required index. • BaseX has no numeric index (not yet [1]), which is why the predicate can’t be rewritten. Queries of the following pattern …
Quoted text · Liam R. E. Quin · 1 line
> db:open('database')/doc-where-used-index/where-used-entry/@key[. eq '9151416']
… should definitely be optimized. It’s sometimes possible to enforce the use of an index [2], but to simplify things, it’s just fine to explicitly address the index via db:text and db:attribute. We have listed some more examples for index rewritings in our documentation [3]. I’ll need to extend the compilation, as various more rewriting patterns were implemented since then. For example, both '=' and 'eq' will now be rewritten if we can see at compile time that the rewriting of 'eq' won’t ignore errors that might be raised in its original form (more than single items to be compared; invalid input types; …). [1] https://github.com/BaseXdb/basex/issues/2069 [2] https://docs.basex.org/wiki/Indexes#Enforce_Rewritings [3] https://docs.basex.org/wiki/XQuery_Optimizations#Index_Rewritings

Written for BaseX 9.7. BaseX may have changed since: please check the current documentation.

« RESTXQ: How to Reflect Progress of Long-Running QuerriesAdding Attributes After the Fact: How To? »
⚡Generated with XQuery