Db:token() vs db:attribute(): Attribute is much slower -- why

From: Eliot Kimber <eliot.kimber at servicenow.com> Date: Tuesday, October 10, 2023 at 6:02 PM To: basex-talk at mailman.uni-konstanz.de <basex-talk at mailman.uni-konstanz.de> Subject: Db:token() vs db:attribute(): Attribute is much slower -- why Using 10.6 on macOS: I have a data file with elements like: <record language="en-US" family="pce-test-data" bundlename="bundle-appportman-application-portfolio-management" topicpath="product/application-portfolio-management/concept/application-portfolio-management-landing-page.dita" notviewed="viewed" viewcount="13579"> One such entry for each topic in our public docs server that is accessed at least once (Adobe page views data). I generate a report for all the topics used from a root map that includes the page views data for each topic for which there is an entry. The database with this data set is indexed with an attribute and a token index. In my testing I’m finding that lookup using db:attribute() on the @topicpath attribute (which is always a single value) is about 10x slower than using db:token(). That is, I have this code: $records := db:token($analyticsmgmt:analyticsDatabase, $docPath, 'topicpath')/.. When I run this for a set of about 1200 topics, the db:token version takes about 400ms but the db:attribute version takes 9 seconds! I’m surprised at this result—I would expect the two forms of index to perform with the same speed: the attribute is a single blank-delimited token and I would expect the string match under the covers to be using the same code path. But using the db:token database is clearly faster by a lot. If I report the time each lookup uses, the times vary quite a lot, but for the token lookup they tend to be in the 10-60ms range, but for the attribute lookup, many are in the 1 or 2 second range, which explains the much longer run time. I’m wondering if this result is expected due to implementation details of the attribute index as distinct from the token index or if this indicates some logic bug in the attribute index implementation. Cheers, E. Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>

On Wed, Oct 11, 2023 at 1:16 AM Eliot Kimber <eliot.kimber at servicenow.com> wrote: > Sorry, my times are off—I was doing a nonsense multiplication of the > result of prof:track. > > Reporting the raw times returned by db:token(), I get: > > "lookup by docPath took 0.255696ms" > > "lookup by docPath took 0.054183ms" > > "lookup by docPath took 0.211373ms" > > "lookup by docPath took 0.175889ms" > > "lookup by docPath took 0.220833ms" > > "lookup by docPath took 0.156875ms" > > "lookup by docPath took 0.139612ms" > > "lookup by docPath took 0.007018ms" > > "lookup by docPath took 0.179297ms" > > "lookup by docPath took 0.00964ms" > > "lookup by docPath took 0.093668ms" > > > > Which adds up to about 400ms for 1300 topics looked up. > > > > Using db:attribute() I get: > > "lookup by docPath took 3.794676ms" > > "lookup by docPath took 0.032425ms" > > "lookup by docPath took 0.092237ms" > > "lookup by docPath took 0.097815ms" > > "lookup by docPath took 1.246617ms" > > "lookup by docPath took 0.050855ms" > > "lookup by docPath took 0.065685ms" > > "lookup by docPath took 0.00511ms" > > "lookup by docPath took 0.057833ms" > > > > Cheers, > > > > E. > > > > _____________________________________________ > > *Eliot Kimber* > > Sr Staff Content Engineer > > O: 512 554 9368 > > M: 512 554 9368 > > servicenow.com <https://www.servicenow.com> > > LinkedIn <https://www.linkedin.com/company/servicenow> | Twitter > <https://twitter.com/servicenow> | YouTube > <https://www.youtube.com/user/servicenowinc> | Facebook > <https://www.facebook.com/servicenow> > > > > *From: *Eliot Kimber <eliot.kimber at servicenow.com> > *Date: *Tuesday, October 10, 2023 at 6:02 PM > *To: *basex-talk at mailman.uni-konstanz.de < > basex-talk at mailman.uni-konstanz.de> > *Subject: *Db:token() vs db:attribute(): Attribute is much slower -- why > > Using 10.6 on macOS: > > > > I have a data file with elements like: > > <record language="en-US" family="pce-test-data" > bundlename="bundle-appportman-application-portfolio-management" > topicpath="product/application-portfolio-management/concept/application-portfolio-management-landing-page.dita" > notviewed="viewed" viewcount="13579"> > > > > One such entry for each topic in our public docs server that is accessed > at least once (Adobe page views data). > > > > I generate a report for all the topics used from a root map that includes > the page views data for each topic for which there is an entry. > > > > The database with this data set is indexed with an attribute and a token > index. > > > > In my testing I’m finding that lookup using db:attribute() on the > @topicpath attribute (which is always a single value) is about 10x slower > than using db:token(). > > > > That is, I have this code: > > $records := db:token($analyticsmgmt:analyticsDatabase, $docPath, > 'topicpath')/.. > > > > When I run this for a set of about 1200 topics, the db:token version takes > about 400ms but the db:attribute version takes 9 seconds! > > > > I’m surprised at this result—I would expect the two forms of index to > perform with the same speed: the attribute is a single blank-delimited > token and I would expect the string match under the covers to be using the > same code path. > > > > But using the db:token database is clearly faster by a lot. > > > > If I report the time each lookup uses, the times vary quite a lot, but for > the token lookup they tend to be in the 10-60ms range, but for the > attribute lookup, many are in the 1 or 2 second range, which explains the > much longer run time. > > > > I’m wondering if this result is expected due to implementation details of > the attribute index as distinct from the token index or if this indicates > some logic bug in the attribute index implementation. > > > > Cheers, > > > > E. > > _____________________________________________ > > *Eliot Kimber* > > Sr Staff Content Engineer > > O: 512 554 9368 > > M: 512 554 9368 > > servicenow.com <https://www.servicenow.com> > > LinkedIn <https://www.linkedin.com/company/servicenow> | Twitter > <https://twitter.com/servicenow> | YouTube > <https://www.youtube.com/user/servicenowinc> | Facebook > <https://www.facebook.com/servicenow> >

From: Christian Grün <christian.gruen at gmail.com> Date: Sunday, October 15, 2023 at 6:41 AM To: Eliot Kimber <eliot.kimber at servicenow.com> Cc: basex-talk at mailman.uni-konstanz.de <basex-talk at mailman.uni-konstanz.de> Subject: Re: [basex-talk] Db:token() vs db:attribute(): Attribute is much slower -- why [External Email] Hi Eliot,If the length of a search strings exceeds MAXLEN [1], a sequential search will be performed instead. You can achieve index-based lookups by increasing the limit when creating/optimizing a database.Hope this helps,Christian[1] https://docs.basex.org/wiki/Options#MAXLENOn<https://docs.basex.org/wiki/Options#MAXLENOn> i This message needs your attention • Someone new is on this email. Provided by ServiceNow DT (Employee Portal KB0077950) - This banner is visible only to ServiceNow employees.<https://mimecast.com> Hi Eliot, If the length of a search strings exceeds MAXLEN [1], a sequential search will be performed instead. You can achieve index-based lookups by increasing the limit when creating/optimizing a database. Hope this helps, Christian [1] https://docs.basex.org/wiki/Options#MAXLEN<https://docs.basex.org/wiki/Options#MAXLEN> On Wed, Oct 11, 2023 at 1:16 AM Eliot Kimber <eliot.kimber at servicenow.com<mailto:eliot.kimber at servicenow.com>> wrote: Sorry, my times are off—I was doing a nonsense multiplication of the result of prof:track. Reporting the raw times returned by db:token(), I get: "lookup by docPath took 0.255696ms" "lookup by docPath took 0.054183ms" "lookup by docPath took 0.211373ms" "lookup by docPath took 0.175889ms" "lookup by docPath took 0.220833ms" "lookup by docPath took 0.156875ms" "lookup by docPath took 0.139612ms" "lookup by docPath took 0.007018ms" "lookup by docPath took 0.179297ms" "lookup by docPath took 0.00964ms" "lookup by docPath took 0.093668ms" Which adds up to about 400ms for 1300 topics looked up. Using db:attribute() I get: "lookup by docPath took 3.794676ms" "lookup by docPath took 0.032425ms" "lookup by docPath took 0.092237ms" "lookup by docPath took 0.097815ms" "lookup by docPath took 1.246617ms" "lookup by docPath took 0.050855ms" "lookup by docPath took 0.065685ms" "lookup by docPath took 0.00511ms" "lookup by docPath took 0.057833ms" Cheers, E. Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow> From: Eliot Kimber <eliot.kimber at servicenow.com<mailto:eliot.kimber at servicenow.com>> Date: Tuesday, October 10, 2023 at 6:02 PM To: basex-talk at mailman.uni-konstanz.de<mailto:basex-talk at mailman.uni-konstanz.de> <basex-talk at mailman.uni-konstanz.de<mailto:basex-talk at mailman.uni-konstanz.de>> Subject: Db:token() vs db:attribute(): Attribute is much slower -- why Using 10.6 on macOS: I have a data file with elements like: <record language="en-US" family="pce-test-data" bundlename="bundle-appportman-application-portfolio-management" topicpath="product/application-portfolio-management/concept/application-portfolio-management-landing-page.dita" notviewed="viewed" viewcount="13579"> One such entry for each topic in our public docs server that is accessed at least once (Adobe page views data). I generate a report for all the topics used from a root map that includes the page views data for each topic for which there is an entry. The database with this data set is indexed with an attribute and a token index. In my testing I’m finding that lookup using db:attribute() on the @topicpath attribute (which is always a single value) is about 10x slower than using db:token(). That is, I have this code: $records := db:token($analyticsmgmt:analyticsDatabase, $docPath, 'topicpath')/.. When I run this for a set of about 1200 topics, the db:token version takes about 400ms but the db:attribute version takes 9 seconds! I’m surprised at this result—I would expect the two forms of index to perform with the same speed: the attribute is a single blank-delimited token and I would expect the string match under the covers to be using the same code path. But using the db:token database is clearly faster by a lot. If I report the time each lookup uses, the times vary quite a lot, but for the token lookup they tend to be in the 10-60ms range, but for the attribute lookup, many are in the 1 or 2 second range, which explains the much longer run time. I’m wondering if this result is expected due to implementation details of the attribute index as distinct from the token index or if this indicates some logic bug in the attribute index implementation. Cheers, E. Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>

From: Eliot Kimber <eliot.kimber at servicenow.com> Date: Sunday, October 15, 2023 at 9:04 AM To: Christian Grün <christian.gruen at gmail.com> Cc: basex-talk at mailman.uni-konstanz.de <basex-talk at mailman.uni-konstanz.de> Subject: Re: [basex-talk] Db:token() vs db:attribute(): Attribute is much slower -- why I see, so if I’m indexing on longer strings, such as the URLs in my data sets, I need to set MAXLEN to accommodate. Thanks for the tip. Cheers, E. Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow> From: Christian Grün <christian.gruen at gmail.com> Date: Sunday, October 15, 2023 at 6:41 AM To: Eliot Kimber <eliot.kimber at servicenow.com> Cc: basex-talk at mailman.uni-konstanz.de <basex-talk at mailman.uni-konstanz.de> Subject: Re: [basex-talk] Db:token() vs db:attribute(): Attribute is much slower -- why [External Email] Hi Eliot,If the length of a search strings exceeds MAXLEN [1], a sequential search will be performed instead. You can achieve index-based lookups by increasing the limit when creating/optimizing a database.Hope this helps,Christian[1] https://docs.basex.org/wiki/Options#MAXLENOn<https://docs.basex.org/wiki/Options#MAXLENOn> i This message needs your attention • Someone new is on this email. Provided by ServiceNow DT (Employee Portal KB0077950) - This banner is visible only to ServiceNow employees.<https://mimecast.com> Hi Eliot, If the length of a search strings exceeds MAXLEN [1], a sequential search will be performed instead. You can achieve index-based lookups by increasing the limit when creating/optimizing a database. Hope this helps, Christian [1] https://docs.basex.org/wiki/Options#MAXLEN<https://docs.basex.org/wiki/Options#MAXLEN> On Wed, Oct 11, 2023 at 1:16 AM Eliot Kimber <eliot.kimber at servicenow.com<mailto:eliot.kimber at servicenow.com>> wrote: Sorry, my times are off—I was doing a nonsense multiplication of the result of prof:track. Reporting the raw times returned by db:token(), I get: "lookup by docPath took 0.255696ms" "lookup by docPath took 0.054183ms" "lookup by docPath took 0.211373ms" "lookup by docPath took 0.175889ms" "lookup by docPath took 0.220833ms" "lookup by docPath took 0.156875ms" "lookup by docPath took 0.139612ms" "lookup by docPath took 0.007018ms" "lookup by docPath took 0.179297ms" "lookup by docPath took 0.00964ms" "lookup by docPath took 0.093668ms" Which adds up to about 400ms for 1300 topics looked up. Using db:attribute() I get: "lookup by docPath took 3.794676ms" "lookup by docPath took 0.032425ms" "lookup by docPath took 0.092237ms" "lookup by docPath took 0.097815ms" "lookup by docPath took 1.246617ms" "lookup by docPath took 0.050855ms" "lookup by docPath took 0.065685ms" "lookup by docPath took 0.00511ms" "lookup by docPath took 0.057833ms" Cheers, E. Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow> From: Eliot Kimber <eliot.kimber at servicenow.com<mailto:eliot.kimber at servicenow.com>> Date: Tuesday, October 10, 2023 at 6:02 PM To: basex-talk at mailman.uni-konstanz.de<mailto:basex-talk at mailman.uni-konstanz.de> <basex-talk at mailman.uni-konstanz.de<mailto:basex-talk at mailman.uni-konstanz.de>> Subject: Db:token() vs db:attribute(): Attribute is much slower -- why Using 10.6 on macOS: I have a data file with elements like: <record language="en-US" family="pce-test-data" bundlename="bundle-appportman-application-portfolio-management" topicpath="product/application-portfolio-management/concept/application-portfolio-management-landing-page.dita" notviewed="viewed" viewcount="13579"> One such entry for each topic in our public docs server that is accessed at least once (Adobe page views data). I generate a report for all the topics used from a root map that includes the page views data for each topic for which there is an entry. The database with this data set is indexed with an attribute and a token index. In my testing I’m finding that lookup using db:attribute() on the @topicpath attribute (which is always a single value) is about 10x slower than using db:token(). That is, I have this code: $records := db:token($analyticsmgmt:analyticsDatabase, $docPath, 'topicpath')/.. When I run this for a set of about 1200 topics, the db:token version takes about 400ms but the db:attribute version takes 9 seconds! I’m surprised at this result—I would expect the two forms of index to perform with the same speed: the attribute is a single blank-delimited token and I would expect the string match under the covers to be using the same code path. But using the db:token database is clearly faster by a lot. If I report the time each lookup uses, the times vary quite a lot, but for the token lookup they tend to be in the 10-60ms range, but for the attribute lookup, many are in the 1 or 2 second range, which explains the much longer run time. I’m wondering if this result is expected due to implementation details of the attribute index as distinct from the token index or if this indicates some logic bug in the attribute index implementation. Cheers, E. Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>