Slow query

> So reordering the conditions for having smaller subset right from the > begging isn't relevant.

On Fri, Jan 30, 2015 at 3:55 PM, Menashè Eliezer <meliezer at ogs.trieste.it> wrote: > Hello, > I wonder if the attached query can be optimised. I'm attaching all relevant > information. > Basex 7.9, Debian, powerful server. > This is just an example. The queries will be built based on a compilation of > a search form. > Any help would be appreciated. > 40 seconds are not acceptable. > > -- > With kind regards, > Menashè >

On 01/30/2015 04:35 PM, Christian Grün wrote: > Hi Menashè, > > First of all, I wonder if your query really does what you want it to > do. I noticed for example that some of the where conditions start with > "$x/", while others start with "/" and some others start with no > slash. Is this intentional?

> > Some more comments: > > * I would recommend you to avoid numeric tests in the @codeListValue > tests and use string tests instead (/@codeListValue = "7827", etc).

> * Usually, you can also get rid of the xs:dateTime() conversions, > because items of type date and time can also be compared as strings.

> * I'm not sure what the predicates [*] are supposed to do in your > query. If you remove them, you will get the same results.

> * In some cases, if you know that an element name is distinct, you can > get rid of all the explicit child steps and directly address the node > via the descendant axis.

> >> So reordering the conditions for having smaller subset right from the >> begging isn't relevant. > Reordering shouldn't make a big difference anyway, because BaseX tries > to find the cheapest index request by itself, based on the database > statistics.

> > Beside that, I would be interested to hear if you get better results > with BaseX 8.0 [1], because we recently spent quite some time to > further improve our index rewriting rules.

> > Hope this helps, > Christian > > [1] http://files.basex.org/releases/latest > > > > On Fri, Jan 30, 2015 at 3:55 PM, Menashè Eliezer > <meliezer at ogs.trieste.it> wrote: >> Hello, >> I wonder if the attached query can be optimised. I'm attaching all relevant >> information. >> Basex 7.9, Debian, powerful server. >> This is just an example. The queries will be built based on a compilation of >> a search form. >> Any help would be appreciated. >> 40 seconds are not acceptable. >> >> -- >> With kind regards, >> Menashè >>

> How can I remove *?

>> * In some cases, if you know that an element name is distinct, you can >> get rid of all the explicit child steps and directly address the node >> via the descendant axis. > > Thanks, but it's not relevant in my case.

> Sure, I'l also try BaseX 8.0 and compare. Should I recreate the db importing > the xml files for testing the improved indexing?

>> [1] http://files.basex.org/releases/latest >> >> >> >> On Fri, Jan 30, 2015 at 3:55 PM, Menashè Eliezer >> <meliezer at ogs.trieste.it> wrote: >>> >>> Hello, >>> I wonder if the attached query can be optimised. I'm attaching all >>> relevant >>> information. >>> Basex 7.9, Debian, powerful server. >>> This is just an example. The queries will be built based on a compilation >>> of >>> a search form. >>> Any help would be appreciated. >>> 40 seconds are not acceptable. >>> >>> -- >>> With kind regards, >>> Menashè >>> > > -- > With kind regards, > Menashè > >

On 01/30/2015 05:18 PM, Christian Grün wrote: > /gmd:MD_Metadata/gmd:identificationInfo/sdn:SDN_DataIdentification/gmd:descriptiveKeywords[1]/gmd:MD_Keywords/gmd:keyword[2]/sdn:SDN_ParameterDiscoveryCode/@codeListValue >> How can I remove *? > Simply remove the predicate; a[*]/b is the same as a/b.

> >>> * In some cases, if you know that an element name is distinct, you can >>> get rid of all the explicit child steps and directly address the node >>> via the descendant axis. >> Thanks, but it's not relevant in my case. > Is it because the element names are not distinct? Or is it because > your input form allows users to choose arbitrary paths for arbitrary > documents?

> >> Sure, I'l also try BaseX 8.0 and compare. Should I recreate the db importing >> the xml files for testing the improved indexing? > We have actually improved support for collections, but the database > format itself has not changed, so it shouldn't make a difference in > your case. > > Christian > > >>> [1] http://files.basex.org/releases/latest >>> >>> >>> >>> On Fri, Jan 30, 2015 at 3:55 PM, Menashè Eliezer >>> <meliezer at ogs.trieste.it> wrote: >>>> Hello, >>>> I wonder if the attached query can be optimised. I'm attaching all >>>> relevant >>>> information. >>>> Basex 7.9, Debian, powerful server. >>>> This is just an example. The queries will be built based on a compilation >>>> of >>>> a search form. >>>> Any help would be appreciated. >>>> 40 seconds are not acceptable. >>>> >>>> -- >>>> With kind regards, >>>> Menashè >>>> >> -- >> With kind regards, >> Menashè >> >>

On Fri, Jan 30, 2015 at 5:26 PM, Menashè Eliezer <meliezer at ogs.trieste.it> wrote: > On 01/30/2015 05:18 PM, Christian Grün wrote: >> >> >> /gmd:MD_Metadata/gmd:identificationInfo/sdn:SDN_DataIdentification/gmd:descriptiveKeywords[1]/gmd:MD_Keywords/gmd:keyword[2]/sdn:SDN_ParameterDiscoveryCode/@codeListValue >>> >>> How can I remove *? >> >> Simply remove the predicate; a[*]/b is the same as a/b. > > Maybe I wasn't clear. The actual number appears in the xml file, e.g., > gmd:descriptiveKeywords[1] > Anyway, I've removed all [*] and I get the same correct result, however the > processing time is doubled... >> >> >>>> * In some cases, if you know that an element name is distinct, you can >>>> get rid of all the explicit child steps and directly address the node >>>> via the descendant axis. >>> >>> Thanks, but it's not relevant in my case. >> >> Is it because the element names are not distinct? Or is it because >> your input form allows users to choose arbitrary paths for arbitrary >> documents? > > The element names are not distinct. > >> >>> Sure, I'l also try BaseX 8.0 and compare. Should I recreate the db >>> importing >>> the xml files for testing the improved indexing? >> >> We have actually improved support for collections, but the database >> format itself has not changed, so it shouldn't make a difference in >> your case. >> >> Christian >> >> >>>> [1] http://files.basex.org/releases/latest >>>> >>>> >>>> >>>> On Fri, Jan 30, 2015 at 3:55 PM, Menashè Eliezer >>>> <meliezer at ogs.trieste.it> wrote: >>>>> >>>>> Hello, >>>>> I wonder if the attached query can be optimised. I'm attaching all >>>>> relevant >>>>> information. >>>>> Basex 7.9, Debian, powerful server. >>>>> This is just an example. The queries will be built based on a >>>>> compilation >>>>> of >>>>> a search form. >>>>> Any help would be appreciated. >>>>> 40 seconds are not acceptable. >>>>> >>>>> -- >>>>> With kind regards, >>>>> Menashè >>>>> >>> -- >>> With kind regards, >>> Menashè >>> >>> > > With kind regards, > Menashè >

On 01/30/2015 05:31 PM, Christian Grün wrote: > It's indeed interesting that your query does not use any of the > existing index structures (if they did, you would find strings like > "applying text index" or "applying attribute index" in the query > info). Maybe/hopefully things look different with Version 8.0. > > > On Fri, Jan 30, 2015 at 5:26 PM, Menashè Eliezer > <meliezer at ogs.trieste.it> wrote: >> On 01/30/2015 05:18 PM, Christian Grün wrote: >>> >>> /gmd:MD_Metadata/gmd:identificationInfo/sdn:SDN_DataIdentification/gmd:descriptiveKeywords[1]/gmd:MD_Keywords/gmd:keyword[2]/sdn:SDN_ParameterDiscoveryCode/@codeListValue >>>> How can I remove *? >>> Simply remove the predicate; a[*]/b is the same as a/b. >> Maybe I wasn't clear. The actual number appears in the xml file, e.g., >> gmd:descriptiveKeywords[1] >> Anyway, I've removed all [*] and I get the same correct result, however the >> processing time is doubled... >>> >>>>> * In some cases, if you know that an element name is distinct, you can >>>>> get rid of all the explicit child steps and directly address the node >>>>> via the descendant axis. >>>> Thanks, but it's not relevant in my case. >>> Is it because the element names are not distinct? Or is it because >>> your input form allows users to choose arbitrary paths for arbitrary >>> documents? >> The element names are not distinct. >> >>>> Sure, I'l also try BaseX 8.0 and compare. Should I recreate the db >>>> importing >>>> the xml files for testing the improved indexing? >>> We have actually improved support for collections, but the database >>> format itself has not changed, so it shouldn't make a difference in >>> your case. >>> >>> Christian >>> >>> >>>>> [1] http://files.basex.org/releases/latest >>>>> >>>>> >>>>> >>>>> On Fri, Jan 30, 2015 at 3:55 PM, Menashè Eliezer >>>>> <meliezer at ogs.trieste.it> wrote: >>>>>> Hello, >>>>>> I wonder if the attached query can be optimised. I'm attaching all >>>>>> relevant >>>>>> information. >>>>>> Basex 7.9, Debian, powerful server. >>>>>> This is just an example. The queries will be built based on a >>>>>> compilation >>>>>> of >>>>>> a search form. >>>>>> Any help would be appreciated. >>>>>> 40 seconds are not acceptable. >>>>>> >>>>>> -- >>>>>> With kind regards, >>>>>> Menashè >>>>>> >>>> -- >>>> With kind regards, >>>> Menashè >>>> >>>> >> With kind regards, >> Menashè >>

On Fri, Jan 30, 2015 at 5:52 PM, Menashè Eliezer <meliezer at ogs.trieste.it> wrote: > Almost the same speed with version 8.0. > No indexing (no "applying" in the query info). > As I've attached before, indexes are active for this DB. > > With kind regards, > Menashè > > > On 01/30/2015 05:31 PM, Christian Grün wrote: >> >> It's indeed interesting that your query does not use any of the >> existing index structures (if they did, you would find strings like >> "applying text index" or "applying attribute index" in the query >> info). Maybe/hopefully things look different with Version 8.0. >> >> >> On Fri, Jan 30, 2015 at 5:26 PM, Menashè Eliezer >> <meliezer at ogs.trieste.it> wrote: >>> >>> On 01/30/2015 05:18 PM, Christian Grün wrote: >>>> >>>> >>>> >>>> /gmd:MD_Metadata/gmd:identificationInfo/sdn:SDN_DataIdentification/gmd:descriptiveKeywords[1]/gmd:MD_Keywords/gmd:keyword[2]/sdn:SDN_ParameterDiscoveryCode/@codeListValue >>>>> >>>>> How can I remove *? >>>> >>>> Simply remove the predicate; a[*]/b is the same as a/b. >>> >>> Maybe I wasn't clear. The actual number appears in the xml file, e.g., >>> gmd:descriptiveKeywords[1] >>> Anyway, I've removed all [*] and I get the same correct result, however >>> the >>> processing time is doubled... >>>> >>>> >>>>>> * In some cases, if you know that an element name is distinct, you can >>>>>> get rid of all the explicit child steps and directly address the node >>>>>> via the descendant axis. >>>>> >>>>> Thanks, but it's not relevant in my case. >>>> >>>> Is it because the element names are not distinct? Or is it because >>>> your input form allows users to choose arbitrary paths for arbitrary >>>> documents? >>> >>> The element names are not distinct. >>> >>>>> Sure, I'l also try BaseX 8.0 and compare. Should I recreate the db >>>>> importing >>>>> the xml files for testing the improved indexing? >>>> >>>> We have actually improved support for collections, but the database >>>> format itself has not changed, so it shouldn't make a difference in >>>> your case. >>>> >>>> Christian >>>> >>>> >>>>>> [1] http://files.basex.org/releases/latest >>>>>> >>>>>> >>>>>> >>>>>> On Fri, Jan 30, 2015 at 3:55 PM, Menashè Eliezer >>>>>> <meliezer at ogs.trieste.it> wrote: >>>>>>> >>>>>>> Hello, >>>>>>> I wonder if the attached query can be optimised. I'm attaching all >>>>>>> relevant >>>>>>> information. >>>>>>> Basex 7.9, Debian, powerful server. >>>>>>> This is just an example. The queries will be built based on a >>>>>>> compilation >>>>>>> of >>>>>>> a search form. >>>>>>> Any help would be appreciated. >>>>>>> 40 seconds are not acceptable. >>>>>>> >>>>>>> -- >>>>>>> With kind regards, >>>>>>> Menashè >>>>>>> >>>>> -- >>>>> With kind regards, >>>>> Menashè >>>>> >>>>> >>> With kind regards, >>> Menashè >>> >

On Fri, Jan 30, 2015 at 5:55 PM, Christian Grün <christian.gruen at gmail.com> wrote: > Could you possibly provide me with a small snapshot of your data > sources (one, two documents might be sufficient)? > > > On Fri, Jan 30, 2015 at 5:52 PM, Menashè Eliezer > <meliezer at ogs.trieste.it> wrote: >> Almost the same speed with version 8.0. >> No indexing (no "applying" in the query info). >> As I've attached before, indexes are active for this DB. >> >> With kind regards, >> Menashè >> >> >> On 01/30/2015 05:31 PM, Christian Grün wrote: >>> >>> It's indeed interesting that your query does not use any of the >>> existing index structures (if they did, you would find strings like >>> "applying text index" or "applying attribute index" in the query >>> info). Maybe/hopefully things look different with Version 8.0. >>> >>> >>> On Fri, Jan 30, 2015 at 5:26 PM, Menashè Eliezer >>> <meliezer at ogs.trieste.it> wrote: >>>> >>>> On 01/30/2015 05:18 PM, Christian Grün wrote: >>>>> >>>>> >>>>> >>>>> /gmd:MD_Metadata/gmd:identificationInfo/sdn:SDN_DataIdentification/gmd:descriptiveKeywords[1]/gmd:MD_Keywords/gmd:keyword[2]/sdn:SDN_ParameterDiscoveryCode/@codeListValue >>>>>> >>>>>> How can I remove *? >>>>> >>>>> Simply remove the predicate; a[*]/b is the same as a/b. >>>> >>>> Maybe I wasn't clear. The actual number appears in the xml file, e.g., >>>> gmd:descriptiveKeywords[1] >>>> Anyway, I've removed all [*] and I get the same correct result, however >>>> the >>>> processing time is doubled... >>>>> >>>>> >>>>>>> * In some cases, if you know that an element name is distinct, you can >>>>>>> get rid of all the explicit child steps and directly address the node >>>>>>> via the descendant axis. >>>>>> >>>>>> Thanks, but it's not relevant in my case. >>>>> >>>>> Is it because the element names are not distinct? Or is it because >>>>> your input form allows users to choose arbitrary paths for arbitrary >>>>> documents? >>>> >>>> The element names are not distinct. >>>> >>>>>> Sure, I'l also try BaseX 8.0 and compare. Should I recreate the db >>>>>> importing >>>>>> the xml files for testing the improved indexing? >>>>> >>>>> We have actually improved support for collections, but the database >>>>> format itself has not changed, so it shouldn't make a difference in >>>>> your case. >>>>> >>>>> Christian >>>>> >>>>> >>>>>>> [1] http://files.basex.org/releases/latest >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Fri, Jan 30, 2015 at 3:55 PM, Menashè Eliezer >>>>>>> <meliezer at ogs.trieste.it> wrote: >>>>>>>> >>>>>>>> Hello, >>>>>>>> I wonder if the attached query can be optimised. I'm attaching all >>>>>>>> relevant >>>>>>>> information. >>>>>>>> Basex 7.9, Debian, powerful server. >>>>>>>> This is just an example. The queries will be built based on a >>>>>>>> compilation >>>>>>>> of >>>>>>>> a search form. >>>>>>>> Any help would be appreciated. >>>>>>>> 40 seconds are not acceptable. >>>>>>>> >>>>>>>> -- >>>>>>>> With kind regards, >>>>>>>> Menashè >>>>>>>> >>>>>> -- >>>>>> With kind regards, >>>>>> Menashè >>>>>> >>>>>> >>>> With kind regards, >>>> Menashè >>>> >>

On Fri, 30 Jan 2015 18:11:59 +0100, Christian Grün <christian.gruen at gmail.com> wrote: > Hi Menashè, > > Thanks for the XML samples you sent me in private. I noticed that the > index rewritings will only be triggered if you formulate your query as > follows: > > OLD: > for $x in collection("ALL-CDIS") > where $x/gmd:MD_Metadata/gmd:identificationInfo/... > return ... > > NEW: > for $x in collection("ALL-CDIS")/gmd:MD_Metadata > where $x/gmd:identificationInfo/... > return ... > > It's difficult to explain in short sentences why Variant 1 cannot be > optimized that straightforward (basically, it's quite a different > pattern to look for), but I'll check out if we can extend our matcher > to also support these kind of queries. > > So, if possible, I would recommend you for now (and at least for > testing) to move the root element test after the collection() > function. I noticed that the first three child steps are the same in > all of your conditions: > > gmd:MD_Metadata/gmd:identificationInfo/sdn:SDN_DataIdentification > > If that will be always be the case, it surely makes sense to move all > of them to the "for" clause. > > Looking forward to your updated performance tests, > Christian > _______________________________ > > On Fri, Jan 30, 2015 at 5:55 PM, Christian Grün > <christian.gruen at gmail.com> wrote: >> Could you possibly provide me with a small snapshot of your data >> sources (one, two documents might be sufficient)? >> >> >> On Fri, Jan 30, 2015 at 5:52 PM, Menashè Eliezer >> <meliezer at ogs.trieste.it> wrote: >>> Almost the same speed with version 8.0. >>> No indexing (no "applying" in the query info). >>> As I've attached before, indexes are active for this DB. >>> >>> With kind regards, >>> Menashè >>> >>> >>> On 01/30/2015 05:31 PM, Christian Grün wrote: >>>> >>>> It's indeed interesting that your query does not use any of the >>>> existing index structures (if they did, you would find strings like >>>> "applying text index" or "applying attribute index" in the query >>>> info). Maybe/hopefully things look different with Version 8.0. >>>> >>>> >>>> On Fri, Jan 30, 2015 at 5:26 PM, Menashè Eliezer >>>> <meliezer at ogs.trieste.it> wrote: >>>>> >>>>> On 01/30/2015 05:18 PM, Christian Grün wrote: >>>>>> >>>>>> >>>>>> >>>>>> >

>>>>>>> >>>>>>> How can I remove *? >>>>>> >>>>>> Simply remove the predicate; a[*]/b is the same as a/b. >>>>> >>>>> Maybe I wasn't clear. The actual number appears in the xml file, > e.g., >>>>> gmd:descriptiveKeywords[1] >>>>> Anyway, I've removed all [*] and I get the same correct result, > however >>>>> the >>>>> processing time is doubled... >>>>>> >>>>>> >>>>>>>> * In some cases, if you know that an element name is distinct, you > can >>>>>>>> get rid of all the explicit child steps and directly address the > node >>>>>>>> via the descendant axis. >>>>>>> >>>>>>> Thanks, but it's not relevant in my case. >>>>>> >>>>>> Is it because the element names are not distinct? Or is it because >>>>>> your input form allows users to choose arbitrary paths for arbitrary >>>>>> documents? >>>>> >>>>> The element names are not distinct. >>>>> >>>>>>> Sure, I'l also try BaseX 8.0 and compare. Should I recreate the db >>>>>>> importing >>>>>>> the xml files for testing the improved indexing? >>>>>> >>>>>> We have actually improved support for collections, but the database >>>>>> format itself has not changed, so it shouldn't make a difference in >>>>>> your case. >>>>>> >>>>>> Christian >>>>>> >>>>>> >>>>>>>> [1] http://files.basex.org/releases/latest >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Fri, Jan 30, 2015 at 3:55 PM, Menashè Eliezer >>>>>>>> <meliezer at ogs.trieste.it> wrote: >>>>>>>>> >>>>>>>>> Hello, >>>>>>>>> I wonder if the attached query can be optimised. I'm attaching > all >>>>>>>>> relevant >>>>>>>>> information. >>>>>>>>> Basex 7.9, Debian, powerful server. >>>>>>>>> This is just an example. The queries will be built based on a >>>>>>>>> compilation >>>>>>>>> of >>>>>>>>> a search form. >>>>>>>>> Any help would be appreciated. >>>>>>>>> 40 seconds are not acceptable. >>>>>>>>> >>>>>>>>> -- >>>>>>>>> With kind regards, >>>>>>>>> Menashè >>>>>>>>> >>>>>>> -- >>>>>>> With kind regards, >>>>>>> Menashè >>>>>>> >>>>>>> >>>>> With kind regards, >>>>> Menashè >>>>> >>>

> Should I expect to see the usage of an index for each of the where phrases?

> Have a nice weekend!

> Menashè > > On Fri, 30 Jan 2015 18:11:59 +0100, Christian Grün > <christian.gruen at gmail.com> wrote: >> Hi Menashè, >> >> Thanks for the XML samples you sent me in private. I noticed that the >> index rewritings will only be triggered if you formulate your query as >> follows: >> >> OLD: >> for $x in collection("ALL-CDIS") >> where $x/gmd:MD_Metadata/gmd:identificationInfo/... >> return ... >> >> NEW: >> for $x in collection("ALL-CDIS")/gmd:MD_Metadata >> where $x/gmd:identificationInfo/... >> return ... >> >> It's difficult to explain in short sentences why Variant 1 cannot be >> optimized that straightforward (basically, it's quite a different >> pattern to look for), but I'll check out if we can extend our matcher >> to also support these kind of queries. >> >> So, if possible, I would recommend you for now (and at least for >> testing) to move the root element test after the collection() >> function. I noticed that the first three child steps are the same in >> all of your conditions: >> >> gmd:MD_Metadata/gmd:identificationInfo/sdn:SDN_DataIdentification >> >> If that will be always be the case, it surely makes sense to move all >> of them to the "for" clause. >> >> Looking forward to your updated performance tests, >> Christian >> _______________________________ >> >> On Fri, Jan 30, 2015 at 5:55 PM, Christian Grün >> <christian.gruen at gmail.com> wrote: >>> Could you possibly provide me with a small snapshot of your data >>> sources (one, two documents might be sufficient)? >>> >>> >>> On Fri, Jan 30, 2015 at 5:52 PM, Menashè Eliezer >>> <meliezer at ogs.trieste.it> wrote: >>>> Almost the same speed with version 8.0. >>>> No indexing (no "applying" in the query info). >>>> As I've attached before, indexes are active for this DB. >>>> >>>> With kind regards, >>>> Menashè >>>> >>>> >>>> On 01/30/2015 05:31 PM, Christian Grün wrote: >>>>> >>>>> It's indeed interesting that your query does not use any of the >>>>> existing index structures (if they did, you would find strings like >>>>> "applying text index" or "applying attribute index" in the query >>>>> info). Maybe/hopefully things look different with Version 8.0. >>>>> >>>>> >>>>> On Fri, Jan 30, 2015 at 5:26 PM, Menashè Eliezer >>>>> <meliezer at ogs.trieste.it> wrote: >>>>>> >>>>>> On 01/30/2015 05:18 PM, Christian Grün wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> >> > /gmd:MD_Metadata/gmd:identificationInfo/sdn:SDN_DataIdentification/gmd:descriptiveKeywords[1]/gmd:MD_Keywords/gmd:keyword[2]/sdn:SDN_ParameterDiscoveryCode/@codeListValue >>>>>>>> >>>>>>>> How can I remove *? >>>>>>> >>>>>>> Simply remove the predicate; a[*]/b is the same as a/b. >>>>>> >>>>>> Maybe I wasn't clear. The actual number appears in the xml file, >> e.g., >>>>>> gmd:descriptiveKeywords[1] >>>>>> Anyway, I've removed all [*] and I get the same correct result, >> however >>>>>> the >>>>>> processing time is doubled... >>>>>>> >>>>>>> >>>>>>>>> * In some cases, if you know that an element name is distinct, you >> can >>>>>>>>> get rid of all the explicit child steps and directly address the >> node >>>>>>>>> via the descendant axis. >>>>>>>> >>>>>>>> Thanks, but it's not relevant in my case. >>>>>>> >>>>>>> Is it because the element names are not distinct? Or is it because >>>>>>> your input form allows users to choose arbitrary paths for arbitrary >>>>>>> documents? >>>>>> >>>>>> The element names are not distinct. >>>>>> >>>>>>>> Sure, I'l also try BaseX 8.0 and compare. Should I recreate the db >>>>>>>> importing >>>>>>>> the xml files for testing the improved indexing? >>>>>>> >>>>>>> We have actually improved support for collections, but the database >>>>>>> format itself has not changed, so it shouldn't make a difference in >>>>>>> your case. >>>>>>> >>>>>>> Christian >>>>>>> >>>>>>> >>>>>>>>> [1] http://files.basex.org/releases/latest >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Fri, Jan 30, 2015 at 3:55 PM, Menashè Eliezer >>>>>>>>> <meliezer at ogs.trieste.it> wrote: >>>>>>>>>> >>>>>>>>>> Hello, >>>>>>>>>> I wonder if the attached query can be optimised. I'm attaching >> all >>>>>>>>>> relevant >>>>>>>>>> information. >>>>>>>>>> Basex 7.9, Debian, powerful server. >>>>>>>>>> This is just an example. The queries will be built based on a >>>>>>>>>> compilation >>>>>>>>>> of >>>>>>>>>> a search form. >>>>>>>>>> Any help would be appreciated. >>>>>>>>>> 40 seconds are not acceptable. >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> With kind regards, >>>>>>>>>> Menashè >>>>>>>>>> >>>>>>>> -- >>>>>>>> With kind regards, >>>>>>>> Menashè >>>>>>>> >>>>>>>> >>>>>> With kind regards, >>>>>> Menashè >>>>>> >>>> > -- > Menashè

On Fri, Jan 30, 2015 at 9:49 PM, Christian Grün <christian.gruen at gmail.com> wrote: > Hi Menashè, > >> Should I expect to see the usage of an index for each of the where phrases? > > Usually, only one predicate will be rewritten for index access, and > the remaining conditions will be answered sequentially. > >> Have a nice weekend! > > Enjoy, > Christian > > >> Menashè >> >> On Fri, 30 Jan 2015 18:11:59 +0100, Christian Grün >> <christian.gruen at gmail.com> wrote: >>> Hi Menashè, >>> >>> Thanks for the XML samples you sent me in private. I noticed that the >>> index rewritings will only be triggered if you formulate your query as >>> follows: >>> >>> OLD: >>> for $x in collection("ALL-CDIS") >>> where $x/gmd:MD_Metadata/gmd:identificationInfo/... >>> return ... >>> >>> NEW: >>> for $x in collection("ALL-CDIS")/gmd:MD_Metadata >>> where $x/gmd:identificationInfo/... >>> return ... >>> >>> It's difficult to explain in short sentences why Variant 1 cannot be >>> optimized that straightforward (basically, it's quite a different >>> pattern to look for), but I'll check out if we can extend our matcher >>> to also support these kind of queries. >>> >>> So, if possible, I would recommend you for now (and at least for >>> testing) to move the root element test after the collection() >>> function. I noticed that the first three child steps are the same in >>> all of your conditions: >>> >>> gmd:MD_Metadata/gmd:identificationInfo/sdn:SDN_DataIdentification >>> >>> If that will be always be the case, it surely makes sense to move all >>> of them to the "for" clause. >>> >>> Looking forward to your updated performance tests, >>> Christian >>> _______________________________ >>> >>> On Fri, Jan 30, 2015 at 5:55 PM, Christian Grün >>> <christian.gruen at gmail.com> wrote: >>>> Could you possibly provide me with a small snapshot of your data >>>> sources (one, two documents might be sufficient)? >>>> >>>> >>>> On Fri, Jan 30, 2015 at 5:52 PM, Menashè Eliezer >>>> <meliezer at ogs.trieste.it> wrote: >>>>> Almost the same speed with version 8.0. >>>>> No indexing (no "applying" in the query info). >>>>> As I've attached before, indexes are active for this DB. >>>>> >>>>> With kind regards, >>>>> Menashè >>>>> >>>>> >>>>> On 01/30/2015 05:31 PM, Christian Grün wrote: >>>>>> >>>>>> It's indeed interesting that your query does not use any of the >>>>>> existing index structures (if they did, you would find strings like >>>>>> "applying text index" or "applying attribute index" in the query >>>>>> info). Maybe/hopefully things look different with Version 8.0. >>>>>> >>>>>> >>>>>> On Fri, Jan 30, 2015 at 5:26 PM, Menashè Eliezer >>>>>> <meliezer at ogs.trieste.it> wrote: >>>>>>> >>>>>>> On 01/30/2015 05:18 PM, Christian Grün wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>> >> /gmd:MD_Metadata/gmd:identificationInfo/sdn:SDN_DataIdentification/gmd:descriptiveKeywords[1]/gmd:MD_Keywords/gmd:keyword[2]/sdn:SDN_ParameterDiscoveryCode/@codeListValue >>>>>>>>> >>>>>>>>> How can I remove *? >>>>>>>> >>>>>>>> Simply remove the predicate; a[*]/b is the same as a/b. >>>>>>> >>>>>>> Maybe I wasn't clear. The actual number appears in the xml file, >>> e.g., >>>>>>> gmd:descriptiveKeywords[1] >>>>>>> Anyway, I've removed all [*] and I get the same correct result, >>> however >>>>>>> the >>>>>>> processing time is doubled... >>>>>>>> >>>>>>>> >>>>>>>>>> * In some cases, if you know that an element name is distinct, you >>> can >>>>>>>>>> get rid of all the explicit child steps and directly address the >>> node >>>>>>>>>> via the descendant axis. >>>>>>>>> >>>>>>>>> Thanks, but it's not relevant in my case. >>>>>>>> >>>>>>>> Is it because the element names are not distinct? Or is it because >>>>>>>> your input form allows users to choose arbitrary paths for arbitrary >>>>>>>> documents? >>>>>>> >>>>>>> The element names are not distinct. >>>>>>> >>>>>>>>> Sure, I'l also try BaseX 8.0 and compare. Should I recreate the db >>>>>>>>> importing >>>>>>>>> the xml files for testing the improved indexing? >>>>>>>> >>>>>>>> We have actually improved support for collections, but the database >>>>>>>> format itself has not changed, so it shouldn't make a difference in >>>>>>>> your case. >>>>>>>> >>>>>>>> Christian >>>>>>>> >>>>>>>> >>>>>>>>>> [1] http://files.basex.org/releases/latest >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Fri, Jan 30, 2015 at 3:55 PM, Menashè Eliezer >>>>>>>>>> <meliezer at ogs.trieste.it> wrote: >>>>>>>>>>> >>>>>>>>>>> Hello, >>>>>>>>>>> I wonder if the attached query can be optimised. I'm attaching >>> all >>>>>>>>>>> relevant >>>>>>>>>>> information. >>>>>>>>>>> Basex 7.9, Debian, powerful server. >>>>>>>>>>> This is just an example. The queries will be built based on a >>>>>>>>>>> compilation >>>>>>>>>>> of >>>>>>>>>>> a search form. >>>>>>>>>>> Any help would be appreciated. >>>>>>>>>>> 40 seconds are not acceptable. >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> With kind regards, >>>>>>>>>>> Menashè >>>>>>>>>>> >>>>>>>>> -- >>>>>>>>> With kind regards, >>>>>>>>> Menashè >>>>>>>>> >>>>>>>>> >>>>>>> With kind regards, >>>>>>> Menashè >>>>>>> >>>>> >> -- >> Menashè

On Sat, 31 Jan 2015 16:42:32 +0100, Christian Grün <christian.gruen at gmail.com> wrote: > Hi Menashè, > > With the latest snapshot [1], your original query should now be > rewritten for index access as well. Looking forward to your tests, > > Christian > > PS: In terms of performance, it may still be worthwhile to move > redundant paths to the for clause; but just try and see. > > [1] http://files.basex.org/releases/latest/ > > > > On Fri, Jan 30, 2015 at 9:49 PM, Christian Grün > <christian.gruen at gmail.com> wrote: >> Hi Menashè, >> >>> Should I expect to see the usage of an index for each of the where > phrases? >> >> Usually, only one predicate will be rewritten for index access, and >> the remaining conditions will be answered sequentially. >> >>> Have a nice weekend! >> >> Enjoy, >> Christian >> >> >>> Menashè >>> >>> On Fri, 30 Jan 2015 18:11:59 +0100, Christian Grün >>> <christian.gruen at gmail.com> wrote: >>>> Hi Menashè, >>>> >>>> Thanks for the XML samples you sent me in private. I noticed that the >>>> index rewritings will only be triggered if you formulate your query as >>>> follows: >>>> >>>> OLD: >>>> for $x in collection("ALL-CDIS") >>>> where $x/gmd:MD_Metadata/gmd:identificationInfo/... >>>> return ... >>>> >>>> NEW: >>>> for $x in collection("ALL-CDIS")/gmd:MD_Metadata >>>> where $x/gmd:identificationInfo/... >>>> return ... >>>> >>>> It's difficult to explain in short sentences why Variant 1 cannot be >>>> optimized that straightforward (basically, it's quite a different >>>> pattern to look for), but I'll check out if we can extend our matcher >>>> to also support these kind of queries. >>>> >>>> So, if possible, I would recommend you for now (and at least for >>>> testing) to move the root element test after the collection() >>>> function. I noticed that the first three child steps are the same in >>>> all of your conditions: >>>> >>>> gmd:MD_Metadata/gmd:identificationInfo/sdn:SDN_DataIdentification >>>> >>>> If that will be always be the case, it surely makes sense to move all >>>> of them to the "for" clause. >>>> >>>> Looking forward to your updated performance tests, >>>> Christian >>>> _______________________________ >>>> >>>> On Fri, Jan 30, 2015 at 5:55 PM, Christian Grün >>>> <christian.gruen at gmail.com> wrote: >>>>> Could you possibly provide me with a small snapshot of your data >>>>> sources (one, two documents might be sufficient)? >>>>> >>>>> >>>>> On Fri, Jan 30, 2015 at 5:52 PM, Menashè Eliezer >>>>> <meliezer at ogs.trieste.it> wrote: >>>>>> Almost the same speed with version 8.0. >>>>>> No indexing (no "applying" in the query info). >>>>>> As I've attached before, indexes are active for this DB. >>>>>> >>>>>> With kind regards, >>>>>> Menashè >>>>>> >>>>>> >>>>>> On 01/30/2015 05:31 PM, Christian Grün wrote: >>>>>>> >>>>>>> It's indeed interesting that your query does not use any of the >>>>>>> existing index structures (if they did, you would find strings like >>>>>>> "applying text index" or "applying attribute index" in the query >>>>>>> info). Maybe/hopefully things look different with Version 8.0. >>>>>>> >>>>>>> >>>>>>> On Fri, Jan 30, 2015 at 5:26 PM, Menashè Eliezer >>>>>>> <meliezer at ogs.trieste.it> wrote: >>>>>>>> >>>>>>>> On 01/30/2015 05:18 PM, Christian Grün wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>> >>> >

>>>>>>>>>> >>>>>>>>>> How can I remove *? >>>>>>>>> >>>>>>>>> Simply remove the predicate; a[*]/b is the same as a/b. >>>>>>>> >>>>>>>> Maybe I wasn't clear. The actual number appears in the xml file, >>>> e.g., >>>>>>>> gmd:descriptiveKeywords[1] >>>>>>>> Anyway, I've removed all [*] and I get the same correct result, >>>> however >>>>>>>> the >>>>>>>> processing time is doubled... >>>>>>>>> >>>>>>>>> >>>>>>>>>>> * In some cases, if you know that an element name is distinct, > you >>>> can >>>>>>>>>>> get rid of all the explicit child steps and directly address > the >>>> node >>>>>>>>>>> via the descendant axis. >>>>>>>>>> >>>>>>>>>> Thanks, but it's not relevant in my case. >>>>>>>>> >>>>>>>>> Is it because the element names are not distinct? Or is it > because >>>>>>>>> your input form allows users to choose arbitrary paths for > arbitrary >>>>>>>>> documents? >>>>>>>> >>>>>>>> The element names are not distinct. >>>>>>>> >>>>>>>>>> Sure, I'l also try BaseX 8.0 and compare. Should I recreate the > db >>>>>>>>>> importing >>>>>>>>>> the xml files for testing the improved indexing? >>>>>>>>> >>>>>>>>> We have actually improved support for collections, but the > database >>>>>>>>> format itself has not changed, so it shouldn't make a difference > in >>>>>>>>> your case. >>>>>>>>> >>>>>>>>> Christian >>>>>>>>> >>>>>>>>> >>>>>>>>>>> [1] http://files.basex.org/releases/latest >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Fri, Jan 30, 2015 at 3:55 PM, Menashè Eliezer >>>>>>>>>>> <meliezer at ogs.trieste.it> wrote: >>>>>>>>>>>> >>>>>>>>>>>> Hello, >>>>>>>>>>>> I wonder if the attached query can be optimised. I'm attaching >>>> all >>>>>>>>>>>> relevant >>>>>>>>>>>> information. >>>>>>>>>>>> Basex 7.9, Debian, powerful server. >>>>>>>>>>>> This is just an example. The queries will be built based on a >>>>>>>>>>>> compilation >>>>>>>>>>>> of >>>>>>>>>>>> a search form. >>>>>>>>>>>> Any help would be appreciated. >>>>>>>>>>>> 40 seconds are not acceptable. >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> With kind regards, >>>>>>>>>>>> Menashè >>>>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> With kind regards, >>>>>>>>>> Menashè >>>>>>>>>> >>>>>>>>>> >>>>>>>> With kind regards, >>>>>>>> Menashè >>>>>>>> >>>>>> >>> -- >>> Menashè

On 02/02/2015 02:02 PM, Menashè Eliezer wrote: > Hi Christian, > > Thank you very much! Unfortunately I'll be at the office only tomorrow. > > Menashè > > On Sat, 31 Jan 2015 16:42:32 +0100, Christian Grün > <christian.gruen at gmail.com> wrote: >> Hi Menashè, >> >> With the latest snapshot [1], your original query should now be >> rewritten for index access as well. Looking forward to your tests, >> >> Christian >> >> PS: In terms of performance, it may still be worthwhile to move >> redundant paths to the for clause; but just try and see. >> >> [1] http://files.basex.org/releases/latest/ >> >> >> >> On Fri, Jan 30, 2015 at 9:49 PM, Christian Grün >> <christian.gruen at gmail.com> wrote: >>> Hi Menashè, >>> >>>> Should I expect to see the usage of an index for each of the where >> phrases? >>> Usually, only one predicate will be rewritten for index access, and >>> the remaining conditions will be answered sequentially. >>> >>>> Have a nice weekend! >>> Enjoy, >>> Christian >>> >>> >>>> Menashè >>>> >>>> On Fri, 30 Jan 2015 18:11:59 +0100, Christian Grün >>>> <christian.gruen at gmail.com> wrote: >>>>> Hi Menashè, >>>>> >>>>> Thanks for the XML samples you sent me in private. I noticed that the >>>>> index rewritings will only be triggered if you formulate your query as >>>>> follows: >>>>> >>>>> OLD: >>>>> for $x in collection("ALL-CDIS") >>>>> where $x/gmd:MD_Metadata/gmd:identificationInfo/... >>>>> return ... >>>>> >>>>> NEW: >>>>> for $x in collection("ALL-CDIS")/gmd:MD_Metadata >>>>> where $x/gmd:identificationInfo/... >>>>> return ... >>>>> >>>>> It's difficult to explain in short sentences why Variant 1 cannot be >>>>> optimized that straightforward (basically, it's quite a different >>>>> pattern to look for), but I'll check out if we can extend our matcher >>>>> to also support these kind of queries. >>>>> >>>>> So, if possible, I would recommend you for now (and at least for >>>>> testing) to move the root element test after the collection() >>>>> function. I noticed that the first three child steps are the same in >>>>> all of your conditions: >>>>> >>>>> gmd:MD_Metadata/gmd:identificationInfo/sdn:SDN_DataIdentification >>>>> >>>>> If that will be always be the case, it surely makes sense to move all >>>>> of them to the "for" clause. >>>>> >>>>> Looking forward to your updated performance tests, >>>>> Christian >>>>> _______________________________ >>>>> >>>>> On Fri, Jan 30, 2015 at 5:55 PM, Christian Grün >>>>> <christian.gruen at gmail.com> wrote: >>>>>> Could you possibly provide me with a small snapshot of your data >>>>>> sources (one, two documents might be sufficient)? >>>>>> >>>>>> >>>>>> On Fri, Jan 30, 2015 at 5:52 PM, Menashè Eliezer >>>>>> <meliezer at ogs.trieste.it> wrote: >>>>>>> Almost the same speed with version 8.0. >>>>>>> No indexing (no "applying" in the query info). >>>>>>> As I've attached before, indexes are active for this DB. >>>>>>> >>>>>>> With kind regards, >>>>>>> Menashè >>>>>>> >>>>>>> >>>>>>> On 01/30/2015 05:31 PM, Christian Grün wrote: >>>>>>>> It's indeed interesting that your query does not use any of the >>>>>>>> existing index structures (if they did, you would find strings like >>>>>>>> "applying text index" or "applying attribute index" in the query >>>>>>>> info). Maybe/hopefully things look different with Version 8.0. >>>>>>>> >>>>>>>> >>>>>>>> On Fri, Jan 30, 2015 at 5:26 PM, Menashè Eliezer >>>>>>>> <meliezer at ogs.trieste.it> wrote: >>>>>>>>> On 01/30/2015 05:18 PM, Christian Grün wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> > /gmd:MD_Metadata/gmd:identificationInfo/sdn:SDN_DataIdentification/gmd:descriptiveKeywords[1]/gmd:MD_Keywords/gmd:keyword[2]/sdn:SDN_ParameterDiscoveryCode/@codeListValue >>>>>>>>>>> How can I remove *? >>>>>>>>>> Simply remove the predicate; a[*]/b is the same as a/b. >>>>>>>>> Maybe I wasn't clear. The actual number appears in the xml file, >>>>> e.g., >>>>>>>>> gmd:descriptiveKeywords[1] >>>>>>>>> Anyway, I've removed all [*] and I get the same correct result, >>>>> however >>>>>>>>> the >>>>>>>>> processing time is doubled... >>>>>>>>>> >>>>>>>>>>>> * In some cases, if you know that an element name is distinct, >> you >>>>> can >>>>>>>>>>>> get rid of all the explicit child steps and directly address >> the >>>>> node >>>>>>>>>>>> via the descendant axis. >>>>>>>>>>> Thanks, but it's not relevant in my case. >>>>>>>>>> Is it because the element names are not distinct? Or is it >> because >>>>>>>>>> your input form allows users to choose arbitrary paths for >> arbitrary >>>>>>>>>> documents? >>>>>>>>> The element names are not distinct. >>>>>>>>> >>>>>>>>>>> Sure, I'l also try BaseX 8.0 and compare. Should I recreate the >> db >>>>>>>>>>> importing >>>>>>>>>>> the xml files for testing the improved indexing? >>>>>>>>>> We have actually improved support for collections, but the >> database >>>>>>>>>> format itself has not changed, so it shouldn't make a difference >> in >>>>>>>>>> your case. >>>>>>>>>> >>>>>>>>>> Christian >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>> [1] http://files.basex.org/releases/latest >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Fri, Jan 30, 2015 at 3:55 PM, Menashè Eliezer >>>>>>>>>>>> <meliezer at ogs.trieste.it> wrote: >>>>>>>>>>>>> Hello, >>>>>>>>>>>>> I wonder if the attached query can be optimised. I'm attaching >>>>> all >>>>>>>>>>>>> relevant >>>>>>>>>>>>> information. >>>>>>>>>>>>> Basex 7.9, Debian, powerful server. >>>>>>>>>>>>> This is just an example. The queries will be built based on a >>>>>>>>>>>>> compilation >>>>>>>>>>>>> of >>>>>>>>>>>>> a search form. >>>>>>>>>>>>> Any help would be appreciated. >>>>>>>>>>>>> 40 seconds are not acceptable. >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> With kind regards, >>>>>>>>>>>>> Menashè >>>>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> With kind regards, >>>>>>>>>>> Menashè >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> With kind regards, >>>>>>>>> Menashè >>>>>>>>> >>>> -- >>>> Menashè

On Tue, Feb 3, 2015 at 1:13 PM, Menashè Eliezer <meliezer at ogs.trieste.it> wrote: > Hi Christian, > > Thank you! The performance arrives to 0.5 sec! > > The biggest improvement is related to the query rephrasing you've suggested. > Then the latest snapshot also helps a lot! > You may want to know that in the log of the latest snapshot I see > applying attribute index for "7827" > which is not clear to the user, instead of BaseX80-20150130.124009 which has > also used indexing: > applying attribute index for ("ALKY", "AYMD") > > I'm attaching the first and the second launch of the query using BaseXGUI. > Relaunching the same query reduces the time from over 1 second to 0.5 > second. > Some data: > BaseX80-20150130.124009 > Total Time: 30676.02 ms > After using "for $x in > collection("ALL-CDIS")/gmd:MD_Metadata/gmd:identificationInfo/sdn:SDN_DataIdentification": > Total Time: 5456.74 ms > applying attribute index for ("ALKY", "AYMD") in log. > Second launch: 1333.71 ms > Latest snapshot (BaseX80-20150202.121033): > 1st: Total Time: 1873.02 ms > 2nd: Total Time: 548.62 ms > > With kind regards, > Menashè > > > On 02/02/2015 02:02 PM, Menashè Eliezer wrote: >> >> Hi Christian, >> >> Thank you very much! Unfortunately I'll be at the office only tomorrow. >> >> Menashè >> >> On Sat, 31 Jan 2015 16:42:32 +0100, Christian Grün >> <christian.gruen at gmail.com> wrote: >>> >>> Hi Menashè, >>> >>> With the latest snapshot [1], your original query should now be >>> rewritten for index access as well. Looking forward to your tests, >>> >>> Christian >>> >>> PS: In terms of performance, it may still be worthwhile to move >>> redundant paths to the for clause; but just try and see. >>> >>> [1] http://files.basex.org/releases/latest/ >>> >>> >>> >>> On Fri, Jan 30, 2015 at 9:49 PM, Christian Grün >>> <christian.gruen at gmail.com> wrote: >>>> >>>> Hi Menashè, >>>> >>>>> Should I expect to see the usage of an index for each of the where >>> >>> phrases? >>>> >>>> Usually, only one predicate will be rewritten for index access, and >>>> the remaining conditions will be answered sequentially. >>>> >>>>> Have a nice weekend! >>>> >>>> Enjoy, >>>> Christian >>>> >>>> >>>>> Menashè >>>>> >>>>> On Fri, 30 Jan 2015 18:11:59 +0100, Christian Grün >>>>> <christian.gruen at gmail.com> wrote: >>>>>> >>>>>> Hi Menashè, >>>>>> >>>>>> Thanks for the XML samples you sent me in private. I noticed that the >>>>>> index rewritings will only be triggered if you formulate your query as >>>>>> follows: >>>>>> >>>>>> OLD: >>>>>> for $x in collection("ALL-CDIS") >>>>>> where $x/gmd:MD_Metadata/gmd:identificationInfo/... >>>>>> return ... >>>>>> >>>>>> NEW: >>>>>> for $x in collection("ALL-CDIS")/gmd:MD_Metadata >>>>>> where $x/gmd:identificationInfo/... >>>>>> return ... >>>>>> >>>>>> It's difficult to explain in short sentences why Variant 1 cannot be >>>>>> optimized that straightforward (basically, it's quite a different >>>>>> pattern to look for), but I'll check out if we can extend our matcher >>>>>> to also support these kind of queries. >>>>>> >>>>>> So, if possible, I would recommend you for now (and at least for >>>>>> testing) to move the root element test after the collection() >>>>>> function. I noticed that the first three child steps are the same in >>>>>> all of your conditions: >>>>>> >>>>>> gmd:MD_Metadata/gmd:identificationInfo/sdn:SDN_DataIdentification >>>>>> >>>>>> If that will be always be the case, it surely makes sense to move all >>>>>> of them to the "for" clause. >>>>>> >>>>>> Looking forward to your updated performance tests, >>>>>> Christian >>>>>> _______________________________ >>>>>> >>>>>> On Fri, Jan 30, 2015 at 5:55 PM, Christian Grün >>>>>> <christian.gruen at gmail.com> wrote: >>>>>>> >>>>>>> Could you possibly provide me with a small snapshot of your data >>>>>>> sources (one, two documents might be sufficient)? >>>>>>> >>>>>>> >>>>>>> On Fri, Jan 30, 2015 at 5:52 PM, Menashè Eliezer >>>>>>> <meliezer at ogs.trieste.it> wrote: >>>>>>>> >>>>>>>> Almost the same speed with version 8.0. >>>>>>>> No indexing (no "applying" in the query info). >>>>>>>> As I've attached before, indexes are active for this DB. >>>>>>>> >>>>>>>> With kind regards, >>>>>>>> Menashè >>>>>>>> >>>>>>>> >>>>>>>> On 01/30/2015 05:31 PM, Christian Grün wrote: >>>>>>>>> >>>>>>>>> It's indeed interesting that your query does not use any of the >>>>>>>>> existing index structures (if they did, you would find strings like >>>>>>>>> "applying text index" or "applying attribute index" in the query >>>>>>>>> info). Maybe/hopefully things look different with Version 8.0. >>>>>>>>> >>>>>>>>> >>>>>>>>> On Fri, Jan 30, 2015 at 5:26 PM, Menashè Eliezer >>>>>>>>> <meliezer at ogs.trieste.it> wrote: >>>>>>>>>> >>>>>>>>>> On 01/30/2015 05:18 PM, Christian Grün wrote: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >> >> /gmd:MD_Metadata/gmd:identificationInfo/sdn:SDN_DataIdentification/gmd:descriptiveKeywords[1]/gmd:MD_Keywords/gmd:keyword[2]/sdn:SDN_ParameterDiscoveryCode/@codeListValue >>>>>>>>>>>> >>>>>>>>>>>> How can I remove *? >>>>>>>>>>> >>>>>>>>>>> Simply remove the predicate; a[*]/b is the same as a/b. >>>>>>>>>> >>>>>>>>>> Maybe I wasn't clear. The actual number appears in the xml file, >>>>>> >>>>>> e.g., >>>>>>>>>> >>>>>>>>>> gmd:descriptiveKeywords[1] >>>>>>>>>> Anyway, I've removed all [*] and I get the same correct result, >>>>>> >>>>>> however >>>>>>>>>> >>>>>>>>>> the >>>>>>>>>> processing time is doubled... >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>> * In some cases, if you know that an element name is distinct, >>> >>> you >>>>>> >>>>>> can >>>>>>>>>>>>> >>>>>>>>>>>>> get rid of all the explicit child steps and directly address >>> >>> the >>>>>> >>>>>> node >>>>>>>>>>>>> >>>>>>>>>>>>> via the descendant axis. >>>>>>>>>>>> >>>>>>>>>>>> Thanks, but it's not relevant in my case. >>>>>>>>>>> >>>>>>>>>>> Is it because the element names are not distinct? Or is it >>> >>> because >>>>>>>>>>> >>>>>>>>>>> your input form allows users to choose arbitrary paths for >>> >>> arbitrary >>>>>>>>>>> >>>>>>>>>>> documents? >>>>>>>>>> >>>>>>>>>> The element names are not distinct. >>>>>>>>>> >>>>>>>>>>>> Sure, I'l also try BaseX 8.0 and compare. Should I recreate the >>> >>> db >>>>>>>>>>>> >>>>>>>>>>>> importing >>>>>>>>>>>> the xml files for testing the improved indexing? >>>>>>>>>>> >>>>>>>>>>> We have actually improved support for collections, but the >>> >>> database >>>>>>>>>>> >>>>>>>>>>> format itself has not changed, so it shouldn't make a difference >>> >>> in >>>>>>>>>>> >>>>>>>>>>> your case. >>>>>>>>>>> >>>>>>>>>>> Christian >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>> [1] http://files.basex.org/releases/latest >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Fri, Jan 30, 2015 at 3:55 PM, Menashè Eliezer >>>>>>>>>>>>> <meliezer at ogs.trieste.it> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hello, >>>>>>>>>>>>>> I wonder if the attached query can be optimised. I'm attaching >>>>>> >>>>>> all >>>>>>>>>>>>>> >>>>>>>>>>>>>> relevant >>>>>>>>>>>>>> information. >>>>>>>>>>>>>> Basex 7.9, Debian, powerful server. >>>>>>>>>>>>>> This is just an example. The queries will be built based on a >>>>>>>>>>>>>> compilation >>>>>>>>>>>>>> of >>>>>>>>>>>>>> a search form. >>>>>>>>>>>>>> Any help would be appreciated. >>>>>>>>>>>>>> 40 seconds are not acceptable. >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> With kind regards, >>>>>>>>>>>>>> Menashè >>>>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> With kind regards, >>>>>>>>>>>> Menashè >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>> With kind regards, >>>>>>>>>> Menashè >>>>>>>>>> >>>>> -- >>>>> Menashè > >

On 02/03/2015 01:13 PM, Menashè Eliezer wrote: > Hi Christian, > > Thank you! The performance arrives to 0.5 sec! > > The biggest improvement is related to the query rephrasing you've > suggested. > Then the latest snapshot also helps a lot! > You may want to know that in the log of the latest snapshot I see > applying attribute index for "7827" > which is not clear to the user, instead of BaseX80-20150130.124009 > which has also used indexing: > applying attribute index for ("ALKY", "AYMD") > > I'm attaching the first and the second launch of the query using > BaseXGUI. Relaunching the same query reduces the time from over 1 > second to 0.5 second. > Some data: > BaseX80-20150130.124009 > Total Time: 30676.02 ms > After using "for $x in > collection("ALL-CDIS")/gmd:MD_Metadata/gmd:identificationInfo/sdn:SDN_DataIdentification": > Total Time: 5456.74 ms > applying attribute index for ("ALKY", "AYMD") in log. > Second launch: 1333.71 ms > Latest snapshot (BaseX80-20150202.121033): > 1st: Total Time: 1873.02 ms > 2nd: Total Time: 548.62 ms > > With kind regards, > Menashè > > On 02/02/2015 02:02 PM, Menashè Eliezer wrote: >> Hi Christian, >> >> Thank you very much! Unfortunately I'll be at the office only tomorrow. >> >> Menashè >> >> On Sat, 31 Jan 2015 16:42:32 +0100, Christian Grün >> <christian.gruen at gmail.com> wrote: >>> Hi Menashè, >>> >>> With the latest snapshot [1], your original query should now be >>> rewritten for index access as well. Looking forward to your tests, >>> >>> Christian >>> >>> PS: In terms of performance, it may still be worthwhile to move >>> redundant paths to the for clause; but just try and see. >>> >>> [1] http://files.basex.org/releases/latest/ >>> >>> >>> >>> On Fri, Jan 30, 2015 at 9:49 PM, Christian Grün >>> <christian.gruen at gmail.com> wrote: >>>> Hi Menashè, >>>> >>>>> Should I expect to see the usage of an index for each of the where >>> phrases? >>>> Usually, only one predicate will be rewritten for index access, and >>>> the remaining conditions will be answered sequentially. >>>> >>>>> Have a nice weekend! >>>> Enjoy, >>>> Christian >>>> >>>> >>>>> Menashè >>>>> >>>>> On Fri, 30 Jan 2015 18:11:59 +0100, Christian Grün >>>>> <christian.gruen at gmail.com> wrote: >>>>>> Hi Menashè, >>>>>> >>>>>> Thanks for the XML samples you sent me in private. I noticed that >>>>>> the >>>>>> index rewritings will only be triggered if you formulate your >>>>>> query as >>>>>> follows: >>>>>> >>>>>> OLD: >>>>>> for $x in collection("ALL-CDIS") >>>>>> where $x/gmd:MD_Metadata/gmd:identificationInfo/... >>>>>> return ... >>>>>> >>>>>> NEW: >>>>>> for $x in collection("ALL-CDIS")/gmd:MD_Metadata >>>>>> where $x/gmd:identificationInfo/... >>>>>> return ... >>>>>> >>>>>> It's difficult to explain in short sentences why Variant 1 cannot be >>>>>> optimized that straightforward (basically, it's quite a different >>>>>> pattern to look for), but I'll check out if we can extend our >>>>>> matcher >>>>>> to also support these kind of queries. >>>>>> >>>>>> So, if possible, I would recommend you for now (and at least for >>>>>> testing) to move the root element test after the collection() >>>>>> function. I noticed that the first three child steps are the same in >>>>>> all of your conditions: >>>>>> >>>>>> gmd:MD_Metadata/gmd:identificationInfo/sdn:SDN_DataIdentification >>>>>> >>>>>> If that will be always be the case, it surely makes sense to move >>>>>> all >>>>>> of them to the "for" clause. >>>>>> >>>>>> Looking forward to your updated performance tests, >>>>>> Christian >>>>>> _______________________________ >>>>>> >>>>>> On Fri, Jan 30, 2015 at 5:55 PM, Christian Grün >>>>>> <christian.gruen at gmail.com> wrote: >>>>>>> Could you possibly provide me with a small snapshot of your data >>>>>>> sources (one, two documents might be sufficient)? >>>>>>> >>>>>>> >>>>>>> On Fri, Jan 30, 2015 at 5:52 PM, Menashè Eliezer >>>>>>> <meliezer at ogs.trieste.it> wrote: >>>>>>>> Almost the same speed with version 8.0. >>>>>>>> No indexing (no "applying" in the query info). >>>>>>>> As I've attached before, indexes are active for this DB. >>>>>>>> >>>>>>>> With kind regards, >>>>>>>> Menashè >>>>>>>> >>>>>>>> >>>>>>>> On 01/30/2015 05:31 PM, Christian Grün wrote: >>>>>>>>> It's indeed interesting that your query does not use any of the >>>>>>>>> existing index structures (if they did, you would find strings >>>>>>>>> like >>>>>>>>> "applying text index" or "applying attribute index" in the query >>>>>>>>> info). Maybe/hopefully things look different with Version 8.0. >>>>>>>>> >>>>>>>>> >>>>>>>>> On Fri, Jan 30, 2015 at 5:26 PM, Menashè Eliezer >>>>>>>>> <meliezer at ogs.trieste.it> wrote: >>>>>>>>>> On 01/30/2015 05:18 PM, Christian Grün wrote: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >> /gmd:MD_Metadata/gmd:identificationInfo/sdn:SDN_DataIdentification/gmd:descriptiveKeywords[1]/gmd:MD_Keywords/gmd:keyword[2]/sdn:SDN_ParameterDiscoveryCode/@codeListValue >> >>>>>>>>>>>> How can I remove *? >>>>>>>>>>> Simply remove the predicate; a[*]/b is the same as a/b. >>>>>>>>>> Maybe I wasn't clear. The actual number appears in the xml file, >>>>>> e.g., >>>>>>>>>> gmd:descriptiveKeywords[1] >>>>>>>>>> Anyway, I've removed all [*] and I get the same correct result, >>>>>> however >>>>>>>>>> the >>>>>>>>>> processing time is doubled... >>>>>>>>>>> >>>>>>>>>>>>> * In some cases, if you know that an element name is >>>>>>>>>>>>> distinct, >>> you >>>>>> can >>>>>>>>>>>>> get rid of all the explicit child steps and directly address >>> the >>>>>> node >>>>>>>>>>>>> via the descendant axis. >>>>>>>>>>>> Thanks, but it's not relevant in my case. >>>>>>>>>>> Is it because the element names are not distinct? Or is it >>> because >>>>>>>>>>> your input form allows users to choose arbitrary paths for >>> arbitrary >>>>>>>>>>> documents? >>>>>>>>>> The element names are not distinct. >>>>>>>>>> >>>>>>>>>>>> Sure, I'l also try BaseX 8.0 and compare. Should I recreate >>>>>>>>>>>> the >>> db >>>>>>>>>>>> importing >>>>>>>>>>>> the xml files for testing the improved indexing? >>>>>>>>>>> We have actually improved support for collections, but the >>> database >>>>>>>>>>> format itself has not changed, so it shouldn't make a >>>>>>>>>>> difference >>> in >>>>>>>>>>> your case. >>>>>>>>>>> >>>>>>>>>>> Christian >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>> [1] http://files.basex.org/releases/latest >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Fri, Jan 30, 2015 at 3:55 PM, Menashè Eliezer >>>>>>>>>>>>> <meliezer at ogs.trieste.it> wrote: >>>>>>>>>>>>>> Hello, >>>>>>>>>>>>>> I wonder if the attached query can be optimised. I'm >>>>>>>>>>>>>> attaching >>>>>> all >>>>>>>>>>>>>> relevant >>>>>>>>>>>>>> information. >>>>>>>>>>>>>> Basex 7.9, Debian, powerful server. >>>>>>>>>>>>>> This is just an example. The queries will be built based >>>>>>>>>>>>>> on a >>>>>>>>>>>>>> compilation >>>>>>>>>>>>>> of >>>>>>>>>>>>>> a search form. >>>>>>>>>>>>>> Any help would be appreciated. >>>>>>>>>>>>>> 40 seconds are not acceptable. >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> With kind regards, >>>>>>>>>>>>>> Menashè >>>>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> With kind regards, >>>>>>>>>>>> Menashè >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>> With kind regards, >>>>>>>>>> Menashè >>>>>>>>>> >>>>> -- >>>>> Menashè

On 06/22/2015 04:48 PM, Menashè Eliezer wrote: > Hi Christian, > I'm have again performance problems. I have BaseX 8.2.1. > As you may remember, you've recommended changing > 'for $x in collection("CDI")' to 'for $x in > collection("CDI")/gmd:MD_Metadata/gmd:identificationInfo/sdn:SDN_DataIdentification'. > However, I've discovered I cannot specify XPath while working with IDs > (db:node-pre). > It's a multi-step process: client program sends to the server the > search filter defined by end-user and get IDs. > Then there are several queries for getting different information about > this specific subset. Instead of redefining the filters, the only > condition is > where db:node-pre($x)=$ids > for having a better performance. > Once I specific XPath, it seems that the ids have no meaning. The > resultset is always empty once they are being used. > So, I've returned to use 'for $x in collection("CDI")' in the first > query of getting all IDs, but the performance is extremely low. > > **I'm attaching the query and its related info using BaseXGUI (local > server) with much smaller database. The performance seems ok. > > I'm using your BaseXClient.java, however I see the delay already in > the BaseX server logs: > QUERY[0] xquery version "3.0"; declare namespace queryName ='GetIDS'; > declare namespace gco = "http://www.isotc211.org/2005/gco"; declare > namespace gmd = "http://www.isotc211.org/2005/gmd"; declare > namespace gml = "http://www.opengis.net/gml"; declare namespace > gmx="http://www.isotc211.org/2005/gmx"; declare namespace sdn = > "http://www.seadatanet.org"; dec lare namespace fn = > "http://www.w3.org/2005/xpath-functions"; declare namespace xs = > "http://www.w3.org/2001/XMLSchema"; declare namespace output = > "http://www.w3.org/2010/xslt-xquery-serialization"; declare option > output:method 'xml';declare option output:item-separator ","; let $db > := db:open("CDI") for $x in $db where > $x/gmd:MD_Metadata/gmd:identificationInfo/sdn:SDN_DataIdentification/gmd:extent/gmd:EX_Exte > nt/gmd:geographicElement/gmd:EX_GeographicBoundingBox/gmd:westBoundLongitude/gco:Decimal>="-5.8447265625" > and > $x/gmd:MD_Metadata/gmd:identificationInfo/sdn:SDN_DataIdentification/gmd:extent/gmd:EX_Extent/gmd > :geographicElement... 0.17 ms > 110 16:36:09.713 192.168.155.30:39211 admin OK RESULTS[0] > 25957.11 ms > > Then I have other slow queries, but I would like to focus in this > phase on the biggest delay. > Server: Java 1.7.0_79, VM="-XX:MaxPermSize=512m -Xms3096m -Xmx3096m" > The network layer between client and server is very fast. > > P.S. > Id there an undocumented way to log the full xquery in BaseX server logs? > I've seen the -V option, but I don't use the standalone version, but: > java -cp /usr/share/java/basex.jar org.basex.BaseXServer > -d doesn't give me extra query info. > > > With kind regards, > Menashè > > On 02/03/2015 01:13 PM, Menashè Eliezer wrote: >> Hi Christian, >> >> Thank you! The performance arrives to 0.5 sec! >> >> The biggest improvement is related to the query rephrasing you've >> suggested. >> Then the latest snapshot also helps a lot! >> You may want to know that in the log of the latest snapshot I see >> applying attribute index for "7827" >> which is not clear to the user, instead of BaseX80-20150130.124009 >> which has also used indexing: >> applying attribute index for ("ALKY", "AYMD") >> >> I'm attaching the first and the second launch of the query using >> BaseXGUI. Relaunching the same query reduces the time from over 1 >> second to 0.5 second. >> Some data: >> BaseX80-20150130.124009 >> Total Time: 30676.02 ms >> After using "for $x in >> collection("ALL-CDIS")/gmd:MD_Metadata/gmd:identificationInfo/sdn:SDN_DataIdentification": >> Total Time: 5456.74 ms >> applying attribute index for ("ALKY", "AYMD") in log. >> Second launch: 1333.71 ms >> Latest snapshot (BaseX80-20150202.121033): >> 1st: Total Time: 1873.02 ms >> 2nd: Total Time: 548.62 ms >> >> With kind regards, >> Menashè >> >> On 02/02/2015 02:02 PM, Menashè Eliezer wrote: >>> Hi Christian, >>> >>> Thank you very much! Unfortunately I'll be at the office only tomorrow. >>> >>> Menashè >>> >>> On Sat, 31 Jan 2015 16:42:32 +0100, Christian Grün >>> <christian.gruen at gmail.com> wrote: >>>> Hi Menashè, >>>> >>>> With the latest snapshot [1], your original query should now be >>>> rewritten for index access as well. Looking forward to your tests, >>>> >>>> Christian >>>> >>>> PS: In terms of performance, it may still be worthwhile to move >>>> redundant paths to the for clause; but just try and see. >>>> >>>> [1] http://files.basex.org/releases/latest/ >>>> >>>> >>>> >>>> On Fri, Jan 30, 2015 at 9:49 PM, Christian Grün >>>> <christian.gruen at gmail.com> wrote: >>>>> Hi Menashè, >>>>> >>>>>> Should I expect to see the usage of an index for each of the where >>>> phrases? >>>>> Usually, only one predicate will be rewritten for index access, and >>>>> the remaining conditions will be answered sequentially. >>>>> >>>>>> Have a nice weekend! >>>>> Enjoy, >>>>> Christian >>>>> >>>>> >>>>>> Menashè >>>>>> >>>>>> On Fri, 30 Jan 2015 18:11:59 +0100, Christian Grün >>>>>> <christian.gruen at gmail.com> wrote: >>>>>>> Hi Menashè, >>>>>>> >>>>>>> Thanks for the XML samples you sent me in private. I noticed >>>>>>> that the >>>>>>> index rewritings will only be triggered if you formulate your >>>>>>> query as >>>>>>> follows: >>>>>>> >>>>>>> OLD: >>>>>>> for $x in collection("ALL-CDIS") >>>>>>> where $x/gmd:MD_Metadata/gmd:identificationInfo/... >>>>>>> return ... >>>>>>> >>>>>>> NEW: >>>>>>> for $x in collection("ALL-CDIS")/gmd:MD_Metadata >>>>>>> where $x/gmd:identificationInfo/... >>>>>>> return ... >>>>>>> >>>>>>> It's difficult to explain in short sentences why Variant 1 >>>>>>> cannot be >>>>>>> optimized that straightforward (basically, it's quite a different >>>>>>> pattern to look for), but I'll check out if we can extend our >>>>>>> matcher >>>>>>> to also support these kind of queries. >>>>>>> >>>>>>> So, if possible, I would recommend you for now (and at least for >>>>>>> testing) to move the root element test after the collection() >>>>>>> function. I noticed that the first three child steps are the >>>>>>> same in >>>>>>> all of your conditions: >>>>>>> >>>>>>> gmd:MD_Metadata/gmd:identificationInfo/sdn:SDN_DataIdentification >>>>>>> >>>>>>> If that will be always be the case, it surely makes sense to >>>>>>> move all >>>>>>> of them to the "for" clause. >>>>>>> >>>>>>> Looking forward to your updated performance tests, >>>>>>> Christian >>>>>>> _______________________________ >>>>>>> >>>>>>> On Fri, Jan 30, 2015 at 5:55 PM, Christian Grün >>>>>>> <christian.gruen at gmail.com> wrote: >>>>>>>> Could you possibly provide me with a small snapshot of your data >>>>>>>> sources (one, two documents might be sufficient)? >>>>>>>> >>>>>>>> >>>>>>>> On Fri, Jan 30, 2015 at 5:52 PM, Menashè Eliezer >>>>>>>> <meliezer at ogs.trieste.it> wrote: >>>>>>>>> Almost the same speed with version 8.0. >>>>>>>>> No indexing (no "applying" in the query info). >>>>>>>>> As I've attached before, indexes are active for this DB. >>>>>>>>> >>>>>>>>> With kind regards, >>>>>>>>> Menashè >>>>>>>>> >>>>>>>>> >>>>>>>>> On 01/30/2015 05:31 PM, Christian Grün wrote: >>>>>>>>>> It's indeed interesting that your query does not use any of the >>>>>>>>>> existing index structures (if they did, you would find >>>>>>>>>> strings like >>>>>>>>>> "applying text index" or "applying attribute index" in the query >>>>>>>>>> info). Maybe/hopefully things look different with Version 8.0. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Fri, Jan 30, 2015 at 5:26 PM, Menashè Eliezer >>>>>>>>>> <meliezer at ogs.trieste.it> wrote: >>>>>>>>>>> On 01/30/2015 05:18 PM, Christian Grün wrote: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>> /gmd:MD_Metadata/gmd:identificationInfo/sdn:SDN_DataIdentification/gmd:descriptiveKeywords[1]/gmd:MD_Keywords/gmd:keyword[2]/sdn:SDN_ParameterDiscoveryCode/@codeListValue >>> >>>>>>>>>>>>> How can I remove *? >>>>>>>>>>>> Simply remove the predicate; a[*]/b is the same as a/b. >>>>>>>>>>> Maybe I wasn't clear. The actual number appears in the xml >>>>>>>>>>> file, >>>>>>> e.g., >>>>>>>>>>> gmd:descriptiveKeywords[1] >>>>>>>>>>> Anyway, I've removed all [*] and I get the same correct result, >>>>>>> however >>>>>>>>>>> the >>>>>>>>>>> processing time is doubled... >>>>>>>>>>>> >>>>>>>>>>>>>> * In some cases, if you know that an element name is >>>>>>>>>>>>>> distinct, >>>> you >>>>>>> can >>>>>>>>>>>>>> get rid of all the explicit child steps and directly address >>>> the >>>>>>> node >>>>>>>>>>>>>> via the descendant axis. >>>>>>>>>>>>> Thanks, but it's not relevant in my case. >>>>>>>>>>>> Is it because the element names are not distinct? Or is it >>>> because >>>>>>>>>>>> your input form allows users to choose arbitrary paths for >>>> arbitrary >>>>>>>>>>>> documents? >>>>>>>>>>> The element names are not distinct. >>>>>>>>>>> >>>>>>>>>>>>> Sure, I'l also try BaseX 8.0 and compare. Should I >>>>>>>>>>>>> recreate the >>>> db >>>>>>>>>>>>> importing >>>>>>>>>>>>> the xml files for testing the improved indexing? >>>>>>>>>>>> We have actually improved support for collections, but the >>>> database >>>>>>>>>>>> format itself has not changed, so it shouldn't make a >>>>>>>>>>>> difference >>>> in >>>>>>>>>>>> your case. >>>>>>>>>>>> >>>>>>>>>>>> Christian >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>> [1] http://files.basex.org/releases/latest >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, Jan 30, 2015 at 3:55 PM, Menashè Eliezer >>>>>>>>>>>>>> <meliezer at ogs.trieste.it> wrote: >>>>>>>>>>>>>>> Hello, >>>>>>>>>>>>>>> I wonder if the attached query can be optimised. I'm >>>>>>>>>>>>>>> attaching >>>>>>> all >>>>>>>>>>>>>>> relevant >>>>>>>>>>>>>>> information. >>>>>>>>>>>>>>> Basex 7.9, Debian, powerful server. >>>>>>>>>>>>>>> This is just an example. The queries will be built based >>>>>>>>>>>>>>> on a >>>>>>>>>>>>>>> compilation >>>>>>>>>>>>>>> of >>>>>>>>>>>>>>> a search form. >>>>>>>>>>>>>>> Any help would be appreciated. >>>>>>>>>>>>>>> 40 seconds are not acceptable. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> With kind regards, >>>>>>>>>>>>>>> Menashè >>>>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> With kind regards, >>>>>>>>>>>>> Menashè >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>> With kind regards, >>>>>>>>>>> Menashè >>>>>>>>>>> >>>>>> -- >>>>>> Menashè

>> QUERY[0] xquery version "3.0"; declare namespace queryName ='GetIDS'; >> declare namespace gco = "http://www.isotc211.org/2005/gco"; declare >> [...]

>> Id there an undocumented way to log the full xquery in BaseX server logs?

>> I've seen the -V option, but I don't use the standalone version, but: >> java -cp /usr/share/java/basex.jar org.basex.BaseXServer >> -d doesn't give me extra query info. >> >> >> With kind regards, >> Menashè >> >> On 02/03/2015 01:13 PM, Menashè Eliezer wrote: >>> >>> Hi Christian, >>> >>> Thank you! The performance arrives to 0.5 sec! >>> >>> The biggest improvement is related to the query rephrasing you've >>> suggested. >>> Then the latest snapshot also helps a lot! >>> You may want to know that in the log of the latest snapshot I see >>> applying attribute index for "7827" >>> which is not clear to the user, instead of BaseX80-20150130.124009 which >>> has also used indexing: >>> applying attribute index for ("ALKY", "AYMD") >>> >>> I'm attaching the first and the second launch of the query using >>> BaseXGUI. Relaunching the same query reduces the time from over 1 second to >>> 0.5 second. >>> Some data: >>> BaseX80-20150130.124009 >>> Total Time: 30676.02 ms >>> After using "for $x in >>> collection("ALL-CDIS")/gmd:MD_Metadata/gmd:identificationInfo/sdn:SDN_DataIdentification": >>> Total Time: 5456.74 ms >>> applying attribute index for ("ALKY", "AYMD") in log. >>> Second launch: 1333.71 ms >>> Latest snapshot (BaseX80-20150202.121033): >>> 1st: Total Time: 1873.02 ms >>> 2nd: Total Time: 548.62 ms >>> >>> With kind regards, >>> Menashè >>> >>> On 02/02/2015 02:02 PM, Menashè Eliezer wrote: >>>> >>>> Hi Christian, >>>> >>>> Thank you very much! Unfortunately I'll be at the office only tomorrow. >>>> >>>> Menashè >>>> >>>> On Sat, 31 Jan 2015 16:42:32 +0100, Christian Grün >>>> <christian.gruen at gmail.com> wrote: >>>>> >>>>> Hi Menashè, >>>>> >>>>> With the latest snapshot [1], your original query should now be >>>>> rewritten for index access as well. Looking forward to your tests, >>>>> >>>>> Christian >>>>> >>>>> PS: In terms of performance, it may still be worthwhile to move >>>>> redundant paths to the for clause; but just try and see. >>>>> >>>>> [1] http://files.basex.org/releases/latest/ >>>>> >>>>> >>>>> >>>>> On Fri, Jan 30, 2015 at 9:49 PM, Christian Grün >>>>> <christian.gruen at gmail.com> wrote: >>>>>> >>>>>> Hi Menashè, >>>>>> >>>>>>> Should I expect to see the usage of an index for each of the where >>>>> >>>>> phrases? >>>>>> >>>>>> Usually, only one predicate will be rewritten for index access, and >>>>>> the remaining conditions will be answered sequentially. >>>>>> >>>>>>> Have a nice weekend! >>>>>> >>>>>> Enjoy, >>>>>> Christian >>>>>> >>>>>> >>>>>>> Menashè >>>>>>> >>>>>>> On Fri, 30 Jan 2015 18:11:59 +0100, Christian Grün >>>>>>> <christian.gruen at gmail.com> wrote: >>>>>>>> >>>>>>>> Hi Menashè, >>>>>>>> >>>>>>>> Thanks for the XML samples you sent me in private. I noticed that >>>>>>>> the >>>>>>>> index rewritings will only be triggered if you formulate your query >>>>>>>> as >>>>>>>> follows: >>>>>>>> >>>>>>>> OLD: >>>>>>>> for $x in collection("ALL-CDIS") >>>>>>>> where $x/gmd:MD_Metadata/gmd:identificationInfo/... >>>>>>>> return ... >>>>>>>> >>>>>>>> NEW: >>>>>>>> for $x in collection("ALL-CDIS")/gmd:MD_Metadata >>>>>>>> where $x/gmd:identificationInfo/... >>>>>>>> return ... >>>>>>>> >>>>>>>> It's difficult to explain in short sentences why Variant 1 cannot be >>>>>>>> optimized that straightforward (basically, it's quite a different >>>>>>>> pattern to look for), but I'll check out if we can extend our >>>>>>>> matcher >>>>>>>> to also support these kind of queries. >>>>>>>> >>>>>>>> So, if possible, I would recommend you for now (and at least for >>>>>>>> testing) to move the root element test after the collection() >>>>>>>> function. I noticed that the first three child steps are the same in >>>>>>>> all of your conditions: >>>>>>>> >>>>>>>> gmd:MD_Metadata/gmd:identificationInfo/sdn:SDN_DataIdentification >>>>>>>> >>>>>>>> If that will be always be the case, it surely makes sense to move >>>>>>>> all >>>>>>>> of them to the "for" clause. >>>>>>>> >>>>>>>> Looking forward to your updated performance tests, >>>>>>>> Christian >>>>>>>> _______________________________ >>>>>>>> >>>>>>>> On Fri, Jan 30, 2015 at 5:55 PM, Christian Grün >>>>>>>> <christian.gruen at gmail.com> wrote: >>>>>>>>> >>>>>>>>> Could you possibly provide me with a small snapshot of your data >>>>>>>>> sources (one, two documents might be sufficient)? >>>>>>>>> >>>>>>>>> >>>>>>>>> On Fri, Jan 30, 2015 at 5:52 PM, Menashè Eliezer >>>>>>>>> <meliezer at ogs.trieste.it> wrote: >>>>>>>>>> >>>>>>>>>> Almost the same speed with version 8.0. >>>>>>>>>> No indexing (no "applying" in the query info). >>>>>>>>>> As I've attached before, indexes are active for this DB. >>>>>>>>>> >>>>>>>>>> With kind regards, >>>>>>>>>> Menashè >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 01/30/2015 05:31 PM, Christian Grün wrote: >>>>>>>>>>> >>>>>>>>>>> It's indeed interesting that your query does not use any of the >>>>>>>>>>> existing index structures (if they did, you would find strings >>>>>>>>>>> like >>>>>>>>>>> "applying text index" or "applying attribute index" in the query >>>>>>>>>>> info). Maybe/hopefully things look different with Version 8.0. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Fri, Jan 30, 2015 at 5:26 PM, Menashè Eliezer >>>>>>>>>>> <meliezer at ogs.trieste.it> wrote: >>>>>>>>>>>> >>>>>>>>>>>> On 01/30/2015 05:18 PM, Christian Grün wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>> >>>> /gmd:MD_Metadata/gmd:identificationInfo/sdn:SDN_DataIdentification/gmd:descriptiveKeywords[1]/gmd:MD_Keywords/gmd:keyword[2]/sdn:SDN_ParameterDiscoveryCode/@codeListValue >>>>>>>>>>>>>> >>>>>>>>>>>>>> How can I remove *? >>>>>>>>>>>>> >>>>>>>>>>>>> Simply remove the predicate; a[*]/b is the same as a/b. >>>>>>>>>>>> >>>>>>>>>>>> Maybe I wasn't clear. The actual number appears in the xml file, >>>>>>>> >>>>>>>> e.g., >>>>>>>>>>>> >>>>>>>>>>>> gmd:descriptiveKeywords[1] >>>>>>>>>>>> Anyway, I've removed all [*] and I get the same correct result, >>>>>>>> >>>>>>>> however >>>>>>>>>>>> >>>>>>>>>>>> the >>>>>>>>>>>> processing time is doubled... >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>>> * In some cases, if you know that an element name is >>>>>>>>>>>>>>> distinct, >>>>> >>>>> you >>>>>>>> >>>>>>>> can >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> get rid of all the explicit child steps and directly address >>>>> >>>>> the >>>>>>>> >>>>>>>> node >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> via the descendant axis. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, but it's not relevant in my case. >>>>>>>>>>>>> >>>>>>>>>>>>> Is it because the element names are not distinct? Or is it >>>>> >>>>> because >>>>>>>>>>>>> >>>>>>>>>>>>> your input form allows users to choose arbitrary paths for >>>>> >>>>> arbitrary >>>>>>>>>>>>> >>>>>>>>>>>>> documents? >>>>>>>>>>>> >>>>>>>>>>>> The element names are not distinct. >>>>>>>>>>>> >>>>>>>>>>>>>> Sure, I'l also try BaseX 8.0 and compare. Should I recreate >>>>>>>>>>>>>> the >>>>> >>>>> db >>>>>>>>>>>>>> >>>>>>>>>>>>>> importing >>>>>>>>>>>>>> the xml files for testing the improved indexing? >>>>>>>>>>>>> >>>>>>>>>>>>> We have actually improved support for collections, but the >>>>> >>>>> database >>>>>>>>>>>>> >>>>>>>>>>>>> format itself has not changed, so it shouldn't make a >>>>>>>>>>>>> difference >>>>> >>>>> in >>>>>>>>>>>>> >>>>>>>>>>>>> your case. >>>>>>>>>>>>> >>>>>>>>>>>>> Christian >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>>> [1] http://files.basex.org/releases/latest >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Fri, Jan 30, 2015 at 3:55 PM, Menashè Eliezer >>>>>>>>>>>>>>> <meliezer at ogs.trieste.it> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hello, >>>>>>>>>>>>>>>> I wonder if the attached query can be optimised. I'm >>>>>>>>>>>>>>>> attaching >>>>>>>> >>>>>>>> all >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> relevant >>>>>>>>>>>>>>>> information. >>>>>>>>>>>>>>>> Basex 7.9, Debian, powerful server. >>>>>>>>>>>>>>>> This is just an example. The queries will be built based on >>>>>>>>>>>>>>>> a >>>>>>>>>>>>>>>> compilation >>>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>> a search form. >>>>>>>>>>>>>>>> Any help would be appreciated. >>>>>>>>>>>>>>>> 40 seconds are not acceptable. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> With kind regards, >>>>>>>>>>>>>>>> Menashè >>>>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> With kind regards, >>>>>>>>>>>>>> Menashè >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>> With kind regards, >>>>>>>>>>>> Menashè >>>>>>>>>>>> >>>>>>> -- >>>>>>> Menashè > >

On 06/22/2015 07:57 PM, Christian Grün wrote: > Hi Menashè, > >>> QUERY[0] xquery version "3.0"; declare namespace queryName ='GetIDS'; >>> declare namespace gco = "http://www.isotc211.org/2005/gco"; declare >>> [...] > It would be great if you could help us and simplify the query, such > that we can have a look at the core issue. > >>> Id there an undocumented way to log the full xquery in BaseX server logs? > The maximum size of log entries can be adjusted via the option LOGMSGMAXLEN [1]. > > Cheers, > Christian > > [1] http://docs.basex.org/wiki/Options#LOGMSGMAXLEN > > > >>> I've seen the -V option, but I don't use the standalone version, but: >>> java -cp /usr/share/java/basex.jar org.basex.BaseXServer >>> -d doesn't give me extra query info. >>> >>> >>> With kind regards, >>> Menashè >>> >>> On 02/03/2015 01:13 PM, Menashè Eliezer wrote: >>>> Hi Christian, >>>> >>>> Thank you! The performance arrives to 0.5 sec! >>>> >>>> The biggest improvement is related to the query rephrasing you've >>>> suggested. >>>> Then the latest snapshot also helps a lot! >>>> You may want to know that in the log of the latest snapshot I see >>>> applying attribute index for "7827" >>>> which is not clear to the user, instead of BaseX80-20150130.124009 which >>>> has also used indexing: >>>> applying attribute index for ("ALKY", "AYMD") >>>> >>>> I'm attaching the first and the second launch of the query using >>>> BaseXGUI. Relaunching the same query reduces the time from over 1 second to >>>> 0.5 second. >>>> Some data: >>>> BaseX80-20150130.124009 >>>> Total Time: 30676.02 ms >>>> After using "for $x in >>>> collection("ALL-CDIS")/gmd:MD_Metadata/gmd:identificationInfo/sdn:SDN_DataIdentification": >>>> Total Time: 5456.74 ms >>>> applying attribute index for ("ALKY", "AYMD") in log. >>>> Second launch: 1333.71 ms >>>> Latest snapshot (BaseX80-20150202.121033): >>>> 1st: Total Time: 1873.02 ms >>>> 2nd: Total Time: 548.62 ms >>>> >>>> With kind regards, >>>> Menashè >>>> >>>> On 02/02/2015 02:02 PM, Menashè Eliezer wrote: >>>>> Hi Christian, >>>>> >>>>> Thank you very much! Unfortunately I'll be at the office only tomorrow. >>>>> >>>>> Menashè >>>>> >>>>> On Sat, 31 Jan 2015 16:42:32 +0100, Christian Grün >>>>> <christian.gruen at gmail.com> wrote: >>>>>> Hi Menashè, >>>>>> >>>>>> With the latest snapshot [1], your original query should now be >>>>>> rewritten for index access as well. Looking forward to your tests, >>>>>> >>>>>> Christian >>>>>> >>>>>> PS: In terms of performance, it may still be worthwhile to move >>>>>> redundant paths to the for clause; but just try and see. >>>>>> >>>>>> [1] http://files.basex.org/releases/latest/ >>>>>> >>>>>> >>>>>> >>>>>> On Fri, Jan 30, 2015 at 9:49 PM, Christian Grün >>>>>> <christian.gruen at gmail.com> wrote: >>>>>>> Hi Menashè, >>>>>>> >>>>>>>> Should I expect to see the usage of an index for each of the where >>>>>> phrases? >>>>>>> Usually, only one predicate will be rewritten for index access, and >>>>>>> the remaining conditions will be answered sequentially. >>>>>>> >>>>>>>> Have a nice weekend! >>>>>>> Enjoy, >>>>>>> Christian >>>>>>> >>>>>>> >>>>>>>> Menashè >>>>>>>> >>>>>>>> On Fri, 30 Jan 2015 18:11:59 +0100, Christian Grün >>>>>>>> <christian.gruen at gmail.com> wrote: >>>>>>>>> Hi Menashè, >>>>>>>>> >>>>>>>>> Thanks for the XML samples you sent me in private. I noticed that >>>>>>>>> the >>>>>>>>> index rewritings will only be triggered if you formulate your query >>>>>>>>> as >>>>>>>>> follows: >>>>>>>>> >>>>>>>>> OLD: >>>>>>>>> for $x in collection("ALL-CDIS") >>>>>>>>> where $x/gmd:MD_Metadata/gmd:identificationInfo/... >>>>>>>>> return ... >>>>>>>>> >>>>>>>>> NEW: >>>>>>>>> for $x in collection("ALL-CDIS")/gmd:MD_Metadata >>>>>>>>> where $x/gmd:identificationInfo/... >>>>>>>>> return ... >>>>>>>>> >>>>>>>>> It's difficult to explain in short sentences why Variant 1 cannot be >>>>>>>>> optimized that straightforward (basically, it's quite a different >>>>>>>>> pattern to look for), but I'll check out if we can extend our >>>>>>>>> matcher >>>>>>>>> to also support these kind of queries. >>>>>>>>> >>>>>>>>> So, if possible, I would recommend you for now (and at least for >>>>>>>>> testing) to move the root element test after the collection() >>>>>>>>> function. I noticed that the first three child steps are the same in >>>>>>>>> all of your conditions: >>>>>>>>> >>>>>>>>> gmd:MD_Metadata/gmd:identificationInfo/sdn:SDN_DataIdentification >>>>>>>>> >>>>>>>>> If that will be always be the case, it surely makes sense to move >>>>>>>>> all >>>>>>>>> of them to the "for" clause. >>>>>>>>> >>>>>>>>> Looking forward to your updated performance tests, >>>>>>>>> Christian >>>>>>>>> _______________________________ >>>>>>>>> >>>>>>>>> On Fri, Jan 30, 2015 at 5:55 PM, Christian Grün >>>>>>>>> <christian.gruen at gmail.com> wrote: >>>>>>>>>> Could you possibly provide me with a small snapshot of your data >>>>>>>>>> sources (one, two documents might be sufficient)? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Fri, Jan 30, 2015 at 5:52 PM, Menashè Eliezer >>>>>>>>>> <meliezer at ogs.trieste.it> wrote: >>>>>>>>>>> Almost the same speed with version 8.0. >>>>>>>>>>> No indexing (no "applying" in the query info). >>>>>>>>>>> As I've attached before, indexes are active for this DB. >>>>>>>>>>> >>>>>>>>>>> With kind regards, >>>>>>>>>>> Menashè >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On 01/30/2015 05:31 PM, Christian Grün wrote: >>>>>>>>>>>> It's indeed interesting that your query does not use any of the >>>>>>>>>>>> existing index structures (if they did, you would find strings >>>>>>>>>>>> like >>>>>>>>>>>> "applying text index" or "applying attribute index" in the query >>>>>>>>>>>> info). Maybe/hopefully things look different with Version 8.0. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Fri, Jan 30, 2015 at 5:26 PM, Menashè Eliezer >>>>>>>>>>>> <meliezer at ogs.trieste.it> wrote: >>>>>>>>>>>>> On 01/30/2015 05:18 PM, Christian Grün wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>> /gmd:MD_Metadata/gmd:identificationInfo/sdn:SDN_DataIdentification/gmd:descriptiveKeywords[1]/gmd:MD_Keywords/gmd:keyword[2]/sdn:SDN_ParameterDiscoveryCode/@codeListValue >>>>>>>>>>>>>>> How can I remove *? >>>>>>>>>>>>>> Simply remove the predicate; a[*]/b is the same as a/b. >>>>>>>>>>>>> Maybe I wasn't clear. The actual number appears in the xml file, >>>>>>>>> e.g., >>>>>>>>>>>>> gmd:descriptiveKeywords[1] >>>>>>>>>>>>> Anyway, I've removed all [*] and I get the same correct result, >>>>>>>>> however >>>>>>>>>>>>> the >>>>>>>>>>>>> processing time is doubled... >>>>>>>>>>>>>> >>>>>>>>>>>>>>>> * In some cases, if you know that an element name is >>>>>>>>>>>>>>>> distinct, >>>>>> you >>>>>>>>> can >>>>>>>>>>>>>>>> get rid of all the explicit child steps and directly address >>>>>> the >>>>>>>>> node >>>>>>>>>>>>>>>> via the descendant axis. >>>>>>>>>>>>>>> Thanks, but it's not relevant in my case. >>>>>>>>>>>>>> Is it because the element names are not distinct? Or is it >>>>>> because >>>>>>>>>>>>>> your input form allows users to choose arbitrary paths for >>>>>> arbitrary >>>>>>>>>>>>>> documents? >>>>>>>>>>>>> The element names are not distinct. >>>>>>>>>>>>> >>>>>>>>>>>>>>> Sure, I'l also try BaseX 8.0 and compare. Should I recreate >>>>>>>>>>>>>>> the >>>>>> db >>>>>>>>>>>>>>> importing >>>>>>>>>>>>>>> the xml files for testing the improved indexing? >>>>>>>>>>>>>> We have actually improved support for collections, but the >>>>>> database >>>>>>>>>>>>>> format itself has not changed, so it shouldn't make a >>>>>>>>>>>>>> difference >>>>>> in >>>>>>>>>>>>>> your case. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Christian >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>>> [1] http://files.basex.org/releases/latest >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Fri, Jan 30, 2015 at 3:55 PM, Menashè Eliezer >>>>>>>>>>>>>>>> <meliezer at ogs.trieste.it> wrote: >>>>>>>>>>>>>>>>> Hello, >>>>>>>>>>>>>>>>> I wonder if the attached query can be optimised. I'm >>>>>>>>>>>>>>>>> attaching >>>>>>>>> all >>>>>>>>>>>>>>>>> relevant >>>>>>>>>>>>>>>>> information. >>>>>>>>>>>>>>>>> Basex 7.9, Debian, powerful server. >>>>>>>>>>>>>>>>> This is just an example. The queries will be built based on >>>>>>>>>>>>>>>>> a >>>>>>>>>>>>>>>>> compilation >>>>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>> a search form. >>>>>>>>>>>>>>>>> Any help would be appreciated. >>>>>>>>>>>>>>>>> 40 seconds are not acceptable. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>> With kind regards, >>>>>>>>>>>>>>>>> Menashè >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> With kind regards, >>>>>>>>>>>>>>> Menashè >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>> With kind regards, >>>>>>>>>>>>> Menashè >>>>>>>>>>>>> >>>>>>>> -- >>>>>>>> Menashè >>

> Once I specific XPath, it seems that the ids have no meaning. The resultset is always empty once they are being used.

On Mon, Jun 22, 2015 at 4:48 PM, Menashè Eliezer <meliezer at ogs.trieste.it> wrote: > Hi Christian, > I'm have again performance problems. I have BaseX 8.2.1. > As you may remember, you've recommended changing > 'for $x in collection("CDI")' to 'for $x in > collection("CDI")/gmd:MD_Metadata/gmd:identificationInfo/sdn:SDN_DataIdentification'. > However, I've discovered I cannot specify XPath while working with IDs > (db:node-pre). > It's a multi-step process: client program sends to the server the search > filter defined by end-user and get IDs. > Then there are several queries for getting different information about this > specific subset. Instead of redefining the filters, the only condition is > where db:node-pre($x)=$ids > for having a better performance. > Once I specific XPath, it seems that the ids have no meaning. The resultset > is always empty once they are being used. > So, I've returned to use 'for $x in collection("CDI")' in the first query of > getting all IDs, but the performance is extremely low. > > **I'm attaching the query and its related info using BaseXGUI (local server) > with much smaller database. The performance seems ok. > > I'm using your BaseXClient.java, however I see the delay already in the > BaseX server logs: > QUERY[0] xquery version "3.0"; declare namespace queryName ='GetIDS'; > declare namespace gco = "http://www.isotc211.org/2005/gco"; declare > namespace gmd = "http://www.isotc211.org/2005/gmd"; declare namespace > gml = "http://www.opengis.net/gml"; declare namespace > gmx="http://www.isotc211.org/2005/gmx"; declare namespace sdn = > "http://www.seadatanet.org"; dec lare namespace fn = > "http://www.w3.org/2005/xpath-functions"; declare namespace xs = > "http://www.w3.org/2001/XMLSchema"; declare namespace output = > "http://www.w3.org/2010/xslt-xquery-serialization"; declare option > output:method 'xml';declare option output:item-separator ","; let $db := > db:open("CDI") for $x in $db where > $x/gmd:MD_Metadata/gmd:identificationInfo/sdn:SDN_DataIdentification/gmd:extent/gmd:EX_Exte > nt/gmd:geographicElement/gmd:EX_GeographicBoundingBox/gmd:westBoundLongitude/gco:Decimal>="-5.8447265625" > and > $x/gmd:MD_Metadata/gmd:identificationInfo/sdn:SDN_DataIdentification/gmd:extent/gmd:EX_Extent/gmd > :geographicElement... 0.17 ms > 110 16:36:09.713 192.168.155.30:39211 admin OK RESULTS[0] > 25957.11 ms > > Then I have other slow queries, but I would like to focus in this phase on > the biggest delay. > Server: Java 1.7.0_79, VM="-XX:MaxPermSize=512m -Xms3096m -Xmx3096m" > The network layer between client and server is very fast. > > P.S. > Id there an undocumented way to log the full xquery in BaseX server logs? > I've seen the -V option, but I don't use the standalone version, but: > java -cp /usr/share/java/basex.jar org.basex.BaseXServer > -d doesn't give me extra query info. > > > With kind regards, > Menashè

>Did you think about using db:node-pre($db/root()) ?

On 06/23/2015 12:10 PM, Christian Grün wrote: > Hi Menashè, > > I am not sure if I can propose any way out, because there are too many > factors that would need to be looked at right now (automatically > composed queries, no node ids, gigabytes of data, ...). > > So let's maybe go back to your original observation: > >> Once I specific XPath, it seems that the ids have no meaning. The resultset is always empty once they are being used. > Can you give us more details on that? Can you give me a *simple* > example that worked, and another example that does not work anymore > for you? Did you think about using db:node-pre($db/root()) ? > > I attached two simplified versions of your query. How do they perform > on your database instance? Maybe we should first try to get these > queries fast before looking at more complex examples (even if these > are the ones that are composed in practice). > > Christian > > > > On Mon, Jun 22, 2015 at 4:48 PM, Menashè Eliezer > <meliezer at ogs.trieste.it> wrote: >> Hi Christian, >> I'm have again performance problems. I have BaseX 8.2.1. >> As you may remember, you've recommended changing >> 'for $x in collection("CDI")' to 'for $x in >> collection("CDI")/gmd:MD_Metadata/gmd:identificationInfo/sdn:SDN_DataIdentification'. >> However, I've discovered I cannot specify XPath while working with IDs >> (db:node-pre). >> It's a multi-step process: client program sends to the server the search >> filter defined by end-user and get IDs. >> Then there are several queries for getting different information about this >> specific subset. Instead of redefining the filters, the only condition is >> where db:node-pre($x)=$ids >> for having a better performance. >> Once I specific XPath, it seems that the ids have no meaning. The resultset >> is always empty once they are being used. >> So, I've returned to use 'for $x in collection("CDI")' in the first query of >> getting all IDs, but the performance is extremely low. >> >> **I'm attaching the query and its related info using BaseXGUI (local server) >> with much smaller database. The performance seems ok. >> >> I'm using your BaseXClient.java, however I see the delay already in the >> BaseX server logs: >> QUERY[0] xquery version "3.0"; declare namespace queryName ='GetIDS'; >> declare namespace gco = "http://www.isotc211.org/2005/gco"; declare >> namespace gmd = "http://www.isotc211.org/2005/gmd"; declare namespace >> gml = "http://www.opengis.net/gml"; declare namespace >> gmx="http://www.isotc211.org/2005/gmx"; declare namespace sdn = >> "http://www.seadatanet.org"; dec lare namespace fn = >> "http://www.w3.org/2005/xpath-functions"; declare namespace xs = >> "http://www.w3.org/2001/XMLSchema"; declare namespace output = >> "http://www.w3.org/2010/xslt-xquery-serialization"; declare option >> output:method 'xml';declare option output:item-separator ","; let $db := >> db:open("CDI") for $x in $db where >> $x/gmd:MD_Metadata/gmd:identificationInfo/sdn:SDN_DataIdentification/gmd:extent/gmd:EX_Exte >> nt/gmd:geographicElement/gmd:EX_GeographicBoundingBox/gmd:westBoundLongitude/gco:Decimal>="-5.8447265625" >> and >> $x/gmd:MD_Metadata/gmd:identificationInfo/sdn:SDN_DataIdentification/gmd:extent/gmd:EX_Extent/gmd >> :geographicElement... 0.17 ms >> 110 16:36:09.713 192.168.155.30:39211 admin OK RESULTS[0] >> 25957.11 ms >> >> Then I have other slow queries, but I would like to focus in this phase on >> the biggest delay. >> Server: Java 1.7.0_79, VM="-XX:MaxPermSize=512m -Xms3096m -Xmx3096m" >> The network layer between client and server is very fast. >> >> P.S. >> Id there an undocumented way to log the full xquery in BaseX server logs? >> I've seen the -V option, but I don't use the standalone version, but: >> java -cp /usr/share/java/basex.jar org.basex.BaseXServer >> -d doesn't give me extra query info. >> >> >> With kind regards, >> Menashè

> Is there an option to ask BaseX to parse only a part of the imported xml > files under a specific xpath, (or at least limit useless indexing of non > relevant components)? I don't need the rest of the xml files, even though > it's not too big. Maybe it can help.

On 06/23/2015 12:51 PM, Christian Grün wrote: >> Is there an option to ask BaseX to parse only a part of the imported xml >> files under a specific xpath, (or at least limit useless indexing of non >> relevant components)? I don't need the rest of the xml files, even though >> it's not too big. Maybe it can help. > The usual approach is to simply create another database that only > contains the relevant parts of your document. This can directly be > done in XQuery (using db:create, db:add, ...), or, if memory > consumption is too high, by exporting and importing parts of your > document. > > Hope this helps, > Christian

> Is there also an option to define inside the part only the xpaths which I would need?

> Another question, how can I know if the following values have been exceeded in a specific database? Quoting: > MAXLEN

> MAXCATS

On 06/23/2015 01:32 PM, Christian Grün wrote: >> Is there also an option to define inside the part only the xpaths which I would need? > I guess no, but to be honest, I am not exactly sure what you mean? > Would you like to restrict indexing to specific parts of the document? > In that case, you'll have to wait for someone implementing [1] > (contributors are always welcome)… > >> Another question, how can I know if the following values have been exceeded in a specific database? Quoting: >> MAXLEN > You will know by looking at your data. Just write a query that returns > the maximum string lengths of all distinct paths. > >> MAXCATS > You can e.g. use index:facets(). > > Hope this helps, > Christian > > [1] https://github.com/BaseXdb/basex/issues/59

> The usual approach is to simply create another database that only > contains the relevant parts of your document. This can directly be > done in XQuery (using db:create, db:add, ...), or, if memory > consumption is too high, by exporting and importing parts of your > document.

> I couldn't find an option in db:add to specificy an XPath. In my case, I > need to extract only the elements under > /gmd:MD_Metadata/gmd:identificationInfo/sdn:SDN_DataIdentification

On 06/24/2015 01:58 PM, Christian Grün wrote: >> I couldn't find an option in db:add to specificy an XPath. In my case, I >> need to extract only the elements under >> /gmd:MD_Metadata/gmd:identificationInfo/sdn:SDN_DataIdentification > We try to avoid XPath strings arguments whenever possible. Instead, > simply use XQuery, which allows you to do all kinds of things. > > Example 1 (add one document per element): > > for $node at $pos in /gmd:MD_Metadata/..... > return db:add('db', $pos || '.xml', $node) > > Example 2 (add single document): > > db:add('db', 'doc.xml', element xml { > /gmd:MD_Metadata/..... > }) > > Cheers, > Christian

> I've used map { 'stripns': true(), 'intparse': true() }) in db:add, but the > namespaces were not removed, e.g. there is gml:beginPosition.

> Anyway, maybe because the xml are not valid, I get always 0 hits unless I > ask to return the doc itself.

> True. I forgot to mention that the 'stripns' option (as all other XML > parsing options [1]) only applies to newly parsed XML strings.

>> Anyway, maybe because the xml are not valid, I get always 0 hits unless I >> ask to return the doc itself. > Hm, the stored data must be well-formed, otherwise it couldn't be > stored. And it will always be well-formed if you store it via db:add > (in XML terminology, validity requires a schema [2]). > > Did you already have a look at the data stored in your new database? > Christian

On 06/25/2015 03:05 PM, Christian Grün wrote: > Hi Menashè, > >> I've used map { 'stripns': true(), 'intparse': true() }) in db:add, but the >> namespaces were not removed, e.g. there is gml:beginPosition. > True. I forgot to mention that the 'stripns' option (as all other XML > parsing options [1]) only applies to newly parsed XML strings. > >> Anyway, maybe because the xml are not valid, I get always 0 hits unless I >> ask to return the doc itself. > Hm, the stored data must be well-formed, otherwise it couldn't be > stored. And it will always be well-formed if you store it via db:add > (in XML terminology, validity requires a schema [2]). > > Did you already have a look at the data stored in your new database? > Christian > > [1] http://docs.basex.org/wiki/Options#STRIPNS > [2] https://en.wikipedia.org/wiki/XML#Schemas_and_validation

> Anyway, how can I strip the namespaces in my new database? I don't need > them.

> I've used db:add. The data is not well-formed. It's just like copy&paste of > the relevant xml. No headers.

> Just create a new database from the input data with this option turned on.

> > If it's not well-formed, you can't store it in BaseX.. If you can do > so, it would be an error (and rather surprising to me ;).

> > Cheers, > C.

On Mon, Jul 13, 2015 at 3:32 PM, Menashè Eliezer <meliezer at ogs.trieste.it> wrote: > Hello, > Creating a database of partial xml documents had almost no effect. > Therefore I've created a database with very simple xml structure. I'm > attaching an example (demo.xml). > BaseX version: 8.2.2 > Number of documents: 374739 > > However, the attached query takes 4 seconds (attached simple_query.log). I > don't know if it's considered a normal performance, but my real query is > different: > I'm copying all the documents which correspond to my query to a newly > created temporary collection, for having faster processing for this subset: > reporting, ecc. > Adding to db (Remote Java client): 12 sec. > Optimising the db (Remote Java client): 23 sec. > Both the Java client and BaseX server are installed on powerful servers. > Are these numbers normal? > The attached results are based on a local client (Using the BaseX GUI). > > In the future, I should have even much more documents to handle... > Any ideas? > I can also change the scheme of my new xml. > > As for the idea of creating a new temporary db, I'm checking an alternative: > return in one query all what I need, including reports, all in one xml. > > With kind regards, > Menashè > >

On 07/14/2015 09:32 AM, Christian Grün wrote: > Hi Menashè, > > The attached log file is empty. Maybe it's sufficient if you provide > us with the query and give us information on the query compilation > (are any indexes used?). > > C. > > > On Mon, Jul 13, 2015 at 3:32 PM, Menashè Eliezer > <meliezer at ogs.trieste.it> wrote: >> Hello, >> Creating a database of partial xml documents had almost no effect. >> Therefore I've created a database with very simple xml structure. I'm >> attaching an example (demo.xml). >> BaseX version: 8.2.2 >> Number of documents: 374739 >> >> However, the attached query takes 4 seconds (attached simple_query.log). I >> don't know if it's considered a normal performance, but my real query is >> different: >> I'm copying all the documents which correspond to my query to a newly >> created temporary collection, for having faster processing for this subset: >> reporting, ecc. >> Adding to db (Remote Java client): 12 sec. >> Optimising the db (Remote Java client): 23 sec. >> Both the Java client and BaseX server are installed on powerful servers. >> Are these numbers normal? >> The attached results are based on a local client (Using the BaseX GUI). >> >> In the future, I should have even much more documents to handle... >> Any ideas? >> I can also change the scheme of my new xml. >> >> As for the idea of creating a new temporary db, I'm checking an alternative: >> return in one query all what I need, including reports, all in one xml. >> >> With kind regards, >> Menashè >> >>

> oops, I'm sorry. It's attached. > There are text and attribute indexes.

On 07/14/2015 11:05 AM, Christian Grün wrote: > It may be slightly faster if you remove the explicit string() conversion

> But please note that BaseX provides no native range index, which would > be a good fit for your longitude/latitude filter.

> Should geo:within of http://docs.basex.org/wiki/Geo_Module help?

On 07/14/2015 12:50 PM, Christian Grün wrote: >> Should geo:within of http://docs.basex.org/wiki/Geo_Module help? > The functions of the Geo Module don't use any index structures, so I > am afraid they won't speed up the query. > > One more idea: you could convert all latitudes and longitudes to > strings with a fixed number of digits.... > _____________________________________ > > (:~ Allowed range. :) > declare variable $RANGE := 999999; > (:~ Maximum latitude. :) > declare variable $LAT-MIN := -90; > (:~ Maximum longitude. :) > declare variable $LAT-MAX := 90; > > (:~ > : Converts a double value to a normalized string value > : with a fixed size of digits. > : @param $num number to be converted > : @param $min minimum allowed value > : @param $max maximum allowed value > : @return resulting value > :) > declare function local:normalize( > $num as xs:double, > $min as xs:integer, > $max as xs:integer > ) { > let $norm := $RANGE * ($num - $min) div ($max - $min) > return format-number($norm, '000000') > }; > > (: Run code for various latitude values :) > for $latitude in (-90, -89.9999, -13.345, 0, 89.99999) > return local:normalize($latitude, $LAT-MIN, $LAT-MAX) > _____________________________________ > > Next, you could to do string comparisons on these values: > > for $doc in db:open("CDI") > let $lat := $doc//latitude > let $lon := $doc//longitude > where $lat >= "883387" and $lat <= "893463" > and $lon >= "173467" and $lon <= "178745" > return db:node-pre($doc) > > It should be fast enough if the maximum value is not much bigger than > the minimum value.

On Tue, Jul 14, 2015 at 2:42 PM, Menashè Eliezer <meliezer at ogs.trieste.it> wrote: > Hi, > It sounds like a great idea and I can also implement it to the date > comparisons, but unfortunately the new query is much slower. > Please see the attached log. > > With kind regards, > Menashè > > > On 07/14/2015 12:50 PM, Christian Grün wrote: >>> >>> Should geo:within of http://docs.basex.org/wiki/Geo_Module help? >> >> The functions of the Geo Module don't use any index structures, so I >> am afraid they won't speed up the query. >> >> One more idea: you could convert all latitudes and longitudes to >> strings with a fixed number of digits.... >> _____________________________________ >> >> (:~ Allowed range. :) >> declare variable $RANGE := 999999; >> (:~ Maximum latitude. :) >> declare variable $LAT-MIN := -90; >> (:~ Maximum longitude. :) >> declare variable $LAT-MAX := 90; >> >> (:~ >> : Converts a double value to a normalized string value >> : with a fixed size of digits. >> : @param $num number to be converted >> : @param $min minimum allowed value >> : @param $max maximum allowed value >> : @return resulting value >> :) >> declare function local:normalize( >> $num as xs:double, >> $min as xs:integer, >> $max as xs:integer >> ) { >> let $norm := $RANGE * ($num - $min) div ($max - $min) >> return format-number($norm, '000000') >> }; >> >> (: Run code for various latitude values :) >> for $latitude in (-90, -89.9999, -13.345, 0, 89.99999) >> return local:normalize($latitude, $LAT-MIN, $LAT-MAX) >> _____________________________________ >> >> Next, you could to do string comparisons on these values: >> >> for $doc in db:open("CDI") >> let $lat := $doc//latitude >> let $lon := $doc//longitude >> where $lat >= "883387" and $lat <= "893463" >> and $lon >= "173467" and $lon <= "178745" >> return db:node-pre($doc) >> >> It should be fast enough if the maximum value is not much bigger than >> the minimum value. > >

On 07/14/2015 03:22 PM, Christian Grün wrote: > ...it only makes sense if you store the data in its normalized representation. > > > On Tue, Jul 14, 2015 at 2:42 PM, Menashè Eliezer > <meliezer at ogs.trieste.it> wrote: >> Hi, >> It sounds like a great idea and I can also implement it to the date >> comparisons, but unfortunately the new query is much slower. >> Please see the attached log. >> >> With kind regards, >> Menashè >> >> >> On 07/14/2015 12:50 PM, Christian Grün wrote: >>>> Should geo:within of http://docs.basex.org/wiki/Geo_Module help? >>> The functions of the Geo Module don't use any index structures, so I >>> am afraid they won't speed up the query. >>> >>> One more idea: you could convert all latitudes and longitudes to >>> strings with a fixed number of digits.... >>> _____________________________________ >>> >>> (:~ Allowed range. :) >>> declare variable $RANGE := 999999; >>> (:~ Maximum latitude. :) >>> declare variable $LAT-MIN := -90; >>> (:~ Maximum longitude. :) >>> declare variable $LAT-MAX := 90; >>> >>> (:~ >>> : Converts a double value to a normalized string value >>> : with a fixed size of digits. >>> : @param $num number to be converted >>> : @param $min minimum allowed value >>> : @param $max maximum allowed value >>> : @return resulting value >>> :) >>> declare function local:normalize( >>> $num as xs:double, >>> $min as xs:integer, >>> $max as xs:integer >>> ) { >>> let $norm := $RANGE * ($num - $min) div ($max - $min) >>> return format-number($norm, '000000') >>> }; >>> >>> (: Run code for various latitude values :) >>> for $latitude in (-90, -89.9999, -13.345, 0, 89.99999) >>> return local:normalize($latitude, $LAT-MIN, $LAT-MAX) >>> _____________________________________ >>> >>> Next, you could to do string comparisons on these values: >>> >>> for $doc in db:open("CDI") >>> let $lat := $doc//latitude >>> let $lon := $doc//longitude >>> where $lat >= "883387" and $lat <= "893463" >>> and $lon >= "173467" and $lon <= "178745" >>> return db:node-pre($doc) >>> >>> It should be fast enough if the maximum value is not much bigger than >>> the minimum value. >>

On 06/22/2015 05:11 PM, Menashè Eliezer wrote: > Hi, > I've used ssh -X for producing query info right from the server > machine. Please see attached. > I hope it would help. > > With kind regards, > Menashè > > On 06/22/2015 04:48 PM, Menashè Eliezer wrote: >> Hi Christian, >> I'm have again performance problems. I have BaseX 8.2.1. >> As you may remember, you've recommended changing >> 'for $x in collection("CDI")' to 'for $x in >> collection("CDI")/gmd:MD_Metadata/gmd:identificationInfo/sdn:SDN_DataIdentification'. >> However, I've discovered I cannot specify XPath while working with >> IDs (db:node-pre). >> It's a multi-step process: client program sends to the server the >> search filter defined by end-user and get IDs. >> Then there are several queries for getting different information >> about this specific subset. Instead of redefining the filters, the >> only condition is >> where db:node-pre($x)=$ids >> for having a better performance. >> Once I specific XPath, it seems that the ids have no meaning. The >> resultset is always empty once they are being used. >> So, I've returned to use 'for $x in collection("CDI")' in the first >> query of getting all IDs, but the performance is extremely low. >> >> **I'm attaching the query and its related info using BaseXGUI (local >> server) with much smaller database. The performance seems ok. >> >> I'm using your BaseXClient.java, however I see the delay already in >> the BaseX server logs: >> QUERY[0] xquery version "3.0"; declare namespace queryName >> ='GetIDS'; declare namespace gco = >> "http://www.isotc211.org/2005/gco"; declare namespace gmd = >> "http://www.isotc211.org/2005/gmd"; declare namespace gml = >> "http://www.opengis.net/gml"; declare namespace >> gmx="http://www.isotc211.org/2005/gmx"; declare namespace sdn = >> "http://www.seadatanet.org"; dec lare namespace fn = >> "http://www.w3.org/2005/xpath-functions"; declare namespace xs = >> "http://www.w3.org/2001/XMLSchema"; declare namespace output = >> "http://www.w3.org/2010/xslt-xquery-serialization"; declare option >> output:method 'xml';declare option output:item-separator ","; let $db >> := db:open("CDI") for $x in $db where >> $x/gmd:MD_Metadata/gmd:identificationInfo/sdn:SDN_DataIdentification/gmd:extent/gmd:EX_Exte >> nt/gmd:geographicElement/gmd:EX_GeographicBoundingBox/gmd:westBoundLongitude/gco:Decimal>="-5.8447265625" >> and >> $x/gmd:MD_Metadata/gmd:identificationInfo/sdn:SDN_DataIdentification/gmd:extent/gmd:EX_Extent/gmd >> :geographicElement... 0.17 ms >> 110 16:36:09.713 192.168.155.30:39211 admin OK >> RESULTS[0] 25957.11 ms >> >> Then I have other slow queries, but I would like to focus in this >> phase on the biggest delay. >> Server: Java 1.7.0_79, VM="-XX:MaxPermSize=512m -Xms3096m -Xmx3096m" >> The network layer between client and server is very fast. >> >> P.S. >> Id there an undocumented way to log the full xquery in BaseX server >> logs? >> I've seen the -V option, but I don't use the standalone version, but: >> java -cp /usr/share/java/basex.jar org.basex.BaseXServer >> -d doesn't give me extra query info. >> >> >> With kind regards, >> Menashè >> >> On 02/03/2015 01:13 PM, Menashè Eliezer wrote: >>> Hi Christian, >>> >>> Thank you! The performance arrives to 0.5 sec! >>> >>> The biggest improvement is related to the query rephrasing you've >>> suggested. >>> Then the latest snapshot also helps a lot! >>> You may want to know that in the log of the latest snapshot I see >>> applying attribute index for "7827" >>> which is not clear to the user, instead of BaseX80-20150130.124009 >>> which has also used indexing: >>> applying attribute index for ("ALKY", "AYMD") >>> >>> I'm attaching the first and the second launch of the query using >>> BaseXGUI. Relaunching the same query reduces the time from over 1 >>> second to 0.5 second. >>> Some data: >>> BaseX80-20150130.124009 >>> Total Time: 30676.02 ms >>> After using "for $x in >>> collection("ALL-CDIS")/gmd:MD_Metadata/gmd:identificationInfo/sdn:SDN_DataIdentification": >>> Total Time: 5456.74 ms >>> applying attribute index for ("ALKY", "AYMD") in log. >>> Second launch: 1333.71 ms >>> Latest snapshot (BaseX80-20150202.121033): >>> 1st: Total Time: 1873.02 ms >>> 2nd: Total Time: 548.62 ms >>> >>> With kind regards, >>> Menashè >>> >>> On 02/02/2015 02:02 PM, Menashè Eliezer wrote: >>>> Hi Christian, >>>> >>>> Thank you very much! Unfortunately I'll be at the office only >>>> tomorrow. >>>> >>>> Menashè >>>> >>>> On Sat, 31 Jan 2015 16:42:32 +0100, Christian Grün >>>> <christian.gruen at gmail.com> wrote: >>>>> Hi Menashè, >>>>> >>>>> With the latest snapshot [1], your original query should now be >>>>> rewritten for index access as well. Looking forward to your tests, >>>>> >>>>> Christian >>>>> >>>>> PS: In terms of performance, it may still be worthwhile to move >>>>> redundant paths to the for clause; but just try and see. >>>>> >>>>> [1] http://files.basex.org/releases/latest/ >>>>> >>>>> >>>>> >>>>> On Fri, Jan 30, 2015 at 9:49 PM, Christian Grün >>>>> <christian.gruen at gmail.com> wrote: >>>>>> Hi Menashè, >>>>>> >>>>>>> Should I expect to see the usage of an index for each of the where >>>>> phrases? >>>>>> Usually, only one predicate will be rewritten for index access, and >>>>>> the remaining conditions will be answered sequentially. >>>>>> >>>>>>> Have a nice weekend! >>>>>> Enjoy, >>>>>> Christian >>>>>> >>>>>> >>>>>>> Menashè >>>>>>> >>>>>>> On Fri, 30 Jan 2015 18:11:59 +0100, Christian Grün >>>>>>> <christian.gruen at gmail.com> wrote: >>>>>>>> Hi Menashè, >>>>>>>> >>>>>>>> Thanks for the XML samples you sent me in private. I noticed >>>>>>>> that the >>>>>>>> index rewritings will only be triggered if you formulate your >>>>>>>> query as >>>>>>>> follows: >>>>>>>> >>>>>>>> OLD: >>>>>>>> for $x in collection("ALL-CDIS") >>>>>>>> where $x/gmd:MD_Metadata/gmd:identificationInfo/... >>>>>>>> return ... >>>>>>>> >>>>>>>> NEW: >>>>>>>> for $x in collection("ALL-CDIS")/gmd:MD_Metadata >>>>>>>> where $x/gmd:identificationInfo/... >>>>>>>> return ... >>>>>>>> >>>>>>>> It's difficult to explain in short sentences why Variant 1 >>>>>>>> cannot be >>>>>>>> optimized that straightforward (basically, it's quite a different >>>>>>>> pattern to look for), but I'll check out if we can extend our >>>>>>>> matcher >>>>>>>> to also support these kind of queries. >>>>>>>> >>>>>>>> So, if possible, I would recommend you for now (and at least for >>>>>>>> testing) to move the root element test after the collection() >>>>>>>> function. I noticed that the first three child steps are the >>>>>>>> same in >>>>>>>> all of your conditions: >>>>>>>> >>>>>>>> gmd:MD_Metadata/gmd:identificationInfo/sdn:SDN_DataIdentification >>>>>>>> >>>>>>>> If that will be always be the case, it surely makes sense to >>>>>>>> move all >>>>>>>> of them to the "for" clause. >>>>>>>> >>>>>>>> Looking forward to your updated performance tests, >>>>>>>> Christian >>>>>>>> _______________________________ >>>>>>>> >>>>>>>> On Fri, Jan 30, 2015 at 5:55 PM, Christian Grün >>>>>>>> <christian.gruen at gmail.com> wrote: >>>>>>>>> Could you possibly provide me with a small snapshot of your data >>>>>>>>> sources (one, two documents might be sufficient)? >>>>>>>>> >>>>>>>>> >>>>>>>>> On Fri, Jan 30, 2015 at 5:52 PM, Menashè Eliezer >>>>>>>>> <meliezer at ogs.trieste.it> wrote: >>>>>>>>>> Almost the same speed with version 8.0. >>>>>>>>>> No indexing (no "applying" in the query info). >>>>>>>>>> As I've attached before, indexes are active for this DB. >>>>>>>>>> >>>>>>>>>> With kind regards, >>>>>>>>>> Menashè >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 01/30/2015 05:31 PM, Christian Grün wrote: >>>>>>>>>>> It's indeed interesting that your query does not use any of the >>>>>>>>>>> existing index structures (if they did, you would find >>>>>>>>>>> strings like >>>>>>>>>>> "applying text index" or "applying attribute index" in the >>>>>>>>>>> query >>>>>>>>>>> info). Maybe/hopefully things look different with Version 8.0. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Fri, Jan 30, 2015 at 5:26 PM, Menashè Eliezer >>>>>>>>>>> <meliezer at ogs.trieste.it> wrote: >>>>>>>>>>>> On 01/30/2015 05:18 PM, Christian Grün wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>> /gmd:MD_Metadata/gmd:identificationInfo/sdn:SDN_DataIdentification/gmd:descriptiveKeywords[1]/gmd:MD_Keywords/gmd:keyword[2]/sdn:SDN_ParameterDiscoveryCode/@codeListValue >>>> >>>>>>>>>>>>>> How can I remove *? >>>>>>>>>>>>> Simply remove the predicate; a[*]/b is the same as a/b. >>>>>>>>>>>> Maybe I wasn't clear. The actual number appears in the xml >>>>>>>>>>>> file, >>>>>>>> e.g., >>>>>>>>>>>> gmd:descriptiveKeywords[1] >>>>>>>>>>>> Anyway, I've removed all [*] and I get the same correct >>>>>>>>>>>> result, >>>>>>>> however >>>>>>>>>>>> the >>>>>>>>>>>> processing time is doubled... >>>>>>>>>>>>> >>>>>>>>>>>>>>> * In some cases, if you know that an element name is >>>>>>>>>>>>>>> distinct, >>>>> you >>>>>>>> can >>>>>>>>>>>>>>> get rid of all the explicit child steps and directly >>>>>>>>>>>>>>> address >>>>> the >>>>>>>> node >>>>>>>>>>>>>>> via the descendant axis. >>>>>>>>>>>>>> Thanks, but it's not relevant in my case. >>>>>>>>>>>>> Is it because the element names are not distinct? Or is it >>>>> because >>>>>>>>>>>>> your input form allows users to choose arbitrary paths for >>>>> arbitrary >>>>>>>>>>>>> documents? >>>>>>>>>>>> The element names are not distinct. >>>>>>>>>>>> >>>>>>>>>>>>>> Sure, I'l also try BaseX 8.0 and compare. Should I >>>>>>>>>>>>>> recreate the >>>>> db >>>>>>>>>>>>>> importing >>>>>>>>>>>>>> the xml files for testing the improved indexing? >>>>>>>>>>>>> We have actually improved support for collections, but the >>>>> database >>>>>>>>>>>>> format itself has not changed, so it shouldn't make a >>>>>>>>>>>>> difference >>>>> in >>>>>>>>>>>>> your case. >>>>>>>>>>>>> >>>>>>>>>>>>> Christian >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>>> [1] http://files.basex.org/releases/latest >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Fri, Jan 30, 2015 at 3:55 PM, Menashè Eliezer >>>>>>>>>>>>>>> <meliezer at ogs.trieste.it> wrote: >>>>>>>>>>>>>>>> Hello, >>>>>>>>>>>>>>>> I wonder if the attached query can be optimised. I'm >>>>>>>>>>>>>>>> attaching >>>>>>>> all >>>>>>>>>>>>>>>> relevant >>>>>>>>>>>>>>>> information. >>>>>>>>>>>>>>>> Basex 7.9, Debian, powerful server. >>>>>>>>>>>>>>>> This is just an example. The queries will be built >>>>>>>>>>>>>>>> based on a >>>>>>>>>>>>>>>> compilation >>>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>> a search form. >>>>>>>>>>>>>>>> Any help would be appreciated. >>>>>>>>>>>>>>>> 40 seconds are not acceptable. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> With kind regards, >>>>>>>>>>>>>>>> Menashè >>>>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> With kind regards, >>>>>>>>>>>>>> Menashè >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>> With kind regards, >>>>>>>>>>>> Menashè >>>>>>>>>>>> >>>>>>> -- >>>>>>> Menashè >

> I'm have again performance problems. I have BaseX 8.2.2.

On 08/03/2015 03:24 PM, Christian Grün wrote: > What was the last version it was working with?

>> What was the last version it was working with? > 8.2.1. Not really working, but better...

On 08/03/2015 03:38 PM, Christian Grün wrote: >>> What was the last version it was working with? >> 8.2.1. Not really working, but better... > I ran the attached query with 8.2.1, and no index was used either. Are > you sure you sent me the correct query? > > Sorry for confronting you with all those questions, but to help you, I > really need your help as well. Could you check the attached files > again and give me some hints on how to proceed?