Query Cache?

On Wed, Aug 13, 2014 at 12:02 PM, Christopher Yocum <cyocum at gmail.com> wrote: > declare variable $term as xs:string external; declare variable $col as > xs:string external; <results>{subsequence(ft:mark(for $x in > collection($col)//entry where $x//text()[. contains text {$term} using > wildcards] order by fn:lower-case(fn:replace(($x//orth[1]/text())[1], > '\\p{P}|\\d+','')) collation \"?lang=ga\" return $x), 1, 5000)}</results>

On Wed, Aug 13, 2014 at 12:18 PM, Christian Grün <christian.gruen at gmail.com> wrote: > Hi Chris, > > there are various caches involved when evaluating queries, but I can't > see for the given query where a cache may be utilized. However, your > query may be evaluated faster if you simplify the nested where clause: > > <results>{ > subsequence( > ft:mark( > for $x in collection($col)//entry > where $x//text() contains text { $term } using wildcards > order by fn:lower-case( > fn:replace(($x//orth[1]/text())[1], '\\p{P}|\\d+','') > ) collation "?lang=ga" > return $x > ), 1, 5000 > ) > }</results> > > You could as well use a predicate with position(), it may be evaluated > faster than subsequence (I'm not sure, though, because most time will > probably be spent for ordering all results): > > <results>{ > ft:mark( > for $x in collection($col)//entry > where $x//text() contains text { $term } using wildcards > order by fn:lower-case( > fn:replace(($x//orth[1]/text())[1], '\\p{P}|\\d+','') > ) collation "?lang=ga" > return $x > )[position() = 1 to 5000] > }</results> > > Could you please open the InfoView in the GUI, execute the query again > and check if the full-text index is applied? > > Christian > > > > On Wed, Aug 13, 2014 at 12:02 PM, Christopher Yocum <cyocum at gmail.com> > wrote: > > declare variable $term as xs:string external; declare variable $col as > > xs:string external; <results>{subsequence(ft:mark(for $x in > > collection($col)//entry where $x//text()[. contains text {$term} using > > wildcards] order by fn:lower-case(fn:replace(($x//orth[1]/text())[1], > > '\\p{P}|\\d+','')) collation \"?lang=ga\" return $x), 1, 5000)}</results> >

On Wed, Aug 13, 2014 at 01:18:26PM +0200, Christian Grün wrote: > Hi Chris, > > there are various caches involved when evaluating queries, but I can't > see for the given query where a cache may be utilized. However, your > query may be evaluated faster if you simplify the nested where clause: > > <results>{ > subsequence( > ft:mark( > for $x in collection($col)//entry > where $x//text() contains text { $term } using wildcards > order by fn:lower-case( > fn:replace(($x//orth[1]/text())[1], '\\p{P}|\\d+','') > ) collation "?lang=ga" > return $x > ), 1, 5000 > ) > }</results> > > You could as well use a predicate with position(), it may be evaluated > faster than subsequence (I'm not sure, though, because most time will > probably be spent for ordering all results): > > <results>{ > ft:mark( > for $x in collection($col)//entry > where $x//text() contains text { $term } using wildcards > order by fn:lower-case( > fn:replace(($x//orth[1]/text())[1], '\\p{P}|\\d+','') > ) collation "?lang=ga" > return $x > )[position() = 1 to 5000] > }</results> > > Could you please open the InfoView in the GUI, execute the query again > and check if the full-text index is applied? > > Christian

> > > > On Wed, Aug 13, 2014 at 12:02 PM, Christopher Yocum <cyocum at gmail.com> wrote: > > declare variable $term as xs:string external; declare variable $col as > > xs:string external; <results>{subsequence(ft:mark(for $x in > > collection($col)//entry where $x//text()[. contains text {$term} using > > wildcards] order by fn:lower-case(fn:replace(($x//orth[1]/text())[1], > > '\\p{P}|\\d+','')) collation \"?lang=ga\" return $x), 1, 5000)}</results>

> <FTIndexAccess data="edil">

> I ran the same query on my laptop in the GUI and got this in the Query > Info. I couldn't find the "InfoView" so I hope this helps.

> Compiling: > - - pre-evaluating fn:collection("edil") > ...

On Thu, Aug 14, 2014 at 08:03:39PM +0200, Christian Grün wrote: > Hi Chris, > > > <FTIndexAccess data="edil"> > > This indicates that the full-text index is indeed used here. > > > I ran the same query on my laptop in the GUI and got this in the Query > > Info. I couldn't find the "InfoView" so I hope this helps. > > Sorry, I was referring to the query info, which you found anyway.. > > > Compiling: > > - - pre-evaluating fn:collection("edil") > > ... > > I'm missing the info "applying full-text index..." here, probably > because you need to rewrite your code from... > > collection($col)//entry where $x//text()[. contains text {$term} > using wildcards] > > ...to... > > collection($col)//entry where $x//text() contains text {$term} using wildcards > > Could you try this and report about the performance? > Christian

On Wed, Aug 13, 2014 at 01:18:26PM +0200, Christian Grün wrote: > Hi Chris, > > there are various caches involved when evaluating queries, but I can't > see for the given query where a cache may be utilized. However, your > query may be evaluated faster if you simplify the nested where clause: > > <results>{ > subsequence( > ft:mark( > for $x in collection($col)//entry > where $x//text() contains text { $term } using wildcards > order by fn:lower-case( > fn:replace(($x//orth[1]/text())[1], '\\p{P}|\\d+','') > ) collation "?lang=ga" > return $x > ), 1, 5000 > ) > }</results> > > You could as well use a predicate with position(), it may be evaluated > faster than subsequence (I'm not sure, though, because most time will > probably be spent for ordering all results): > > <results>{ > ft:mark( > for $x in collection($col)//entry > where $x//text() contains text { $term } using wildcards > order by fn:lower-case( > fn:replace(($x//orth[1]/text())[1], '\\p{P}|\\d+','') > ) collation "?lang=ga" > return $x > )[position() = 1 to 5000] > }</results> > > Could you please open the InfoView in the GUI, execute the query again > and check if the full-text index is applied? > > Christian > > > > On Wed, Aug 13, 2014 at 12:02 PM, Christopher Yocum <cyocum at gmail.com> wrote: > > declare variable $term as xs:string external; declare variable $col as > > xs:string external; <results>{subsequence(ft:mark(for $x in > > collection($col)//entry where $x//text()[. contains text {$term} using > > wildcards] order by fn:lower-case(fn:replace(($x//orth[1]/text())[1], > > '\\p{P}|\\d+','')) collation \"?lang=ga\" return $x), 1, 5000)}</results>

On Thu, Aug 14, 2014 at 10:54 PM, Chris Yocum <cyocum at gmail.com> wrote: > Hi Christian, > > Apologies for bringing this back up but if I use "using diacritics > insensitive" in the full text search, it seems to turn full text > searching off. I have diacritics true on the database. I am just > suprised to see diacritics causing the full text searching to be > turned off. > > All the best, > Chris > > On Wed, Aug 13, 2014 at 01:18:26PM +0200, Christian Grün wrote: >> Hi Chris, >> >> there are various caches involved when evaluating queries, but I can't >> see for the given query where a cache may be utilized. However, your >> query may be evaluated faster if you simplify the nested where clause: >> >> <results>{ >> subsequence( >> ft:mark( >> for $x in collection($col)//entry >> where $x//text() contains text { $term } using wildcards >> order by fn:lower-case( >> fn:replace(($x//orth[1]/text())[1], '\\p{P}|\\d+','') >> ) collation "?lang=ga" >> return $x >> ), 1, 5000 >> ) >> }</results> >> >> You could as well use a predicate with position(), it may be evaluated >> faster than subsequence (I'm not sure, though, because most time will >> probably be spent for ordering all results): >> >> <results>{ >> ft:mark( >> for $x in collection($col)//entry >> where $x//text() contains text { $term } using wildcards >> order by fn:lower-case( >> fn:replace(($x//orth[1]/text())[1], '\\p{P}|\\d+','') >> ) collation "?lang=ga" >> return $x >> )[position() = 1 to 5000] >> }</results> >> >> Could you please open the InfoView in the GUI, execute the query again >> and check if the full-text index is applied? >> >> Christian >> >> >> >> On Wed, Aug 13, 2014 at 12:02 PM, Christopher Yocum <cyocum at gmail.com> wrote: >> > declare variable $term as xs:string external; declare variable $col as >> > xs:string external; <results>{subsequence(ft:mark(for $x in >> > collection($col)//entry where $x//text()[. contains text {$term} using >> > wildcards] order by fn:lower-case(fn:replace(($x//orth[1]/text())[1], >> > '\\p{P}|\\d+','')) collation \"?lang=ga\" return $x), 1,

On Thu, Aug 14, 2014 at 10:35 PM, Christian Grün <christian.gruen at gmail.com> wrote: > Hi Chris, > > as you already noted, the full-text index > will > only > be > utilized with > the > options that you choose when creating an index. If you want to do more > fine-grained searches, it’s > usually > recommendable to > choose > the most general options for creating the index (case insensitive, > diacritics insensitive, etc). and > then > refine the results in a second step. > This can e.g. look as follows > : > > declare function local:search($db, $terms) { > for $result in db:open($db)//*[text() contains text { $terms }] > return $result[text() contains text { $terms } using case sensitive] > }; > local:search('factbook', ('German', 'English')) > > Hope this helps, > Christian > > > > On Thu, Aug 14, 2014 at 10:54 PM, Chris Yocum <cyocum at gmail.com> wrote: > > Hi Christian, > > > > Apologies for bringing this back up but if I use "using diacritics > > insensitive" in the full text search, it seems to turn full text > > searching off. I have diacritics true on the database. I am just > > suprised to see diacritics causing the full text searching to be > > turned off. > > > > All the best, > > Chris > > > > On Wed, Aug 13, 2014 at 01:18:26PM +0200, Christian Grün wrote: > >> Hi Chris, > >> > >> there are various caches involved when evaluating queries, but I can't > >> see for the given query where a cache may be utilized. However, your > >> query may be evaluated faster if you simplify the nested where clause: > >> > >> <results>{ > >> subsequence( > >> ft:mark( > >> for $x in collection($col)//entry > >> where $x//text() contains text { $term } using wildcards > >> order by fn:lower-case( > >> fn:replace(($x//orth[1]/text())[1], '\\p{P}|\\d+','') > >> ) collation "?lang=ga" > >> return $x > >> ), 1, 5000 > >> ) > >> }</results> > >> > >> You could as well use a predicate with position(), it may be evaluated > >> faster than subsequence (I'm not sure, though, because most time will > >> probably be spent for ordering all results): > >> > >> <results>{ > >> ft:mark( > >> for $x in collection($col)//entry > >> where $x//text() contains text { $term } using wildcards > >> order by fn:lower-case( > >> fn:replace(($x//orth[1]/text())[1], '\\p{P}|\\d+','') > >> ) collation "?lang=ga" > >> return $x > >> )[position() = 1 to 5000] > >> }</results> > >> > >> Could you please open the InfoView in the GUI, execute the query again > >> and check if the full-text index is applied? > >> > >> Christian > >> > >> > >> > >> On Wed, Aug 13, 2014 at 12:02 PM, Christopher Yocum <cyocum at gmail.com> > wrote: > >> > declare variable $term as xs:string external; declare variable $col as > >> > xs:string external; <results>{subsequence(ft:mark(for $x in > >> > collection($col)//entry where $x//text()[. contains text {$term} using > >> > wildcards] order by fn:lower-case(fn:replace(($x//orth[1]/text())[1], > >> > '\\p{P}|\\d+','')) collation \"?lang=ga\" return $x), 1, > 5000)}</results> >

> Thank you again for all your help. Unfortunately, my documents are > multi-language and multi-diacritics so my users expect it to match > athgabáil, athgabail, and athgabāil as the same word. They also want > wildcard searching to work in the same way.

> At the moment the query looks like this and it does not use the full text > index: > > declare variable $term as xs:string external := 'athgab.*'; declare variable > $col as xs:string external := 'edil'; <results>{subsequence(ft:mark(for $x > in collection($col)//entry where $x//text() contains text {$term} using > wildcards using diacritics insensitive order by > fn:lower-case(fn:replace(($x//orth[1]/text())[1], '\p{P}|\d+','')) collation > "?lang=ga" return $x), 1, 5000)}</results> > > If anyone has any suggestions, I would be grateful. > > All the best, > Chris > > > On Thu, Aug 14, 2014 at 10:35 PM, Christian Grün <christian.gruen at gmail.com> > wrote: >> >> Hi Chris, >> >> as you already noted, the full-text index >> will >> only >> be >> utilized with >> the >> options that you choose when creating an index. If you want to do more >> fine-grained searches, it’s >> usually >> recommendable to >> choose >> the most general options for creating the index (case insensitive, >> diacritics insensitive, etc). and >> then >> refine the results in a second step. >> This can e.g. look as follows >> : >> >> declare function local:search($db, $terms) { >> for $result in db:open($db)//*[text() contains text { $terms }] >> return $result[text() contains text { $terms } using case sensitive] >> }; >> local:search('factbook', ('German', 'English')) >> >> Hope this helps, >> Christian >> >> >> >> On Thu, Aug 14, 2014 at 10:54 PM, Chris Yocum <cyocum at gmail.com> wrote: >> > Hi Christian, >> > >> > Apologies for bringing this back up but if I use "using diacritics >> > insensitive" in the full text search, it seems to turn full text >> > searching off. I have diacritics true on the database. I am just >> > suprised to see diacritics causing the full text searching to be >> > turned off. >> > >> > All the best, >> > Chris >> > >> > On Wed, Aug 13, 2014 at 01:18:26PM +0200, Christian Grün wrote: >> >> Hi Chris, >> >> >> >> there are various caches involved when evaluating queries, but I can't >> >> see for the given query where a cache may be utilized. However, your >> >> query may be evaluated faster if you simplify the nested where clause: >> >> >> >> <results>{ >> >> subsequence( >> >> ft:mark( >> >> for $x in collection($col)//entry >> >> where $x//text() contains text { $term } using wildcards >> >> order by fn:lower-case( >> >> fn:replace(($x//orth[1]/text())[1], '\\p{P}|\\d+','') >> >> ) collation "?lang=ga" >> >> return $x >> >> ), 1, 5000 >> >> ) >> >> }</results> >> >> >> >> You could as well use a predicate with position(), it may be evaluated >> >> faster than subsequence (I'm not sure, though, because most time will >> >> probably be spent for ordering all results): >> >> >> >> <results>{ >> >> ft:mark( >> >> for $x in collection($col)//entry >> >> where $x//text() contains text { $term } using wildcards >> >> order by fn:lower-case( >> >> fn:replace(($x//orth[1]/text())[1], '\\p{P}|\\d+','') >> >> ) collation "?lang=ga" >> >> return $x >> >> )[position() = 1 to 5000] >> >> }</results> >> >> >> >> Could you please open the InfoView in the GUI, execute the query again >> >> and check if the full-text index is applied? >> >> >> >> Christian >> >> >> >> >> >> >> >> On Wed, Aug 13, 2014 at 12:02 PM, Christopher Yocum <cyocum at gmail.com> >> >> wrote: >> >> > declare variable $term as xs:string external; declare variable $col >> >> > as >> >> > xs:string external; <results>{subsequence(ft:mark(for $x in >> >> > collection($col)//entry where $x//text()[. contains text {$term} >> >> > using >> >> > wildcards] order by fn:lower-case(fn:replace(($x//orth[1]/text())[1], >> >> > '\\p{P}|\\d+','')) collation \"?lang=ga\" return $x), 1, >> >> > 5000)}</results> > >

On Tue, Aug 19, 2014 at 10:12 AM, Christian Grün <christian.gruen at gmail.com> wrote: > Hi Chris, > > sorry for letting you wait, I’ve been offline over the weekend. > > > Thank you again for all your help. Unfortunately, my documents are > > multi-language and multi-diacritics so my users expect it to match > > athgabáil, athgabail, and athgabāil as the same word. They also want > > wildcard searching to work in the same way. > > This should be no problem, even with the full-text default settings. > An example: the following query... > > /descendant::*[text() contains text 'athgabāi.*' > using diacritics insensitive > using wildcards] > > ...will give you three results for the following document... > > <xml> > <term>athgabáil</term> > <term>athgabail</term> > <term>athgabāil</term> > </xml> > > ...and the results will be retrieved by the full-text index, using the > default settings: > > - applying full-text index for "athgabāi.*" using wildcards using > language 'English' > > The solution that I mentioned in my last mail is required if you want > to do both diacritics sensitive and insensitive search. > > Does this help? > Christian > > > > > > At the moment the query looks like this and it does not use the full text > > index: > > > > declare variable $term as xs:string external := 'athgab.*'; declare > variable > > $col as xs:string external := 'edil'; <results>{subsequence(ft:mark(for > $x > > in collection($col)//entry where $x//text() contains text {$term} using > > wildcards using diacritics insensitive order by > > fn:lower-case(fn:replace(($x//orth[1]/text())[1], '\p{P}|\d+','')) > collation > > "?lang=ga" return $x), 1, 5000)}</results> > > > > If anyone has any suggestions, I would be grateful. > > > > All the best, > > Chris > > > > > > On Thu, Aug 14, 2014 at 10:35 PM, Christian Grün < > christian.gruen at gmail.com> > > wrote: > >> > >> Hi Chris, > >> > >> as you already noted, the full-text index > >> will > >> only > >> be > >> utilized with > >> the > >> options that you choose when creating an index. If you want to do more > >> fine-grained searches, it’s > >> usually > >> recommendable to > >> choose > >> the most general options for creating the index (case insensitive, > >> diacritics insensitive, etc). and > >> then > >> refine the results in a second step. > >> This can e.g. look as follows > >> : > >> > >> declare function local:search($db, $terms) { > >> for $result in db:open($db)//*[text() contains text { $terms }] > >> return $result[text() contains text { $terms } using case sensitive] > >> }; > >> local:search('factbook', ('German', 'English')) > >> > >> Hope this helps, > >> Christian > >> > >> > >> > >> On Thu, Aug 14, 2014 at 10:54 PM, Chris Yocum <cyocum at gmail.com> wrote: > >> > Hi Christian, > >> > > >> > Apologies for bringing this back up but if I use "using diacritics > >> > insensitive" in the full text search, it seems to turn full text > >> > searching off. I have diacritics true on the database. I am just > >> > suprised to see diacritics causing the full text searching to be > >> > turned off. > >> > > >> > All the best, > >> > Chris > >> > > >> > On Wed, Aug 13, 2014 at 01:18:26PM +0200, Christian Grün wrote: > >> >> Hi Chris, > >> >> > >> >> there are various caches involved when evaluating queries, but I > can't > >> >> see for the given query where a cache may be utilized. However, your > >> >> query may be evaluated faster if you simplify the nested where > clause: > >> >> > >> >> <results>{ > >> >> subsequence( > >> >> ft:mark( > >> >> for $x in collection($col)//entry > >> >> where $x//text() contains text { $term } using wildcards > >> >> order by fn:lower-case( > >> >> fn:replace(($x//orth[1]/text())[1], '\\p{P}|\\d+','') > >> >> ) collation "?lang=ga" > >> >> return $x > >> >> ), 1, 5000 > >> >> ) > >> >> }</results> > >> >> > >> >> You could as well use a predicate with position(), it may be > evaluated > >> >> faster than subsequence (I'm not sure, though, because most time will > >> >> probably be spent for ordering all results): > >> >> > >> >> <results>{ > >> >> ft:mark( > >> >> for $x in collection($col)//entry > >> >> where $x//text() contains text { $term } using wildcards > >> >> order by fn:lower-case( > >> >> fn:replace(($x//orth[1]/text())[1], '\\p{P}|\\d+','') > >> >> ) collation "?lang=ga" > >> >> return $x > >> >> )[position() = 1 to 5000] > >> >> }</results> > >> >> > >> >> Could you please open the InfoView in the GUI, execute the query > again > >> >> and check if the full-text index is applied? > >> >> > >> >> Christian > >> >> > >> >> > >> >> > >> >> On Wed, Aug 13, 2014 at 12:02 PM, Christopher Yocum < > cyocum at gmail.com> > >> >> wrote: > >> >> > declare variable $term as xs:string external; declare variable $col > >> >> > as > >> >> > xs:string external; <results>{subsequence(ft:mark(for $x in > >> >> > collection($col)//entry where $x//text()[. contains text {$term} > >> >> > using > >> >> > wildcards] order by > fn:lower-case(fn:replace(($x//orth[1]/text())[1], > >> >> > '\\p{P}|\\d+','')) collation \"?lang=ga\" return $x), 1, > >> >> > 5000)}</results> > > > > >

> DIACRITICS: true

On Tue, Aug 19, 2014 at 2:19 PM, Christopher Yocum <cyocum at gmail.com> wrote: > Hi Christian, > > I hope you had a good weekend! > > Otherwise, no, this doesn't help as it doesn't choose to use the full text > index on my content :(. This is what I am getting at the moment: > > Compiling: > - pre-evaluating fn:collection("edil") > - simplifying descendant-or-self step(s) > - converting descendant::*:entry to child steps > - simplifying descendant-or-self step(s) > - removing context expression (.) > - rewriting where clause(s) > - simplifying flwor expression > > Query: > declare variable $term as xs:string external := 'athgabāi.*'; declare > variable $col as xs:string external := 'edil'; > <results>{subsequence(ft:mark(for $x in collection($col)//entry where > $x//text() contains text {$term} using diacritics insensitive using > wildcards return $x), 1, 5000)}</results> > > Optimized Query: > element results { (fn:subsequence(ft:mark((db:open-pre("edil",0), > db:open-pre("edil",155748), ...)/*:sample/*:entry[descendant::text() > contains text "athgabāi.*" using wildcards using language 'English']), 1, > 5000)) } > > I tried this as well with the same results: > > Compiling: > - pre-evaluating fn:collection("edil") > - simplifying descendant-or-self step(s) > - converting descendant::*:entry to child steps > - removing context expression (.) > - rewriting where clause(s) > - simplifying flwor expression > > Query: > declare variable $term as xs:string external := 'athgabāi.*'; declare > variable $col as xs:string external := 'edil'; > <results>{subsequence(ft:mark(for $x in collection($col)//entry where > $x/descendant::*[text() contains text 'athgabāi.*' using diacritics > insensitive using wildcards] return $x), 1, 5000)}</results> > Optimized Query: > > element results { (fn:subsequence(ft:mark((db:open-pre("edil",0), > db:open-pre("edil",155748), ...)/*:sample/*:entry[descendant::*[text() > contains text "athgabāi.*" using wildcards using language 'English']]), 1, > 5000)) } > > There are the options set on the database: > > Database Properties > Name: edil > Size: 194 MB > Nodes: 7951662 > Documents: 19 > Binaries: 0 > Timestamp: 2014-08-15-17-00-29 > > Resource Properties > Input Path: /home/cyocum/temp/edil_src/xml_src > Input Size: 87 MB > Timestamp: 2014-08-15-16-46-31 > Encoding: UTF-8 > CHOP: true > > Indexes > Up-to-date: true > TEXTINDEX: true > ATTRINDEX: true > FTINDEX: true > LANGUAGE: > STEMMING: false > CASESENS: false > DIACRITICS: true > STOPWORDS: > UPDINDEX: false > MAXCATS: 100 > MAXLEN: 96 > > I hope this helps. > > All the best, > Chris

On Tue, Aug 19, 2014 at 1:38 PM, Christian Grün <christian.gruen at gmail.com> wrote: > Hi Chris, > > > DIACRITICS: true > > It seems as if you set the diacritics option to true (which is > equivalent to "diacritics sensitive", as it is supposed to say > "consider diacritics: yes, please!"). Could you try to rebuild the > index with the diacritics option disabled? > > Christian > > > > On Tue, Aug 19, 2014 at 2:19 PM, Christopher Yocum <cyocum at gmail.com> > wrote: > > Hi Christian, > > > > I hope you had a good weekend! > > > > Otherwise, no, this doesn't help as it doesn't choose to use the full > text > > index on my content :(. This is what I am getting at the moment: > > > > Compiling: > > - pre-evaluating fn:collection("edil") > > - simplifying descendant-or-self step(s) > > - converting descendant::*:entry to child steps > > - simplifying descendant-or-self step(s) > > - removing context expression (.) > > - rewriting where clause(s) > > - simplifying flwor expression > > > > Query: > > declare variable $term as xs:string external := 'athgabāi.*'; declare > > variable $col as xs:string external := 'edil'; > > <results>{subsequence(ft:mark(for $x in collection($col)//entry where > > $x//text() contains text {$term} using diacritics insensitive using > > wildcards return $x), 1, 5000)}</results> > > > > Optimized Query: > > element results { (fn:subsequence(ft:mark((db:open-pre("edil",0), > > db:open-pre("edil",155748), ...)/*:sample/*:entry[descendant::text() > > contains text "athgabāi.*" using wildcards using language 'English']), 1, > > 5000)) } > > > > I tried this as well with the same results: > > > > Compiling: > > - pre-evaluating fn:collection("edil") > > - simplifying descendant-or-self step(s) > > - converting descendant::*:entry to child steps > > - removing context expression (.) > > - rewriting where clause(s) > > - simplifying flwor expression > > > > Query: > > declare variable $term as xs:string external := 'athgabāi.*'; declare > > variable $col as xs:string external := 'edil'; > > <results>{subsequence(ft:mark(for $x in collection($col)//entry where > > $x/descendant::*[text() contains text 'athgabāi.*' using diacritics > > insensitive using wildcards] return $x), 1, 5000)}</results> > > Optimized Query: > > > > element results { (fn:subsequence(ft:mark((db:open-pre("edil",0), > > db:open-pre("edil",155748), ...)/*:sample/*:entry[descendant::*[text() > > contains text "athgabāi.*" using wildcards using language 'English']]), > 1, > > 5000)) } > > > > There are the options set on the database: > > > > Database Properties > > Name: edil > > Size: 194 MB > > Nodes: 7951662 > > Documents: 19 > > Binaries: 0 > > Timestamp: 2014-08-15-17-00-29 > > > > Resource Properties > > Input Path: /home/cyocum/temp/edil_src/xml_src > > Input Size: 87 MB > > Timestamp: 2014-08-15-16-46-31 > > Encoding: UTF-8 > > CHOP: true > > > > Indexes > > Up-to-date: true > > TEXTINDEX: true > > ATTRINDEX: true > > FTINDEX: true > > LANGUAGE: > > STEMMING: false > > CASESENS: false > > DIACRITICS: true > > STOPWORDS: > > UPDINDEX: false > > MAXCATS: 100 > > MAXLEN: 96 > > > > I hope this helps. > > > > All the best, > > Chris >

> Yes, that seems to make it work correctly. Maybe the wiki needs to be > updated to be more clear about what "diacritics true" does?

> On Tue, Aug 19, 2014 at 1:38 PM, Christian Grün <christian.gruen at gmail.com> > wrote: >> >> Hi Chris, >> >> > DIACRITICS: true >> >> It seems as if you set the diacritics option to true (which is >> equivalent to "diacritics sensitive", as it is supposed to say >> "consider diacritics: yes, please!"). Could you try to rebuild the >> index with the diacritics option disabled? >> >> Christian >> >> >> >> On Tue, Aug 19, 2014 at 2:19 PM, Christopher Yocum <cyocum at gmail.com> >> wrote: >> > Hi Christian, >> > >> > I hope you had a good weekend! >> > >> > Otherwise, no, this doesn't help as it doesn't choose to use the full >> > text >> > index on my content :(. This is what I am getting at the moment: >> > >> > Compiling: >> > - pre-evaluating fn:collection("edil") >> > - simplifying descendant-or-self step(s) >> > - converting descendant::*:entry to child steps >> > - simplifying descendant-or-self step(s) >> > - removing context expression (.) >> > - rewriting where clause(s) >> > - simplifying flwor expression >> > >> > Query: >> > declare variable $term as xs:string external := 'athgabāi.*'; declare >> > variable $col as xs:string external := 'edil'; >> > <results>{subsequence(ft:mark(for $x in collection($col)//entry where >> > $x//text() contains text {$term} using diacritics insensitive using >> > wildcards return $x), 1, 5000)}</results> >> > >> > Optimized Query: >> > element results { (fn:subsequence(ft:mark((db:open-pre("edil",0), >> > db:open-pre("edil",155748), ...)/*:sample/*:entry[descendant::text() >> > contains text "athgabāi.*" using wildcards using language 'English']), >> > 1, >> > 5000)) } >> > >> > I tried this as well with the same results: >> > >> > Compiling: >> > - pre-evaluating fn:collection("edil") >> > - simplifying descendant-or-self step(s) >> > - converting descendant::*:entry to child steps >> > - removing context expression (.) >> > - rewriting where clause(s) >> > - simplifying flwor expression >> > >> > Query: >> > declare variable $term as xs:string external := 'athgabāi.*'; declare >> > variable $col as xs:string external := 'edil'; >> > <results>{subsequence(ft:mark(for $x in collection($col)//entry where >> > $x/descendant::*[text() contains text 'athgabāi.*' using diacritics >> > insensitive using wildcards] return $x), 1, 5000)}</results> >> > Optimized Query: >> > >> > element results { (fn:subsequence(ft:mark((db:open-pre("edil",0), >> > db:open-pre("edil",155748), ...)/*:sample/*:entry[descendant::*[text() >> > contains text "athgabāi.*" using wildcards using language 'English']]), >> > 1, >> > 5000)) } >> > >> > There are the options set on the database: >> > >> > Database Properties >> > Name: edil >> > Size: 194 MB >> > Nodes: 7951662 >> > Documents: 19 >> > Binaries: 0 >> > Timestamp: 2014-08-15-17-00-29 >> > >> > Resource Properties >> > Input Path: /home/cyocum/temp/edil_src/xml_src >> > Input Size: 87 MB >> > Timestamp: 2014-08-15-16-46-31 >> > Encoding: UTF-8 >> > CHOP: true >> > >> > Indexes >> > Up-to-date: true >> > TEXTINDEX: true >> > ATTRINDEX: true >> > FTINDEX: true >> > LANGUAGE: >> > STEMMING: false >> > CASESENS: false >> > DIACRITICS: true >> > STOPWORDS: >> > UPDINDEX: false >> > MAXCATS: 100 >> > MAXLEN: 96 >> > >> > I hope this helps. >> > >> > All the best, >> > Chris > >

On Thu, Aug 21, 2014 at 11:00:52PM +0200, Christian Grün wrote: > Hi Chris, > > > Yes, that seems to make it work correctly. Maybe the wiki needs to be > > updated to be more clear about what "diacritics true" does? > > I have slightly updated the text entries in our Wiki [1]. You are > invited to register for the Wiki and update the text if you believe it > could be further improved.

> > Beside that, I am glad to report that I have made our query optimizer > a bit smarter. With the latest snapshot [2], your original query with > the additional predicate will now be automatically rewritten to the > second version, and will also be rewritten to take advantage of the > full-text index.

> > Hope this helps, > Christian > > [1] http://docs.basex.org/wiki/Options#Full-Text > [2] http://files.basex.org/releases/latest/ > > > > > On Tue, Aug 19, 2014 at 1:38 PM, Christian Grün <christian.gruen at gmail.com> > > wrote: > >> > >> Hi Chris, > >> > >> > DIACRITICS: true > >> > >> It seems as if you set the diacritics option to true (which is > >> equivalent to "diacritics sensitive", as it is supposed to say > >> "consider diacritics: yes, please!"). Could you try to rebuild the > >> index with the diacritics option disabled? > >> > >> Christian > >> > >> > >> > >> On Tue, Aug 19, 2014 at 2:19 PM, Christopher Yocum <cyocum at gmail.com> > >> wrote: > >> > Hi Christian, > >> > > >> > I hope you had a good weekend! > >> > > >> > Otherwise, no, this doesn't help as it doesn't choose to use the full > >> > text > >> > index on my content :(. This is what I am getting at the moment: > >> > > >> > Compiling: > >> > - pre-evaluating fn:collection("edil") > >> > - simplifying descendant-or-self step(s) > >> > - converting descendant::*:entry to child steps > >> > - simplifying descendant-or-self step(s) > >> > - removing context expression (.) > >> > - rewriting where clause(s) > >> > - simplifying flwor expression > >> > > >> > Query: > >> > declare variable $term as xs:string external := 'athgabāi.*'; declare > >> > variable $col as xs:string external := 'edil'; > >> > <results>{subsequence(ft:mark(for $x in collection($col)//entry where > >> > $x//text() contains text {$term} using diacritics insensitive using > >> > wildcards return $x), 1, 5000)}</results> > >> > > >> > Optimized Query: > >> > element results { (fn:subsequence(ft:mark((db:open-pre("edil",0), > >> > db:open-pre("edil",155748), ...)/*:sample/*:entry[descendant::text() > >> > contains text "athgabāi.*" using wildcards using language 'English']), > >> > 1, > >> > 5000)) } > >> > > >> > I tried this as well with the same results: > >> > > >> > Compiling: > >> > - pre-evaluating fn:collection("edil") > >> > - simplifying descendant-or-self step(s) > >> > - converting descendant::*:entry to child steps > >> > - removing context expression (.) > >> > - rewriting where clause(s) > >> > - simplifying flwor expression > >> > > >> > Query: > >> > declare variable $term as xs:string external := 'athgabāi.*'; declare > >> > variable $col as xs:string external := 'edil'; > >> > <results>{subsequence(ft:mark(for $x in collection($col)//entry where > >> > $x/descendant::*[text() contains text 'athgabāi.*' using diacritics > >> > insensitive using wildcards] return $x), 1, 5000)}</results> > >> > Optimized Query: > >> > > >> > element results { (fn:subsequence(ft:mark((db:open-pre("edil",0), > >> > db:open-pre("edil",155748), ...)/*:sample/*:entry[descendant::*[text() > >> > contains text "athgabāi.*" using wildcards using language 'English']]), > >> > 1, > >> > 5000)) } > >> > > >> > There are the options set on the database: > >> > > >> > Database Properties > >> > Name: edil > >> > Size: 194 MB > >> > Nodes: 7951662 > >> > Documents: 19 > >> > Binaries: 0 > >> > Timestamp: 2014-08-15-17-00-29 > >> > > >> > Resource Properties > >> > Input Path: /home/cyocum/temp/edil_src/xml_src > >> > Input Size: 87 MB > >> > Timestamp: 2014-08-15-16-46-31 > >> > Encoding: UTF-8 > >> > CHOP: true > >> > > >> > Indexes > >> > Up-to-date: true > >> > TEXTINDEX: true > >> > ATTRINDEX: true > >> > FTINDEX: true > >> > LANGUAGE: > >> > STEMMING: false > >> > CASESENS: false > >> > DIACRITICS: true > >> > STOPWORDS: > >> > UPDINDEX: false > >> > MAXCATS: 100 > >> > MAXLEN: 96 > >> > > >> > I hope this helps. > >> > > >> > All the best, > >> > Chris > > > >