Fulltext index not applied

On Mon, Nov 12, 2018 at 9:54 AM Sebastian Zimmer < sebastian.zimmer at uni-koeln.de> wrote: > Hi, > > I'm working on a query where the full-text index is not applied, even > though it could be (see attached test.xql). > > I'm sorry for the somewhat complicated look of the example with > data-specific XPaths and so on. I really have tried to simplify it as much > as I could, but when I further simplify it, the fulltext index actually > will be applied or the query is optimized to () due to cached evaluations. > > But if you take the expression of the variable $number and inline it into > below where the variable is referenced, you may see what I mean. The result > is an equivalent query (see attached test-ft.xql) to the original one, but > now the fulltext index *is* applied. > > I'm using the BaseX 9.1 snapshot from 2018-11-05. > > Best, > Sebastian > -- > Sebastian Zimmer > sebastian.zimmer at uni-koeln.de > [image: CCeH Logo] <http://cceh.uni-koeln.de> > > Cologne Center for eHumanities <http://cceh.uni-koeln.de> > DH Center at the University of Cologne > [image: Twitter Logo] <https://twitter.com/CCeHum>@CCeHum > <https://twitter.com/CCeHum> >

Am 12.11.2018 um 17:24 schrieb Christian Grün: > Hi Sebastian, > > It’s difficult to guess what the query optimizer does by just looking > at the query. Could you pass us on a database document that allows us > to run the query? > > Any attempt to simplify your query is appreciated. Here are some more > things that you could try: > • Does it make a difference if you work with namespaces, or could all > prefixes in the location steps be replaced with *? > • Does the issue only occur if your code is defined in the > local:search function? > • What happens if you replace the functx function call with its body > ($value = $seq)? > > Best, > Christian > > > > On Mon, Nov 12, 2018 at 9:54 AM Sebastian Zimmer > <sebastian.zimmer at uni-koeln.de <mailto:sebastian.zimmer at uni-koeln.de>> > wrote: > > Hi, > > I'm working on a query where the full-text index is not applied, > even though it could be (see attached test.xql). > > I'm sorry for the somewhat complicated look of the example with > data-specific XPaths and so on. I really have tried to simplify it > as much as I could, but when I further simplify it, the fulltext > index actually will be applied or the query is optimized to () due > to cached evaluations. > > But if you take the expression of the variable $number and inline > it into below where the variable is referenced, you may see what I > mean. The result is an equivalent query (see attached test-ft.xql) > to the original one, but now the fulltext index *is* applied. > > I'm using the BaseX 9.1 snapshot from 2018-11-05. > > Best, > Sebastian > > -- > Sebastian Zimmer > sebastian.zimmer at uni-koeln.de <mailto:sebastian.zimmer at uni-koeln.de> > CCeH Logo <http://cceh.uni-koeln.de> > > Cologne Center for eHumanities <http://cceh.uni-koeln.de> > DH Center at the University of Cologne > Twitter Logo <https://twitter.com/CCeHum>@CCeHum > <https://twitter.com/CCeHum> >

On Mon, Nov 12, 2018 at 6:11 PM Sebastian Zimmer < sebastian.zimmer at uni-koeln.de> wrote: > Hi Christian, > > thanks for your help attempt. I understand that the query is not really > simple yet and that it's hard to see what's going on. > > Attached is a database document on which you can run the query. > > To your questions: > > * It does make no difference if the namespace prefixes are replaced with > *. Thus you can simplify the query in that manner and the issue persists. > * Yes, the issue does only occur if the code is within the local:search > function. If the code is in the global scope of the file, the ft-index is > applied. Thus, I cannot simplify the query that way. > * If I replace the functx function with its body, the ft-index is applied. > Thus, I cannot simplify the query that way. > > I'm happy to help further if I can. > > Best, > Sebastian > > > Am 12.11.2018 um 17:24 schrieb Christian Grün: > > Hi Sebastian, > > It’s difficult to guess what the query optimizer does by just looking at > the query. Could you pass us on a database document that allows us to run > the query? > > Any attempt to simplify your query is appreciated. Here are some more > things that you could try: > • Does it make a difference if you work with namespaces, or could all > prefixes in the location steps be replaced with *? > • Does the issue only occur if your code is defined in the local:search > function? > • What happens if you replace the functx function call with its body > ($value = $seq)? > > Best, > Christian > > > > On Mon, Nov 12, 2018 at 9:54 AM Sebastian Zimmer < > sebastian.zimmer at uni-koeln.de> wrote: > >> Hi, >> >> I'm working on a query where the full-text index is not applied, even >> though it could be (see attached test.xql). >> >> I'm sorry for the somewhat complicated look of the example with >> data-specific XPaths and so on. I really have tried to simplify it as much >> as I could, but when I further simplify it, the fulltext index actually >> will be applied or the query is optimized to () due to cached evaluations. >> >> But if you take the expression of the variable $number and inline it into >> below where the variable is referenced, you may see what I mean. The result >> is an equivalent query (see attached test-ft.xql) to the original one, but >> now the fulltext index *is* applied. >> >> I'm using the BaseX 9.1 snapshot from 2018-11-05. >> >> Best, >> Sebastian >> -- >> Sebastian Zimmer >> sebastian.zimmer at uni-koeln.de >> [image: CCeH Logo] <http://cceh.uni-koeln.de> >> >> Cologne Center for eHumanities <http://cceh.uni-koeln.de> >> DH Center at the University of Cologne >> [image: Twitter Logo] <https://twitter.com/CCeHum>@CCeHum >> <https://twitter.com/CCeHum> >> > > -- > Sebastian Zimmer > sebastian.zimmer at uni-koeln.de > [image: CCeH Logo] <http://cceh.uni-koeln.de> > > Cologne Center for eHumanities <http://cceh.uni-koeln.de> > DH Center at the University of Cologne > [image: Twitter Logo] <https://twitter.com/CCeHum>@CCeHum > <https://twitter.com/CCeHum> >

Am 16.11.2018 um 13:06 schrieb Christian Grün: > Hi Sebastian, > > I agree, it is kind of impossible to simplify your query. The reason > is function inlining: > > Original query: > declare function local:greet($name) { 'hi ' || $name }; > local:greet('john') > > Optimized query (intermediate): > let $name_1 := 'john' > return 'hi ' || $name_1 > > Optimized query (final): > 'hi john' > > If a function body is too large, it won’t be inlined (otherwise, > recursive functions can easily blow up main memory). The inline limit > in BaseX is a static value, it can be controlled via the INLINELIMIT > option [1]. > > Coming back to your query, test.xql won’t be optimized for index > access because… > 1. The function body is too large > 2. The passed on JSON string cannot be statically converted to a map > 3. As a consequence, it cannot statically be determined if search will > be fuzzy or not. > 4. As a consequence, the predicate with the full-text expression > contains multiple options, and none of them will be rewritten for > index access. > > You can e.g.b tackle this by defining a larger inline limit for this > function in your function declaration (see the attached link). As > stated before, you can also call ft:search directly and pass on the > fuzzy option as parameter. > > A general hint: With BaseX 9.1, your map lookup can be simplified to… > > $map?key2?key3 ?: ("1", "2") > > …or… > > util:or($map?key2?key3, ("1", "2")) > > If you want to use standard XQuery 3.1 language, you can write: > > let $key := $map?key2?key3 > return if(exists($key)) then $key else ("1", "2") > > This will ensure that your map will only be looked up once. > > Best, > Christian > > [1] http://docs.basex.org/wiki/Options#INLINELIMIT > > > > On Mon, Nov 12, 2018 at 6:11 PM Sebastian Zimmer > <sebastian.zimmer at uni-koeln.de <mailto:sebastian.zimmer at uni-koeln.de>> > wrote: > > Hi Christian, > > thanks for your help attempt. I understand that the query is not > really simple yet and that it's hard to see what's going on. > > Attached is a database document on which you can run the query. > > To your questions: > > * It does make no difference if the namespace prefixes are > replaced with *. Thus you can simplify the query in that manner > and the issue persists. > * Yes, the issue does only occur if the code is within the > local:search function. If the code is in the global scope of the > file, the ft-index is applied. Thus, I cannot simplify the query > that way. > * If I replace the functx function with its body, the ft-index is > applied. Thus, I cannot simplify the query that way. > > I'm happy to help further if I can. > > Best, > Sebastian > > > Am 12.11.2018 um 17:24 schrieb Christian Grün: >> Hi Sebastian, >> >> It’s difficult to guess what the query optimizer does by just >> looking at the query. Could you pass us on a database document >> that allows us to run the query? >> >> Any attempt to simplify your query is appreciated. Here are some >> more things that you could try: >> • Does it make a difference if you work with namespaces, or could >> all prefixes in the location steps be replaced with *? >> • Does the issue only occur if your code is defined in the >> local:search function? >> • What happens if you replace the functx function call with its >> body ($value = $seq)? >> >> Best, >> Christian >> >> >> >> On Mon, Nov 12, 2018 at 9:54 AM Sebastian Zimmer >> <sebastian.zimmer at uni-koeln.de >> <mailto:sebastian.zimmer at uni-koeln.de>> wrote: >> >> Hi, >> >> I'm working on a query where the full-text index is not >> applied, even though it could be (see attached test.xql). >> >> I'm sorry for the somewhat complicated look of the example >> with data-specific XPaths and so on. I really have tried to >> simplify it as much as I could, but when I further simplify >> it, the fulltext index actually will be applied or the query >> is optimized to () due to cached evaluations. >> >> But if you take the expression of the variable $number and >> inline it into below where the variable is referenced, you >> may see what I mean. The result is an equivalent query (see >> attached test-ft.xql) to the original one, but now the >> fulltext index *is* applied. >> >> I'm using the BaseX 9.1 snapshot from 2018-11-05. >> >> Best, >> Sebastian >> >> -- >> Sebastian Zimmer >> sebastian.zimmer at uni-koeln.de >> <mailto:sebastian.zimmer at uni-koeln.de> >> CCeH Logo <http://cceh.uni-koeln.de> >> >> Cologne Center for eHumanities <http://cceh.uni-koeln.de> >> DH Center at the University of Cologne >> Twitter Logo <https://twitter.com/CCeHum>@CCeHum >> <https://twitter.com/CCeHum> >> > > -- > Sebastian Zimmer > sebastian.zimmer at uni-koeln.de <mailto:sebastian.zimmer at uni-koeln.de> > CCeH Logo <http://cceh.uni-koeln.de> > > Cologne Center for eHumanities <http://cceh.uni-koeln.de> > DH Center at the University of Cologne > Twitter Logo <https://twitter.com/CCeHum>@CCeHum > <https://twitter.com/CCeHum> >