ft search with wildcards does not work with higher Unicode characters

Omar Siam <Omar.Siam at oeaw.ac.at> schrieb am Mi., 5. Feb. 2020, 17:02: > Hi, > > I just came across this strange behavior > > collection('dc_tunico')//*[text() contains text 'rwḥ' using wildcards] > > yields nothing vs > > collection('dc_tunico')//*[text() contains text 'rwḥ'] > > yields the correct result > > <gram xmlns="http://www.tei-c.org/ns/1.0" <http://www.tei-c.org/ns/1.0> type="root" xml:lang="ar-aeb-x-vicav">rwḥ</gram> > <gram xmlns="http://www.tei-c.org/ns/1.0" <http://www.tei-c.org/ns/1.0> type="root" xml:lang="ar-aeb-x-vicav">rwḥ</gram> > <gram xmlns="http://www.tei-c.org/ns/1.0" <http://www.tei-c.org/ns/1.0> type="root" xml:lang="ar-aeb-x-vicav">rwḥ</gram> > <gram xmlns="http://www.tei-c.org/ns/1.0" <http://www.tei-c.org/ns/1.0> type="root" xml:lang="ar-aeb-x-tunis-vicav">rwḥ</gram> > > Any ideas why this is the case? > > Best regards > > Omar >

Am 05.02.2020 um 19:59 schrieb Christian Grün: > Dear Omar, > > At about the same time when you wrote this, we have fixed a little bug > that occurred with the wildcards option [1]. Could you have a look at > the latest snapshot [2] and report back to us if it resolves the issue? > > Thanks in advance, > Christian > > [1] https://github.com/BaseXdb/basex/issues/1800 > [2] http://files.basex.org/releases/latest/ > > > > > Omar Siam <Omar.Siam at oeaw.ac.at <mailto:Omar.Siam at oeaw.ac.at>> schrieb > am Mi., 5. Feb. 2020, 17:02: > > Hi, > > I just came across this strange behavior > > collection('dc_tunico')//*[text() contains text 'rwḥ' using wildcards] > > yields nothing vs > > collection('dc_tunico')//*[text() contains text 'rwḥ'] > > yields the correct result > > <gram xmlns="http://www.tei-c.org/ns/1.0" <http://www.tei-c.org/ns/1.0> type="root" xml:lang="ar-aeb-x-vicav">rwḥ</gram> > <gram xmlns="http://www.tei-c.org/ns/1.0" <http://www.tei-c.org/ns/1.0> type="root" xml:lang="ar-aeb-x-vicav">rwḥ</gram> > <gram xmlns="http://www.tei-c.org/ns/1.0" <http://www.tei-c.org/ns/1.0> type="root" xml:lang="ar-aeb-x-vicav">rwḥ</gram> > <gram xmlns="http://www.tei-c.org/ns/1.0" <http://www.tei-c.org/ns/1.0> type="root" xml:lang="ar-aeb-x-tunis-vicav">rwḥ</gram> > > Any ideas why this is the case? > > Best regards > > Omar >

> Yes, that seems to solve the problem partly. Using wildcards now yields the same result as no wildcards.

> But if there is a complex unicode character in the search string, "." for one character looses its meaning. > … > Would you like a PR for the test gh1800 using complex unicode characters?

Am 06.02.2020 um 13:35 schrieb Christian Grün: > Hi Omar, > >> Yes, that seems to solve the problem partly. Using wildcards now yields the same result as no wildcards. > Glad to hear. > >> But if there is a complex unicode character in the search string, "." for one character looses its meaning. >> … >> Would you like a PR for the test gh1800 using complex unicode characters? > A little test case would be helpful indeed. It seems to be a different issue: > > • The first expression is evaluated without the full-text expression. > The reason is that the full-text index algorithms are limited to basic > regular expressions; not all of them can be answered by an index (and > 'r{1,1}' is currently not detected as being identical to `r.`). If I > remember correctly, the index will not be accessed either if a pattern > starts with `.*` (this pattern would lead to a full index scan). > > • The second expression is rewritten for index access. I tried to > build a little command script (test.bxs), but it doesn’t seem to > reflect the case you encountered: > > set ftindex true > create db test <xml>rwḥ</xml> > xquery /*[text() contains text 'r.{1,1}ḥ' using wildcards] > xquery /*[text() contains text 'r.ḥ' using wildcards] > close > > Could you extend this example script a little, such that it > demonstrates what goes wrong? > > Thanks in advance, > Christian

> I just tried to use the gh1800 test to replicate my problem and it does > not show there. It fails using the GUI.

> Am 06.02.2020 um 13:35 schrieb Christian Grün: > > Hi Omar, > > > >> Yes, that seems to solve the problem partly. Using wildcards now yields the same result as no wildcards. > > Glad to hear. > > > >> But if there is a complex unicode character in the search string, "." for one character looses its meaning. > >> … > >> Would you like a PR for the test gh1800 using complex unicode characters? > > A little test case would be helpful indeed. It seems to be a different issue: > > > > • The first expression is evaluated without the full-text expression. > > The reason is that the full-text index algorithms are limited to basic > > regular expressions; not all of them can be answered by an index (and > > 'r{1,1}' is currently not detected as being identical to `r.`). If I > > remember correctly, the index will not be accessed either if a pattern > > starts with `.*` (this pattern would lead to a full index scan). > > > > • The second expression is rewritten for index access. I tried to > > build a little command script (test.bxs), but it doesn’t seem to > > reflect the case you encountered: > > > > set ftindex true > > create db test <xml>rwḥ</xml> > > xquery /*[text() contains text 'r.{1,1}ḥ' using wildcards] > > xquery /*[text() contains text 'r.ḥ' using wildcards] > > close > > > > Could you extend this example script a little, such that it > > demonstrates what goes wrong? > > > > Thanks in advance, > > Christian

Am 06.02.2020 um 13:45 schrieb Christian Grün: >> I just tried to use the gh1800 test to replicate my problem and it does >> not show there. It fails using the GUI. > I need your help: What does not show there? What fails, what happens? > > > > >> Am 06.02.2020 um 13:35 schrieb Christian Grün: >>> Hi Omar, >>> >>>> Yes, that seems to solve the problem partly. Using wildcards now yields the same result as no wildcards. >>> Glad to hear. >>> >>>> But if there is a complex unicode character in the search string, "." for one character looses its meaning. >>>> … >>>> Would you like a PR for the test gh1800 using complex unicode characters? >>> A little test case would be helpful indeed. It seems to be a different issue: >>> >>> • The first expression is evaluated without the full-text expression. >>> The reason is that the full-text index algorithms are limited to basic >>> regular expressions; not all of them can be answered by an index (and >>> 'r{1,1}' is currently not detected as being identical to `r.`). If I >>> remember correctly, the index will not be accessed either if a pattern >>> starts with `.*` (this pattern would lead to a full index scan). >>> >>> • The second expression is rewritten for index access. I tried to >>> build a little command script (test.bxs), but it doesn’t seem to >>> reflect the case you encountered: >>> >>> set ftindex true >>> create db test <xml>rwḥ</xml> >>> xquery /*[text() contains text 'r.{1,1}ḥ' using wildcards] >>> xquery /*[text() contains text 'r.ḥ' using wildcards] >>> close >>> >>> Could you extend this example script a little, such that it >>> demonstrates what goes wrong? >>> >>> Thanks in advance, >>> Christian

On Thu, Feb 6, 2020 at 1:51 PM Omar Siam <Omar.Siam at oeaw.ac.at> wrote: > > The following XQuery run in the GUI (pulled from github and built a few minutes ago from source) > > ft:tokens('testdata'), > ft:search('testdata', 'r.ḥ', map {'wildcards': true()})/.., '----------' > ,collection('testdata')//*[text() contains text 'r.ḥ' using wildcards] > > yields > > <entry count="4">rwḥ</entry> > ---------- > > with collection('testdata') > > <_> > <gram xmlns="http://www.tei-c.org/ns/1.0" type="root" xml:lang="ar-aeb-x-vicav">rwḥ</gram> > <gram xmlns="http://www.tei-c.org/ns/1.0" type="root" xml:lang="ar-aeb-x-vicav">rwḥ</gram> > <gram xmlns="http://www.tei-c.org/ns/1.0" type="root" xml:lang="ar-aeb-x-vicav">rwḥ</gram> > <gram xmlns="http://www.tei-c.org/ns/1.0" type="root" xml:lang="ar-aeb-x-tunis-vicav">rwḥ</gram> > </_> > > But the gh1800() test changed like this: > > final String text = "999 aa 1111 rwḥ"; > [...] > query("ft:search('" + NAME + "', 'r.ḥ', " + options + ")", text); > > works. > > Am 06.02.2020 um 13:45 schrieb Christian Grün: > > I just tried to use the gh1800 test to replicate my problem and it does > not show there. It fails using the GUI. > > I need your help: What does not show there? What fails, what happens? > > > > > Am 06.02.2020 um 13:35 schrieb Christian Grün: > > Hi Omar, > > Yes, that seems to solve the problem partly. Using wildcards now yields the same result as no wildcards. > > Glad to hear. > > But if there is a complex unicode character in the search string, "." for one character looses its meaning. > … > Would you like a PR for the test gh1800 using complex unicode characters? > > A little test case would be helpful indeed. It seems to be a different issue: > > • The first expression is evaluated without the full-text expression. > The reason is that the full-text index algorithms are limited to basic > regular expressions; not all of them can be answered by an index (and > 'r{1,1}' is currently not detected as being identical to `r.`). If I > remember correctly, the index will not be accessed either if a pattern > starts with `.*` (this pattern would lead to a full index scan). > > • The second expression is rewritten for index access. I tried to > build a little command script (test.bxs), but it doesn’t seem to > reflect the case you encountered: > > set ftindex true > create db test <xml>rwḥ</xml> > xquery /*[text() contains text 'r.{1,1}ḥ' using wildcards] > xquery /*[text() contains text 'r.ḥ' using wildcards] > close > > Could you extend this example script a little, such that it > demonstrates what goes wrong? > > Thanks in advance, > Christian

Am 06.02.2020 um 14:11 schrieb Christian Grün: > Looks we are getting closer. I have attached a little command script; > do you think you can adapt it such that it captures the potential bug? > > > On Thu, Feb 6, 2020 at 1:51 PM Omar Siam <Omar.Siam at oeaw.ac.at> wrote: >> The following XQuery run in the GUI (pulled from github and built a few minutes ago from source) >> >> ft:tokens('testdata'), >> ft:search('testdata', 'r.ḥ', map {'wildcards': true()})/.., '----------' >> ,collection('testdata')//*[text() contains text 'r.ḥ' using wildcards] >> >> yields >> >> <entry count="4">rwḥ</entry> >> ---------- >> >> with collection('testdata') >> >> <_> >> <gram xmlns="http://www.tei-c.org/ns/1.0" type="root" xml:lang="ar-aeb-x-vicav">rwḥ</gram> >> <gram xmlns="http://www.tei-c.org/ns/1.0" type="root" xml:lang="ar-aeb-x-vicav">rwḥ</gram> >> <gram xmlns="http://www.tei-c.org/ns/1.0" type="root" xml:lang="ar-aeb-x-vicav">rwḥ</gram> >> <gram xmlns="http://www.tei-c.org/ns/1.0" type="root" xml:lang="ar-aeb-x-tunis-vicav">rwḥ</gram> >> </_> >> >> But the gh1800() test changed like this: >> >> final String text = "999 aa 1111 rwḥ"; >> [...] >> query("ft:search('" + NAME + "', 'r.ḥ', " + options + ")", text); >> >> works. >> >> Am 06.02.2020 um 13:45 schrieb Christian Grün: >> >> I just tried to use the gh1800 test to replicate my problem and it does >> not show there. It fails using the GUI. >> >> I need your help: What does not show there? What fails, what happens? >> >> >> >> >> Am 06.02.2020 um 13:35 schrieb Christian Grün: >> >> Hi Omar, >> >> Yes, that seems to solve the problem partly. Using wildcards now yields the same result as no wildcards. >> >> Glad to hear. >> >> But if there is a complex unicode character in the search string, "." for one character looses its meaning. >> … >> Would you like a PR for the test gh1800 using complex unicode characters? >> >> A little test case would be helpful indeed. It seems to be a different issue: >> >> • The first expression is evaluated without the full-text expression. >> The reason is that the full-text index algorithms are limited to basic >> regular expressions; not all of them can be answered by an index (and >> 'r{1,1}' is currently not detected as being identical to `r.`). If I >> remember correctly, the index will not be accessed either if a pattern >> starts with `.*` (this pattern would lead to a full index scan). >> >> • The second expression is rewritten for index access. I tried to >> build a little command script (test.bxs), but it doesn’t seem to >> reflect the case you encountered: >> >> set ftindex true >> create db test <xml>rwḥ</xml> >> xquery /*[text() contains text 'r.{1,1}ḥ' using wildcards] >> xquery /*[text() contains text 'r.ḥ' using wildcards] >> close >> >> Could you extend this example script a little, such that it >> demonstrates what goes wrong? >> >> Thanks in advance, >> Christian

> I can try. But what you tell me and what I saw with the unit test function I guess the GUI does something that can't be observed when using scripts.

> Unfortunately I get new problems with 9.3.2 Beta:

> declare function local:maptest() as map(xs:string, xs:string) { > let $xml := <_ test="test"/> > return map{xs:string($xml/@test): xs:string('test')} > }; > > local:maptest() > > stopped working. > > [XPTY0004] Cannot convert map(*) to map(xs:string, xs:string): map { "test": "test" }. > > And I get more new type conversion related errors that were not reported before. This is temporary I hope? > > Best regards > > Omar > > Am 06.02.2020 um 14:11 schrieb Christian Grün: > > Looks we are getting closer. I have attached a little command script; > do you think you can adapt it such that it captures the potential bug? > > > On Thu, Feb 6, 2020 at 1:51 PM Omar Siam <Omar.Siam at oeaw.ac.at> wrote: > > The following XQuery run in the GUI (pulled from github and built a few minutes ago from source) > > ft:tokens('testdata'), > ft:search('testdata', 'r.ḥ', map {'wildcards': true()})/.., '----------' > ,collection('testdata')//*[text() contains text 'r.ḥ' using wildcards] > > yields > > <entry count="4">rwḥ</entry> > ---------- > > with collection('testdata') > > <_> > <gram xmlns="http://www.tei-c.org/ns/1.0" type="root" xml:lang="ar-aeb-x-vicav">rwḥ</gram> > <gram xmlns="http://www.tei-c.org/ns/1.0" type="root" xml:lang="ar-aeb-x-vicav">rwḥ</gram> > <gram xmlns="http://www.tei-c.org/ns/1.0" type="root" xml:lang="ar-aeb-x-vicav">rwḥ</gram> > <gram xmlns="http://www.tei-c.org/ns/1.0" type="root" xml:lang="ar-aeb-x-tunis-vicav">rwḥ</gram> > </_> > > But the gh1800() test changed like this: > > final String text = "999 aa 1111 rwḥ"; > [...] > query("ft:search('" + NAME + "', 'r.ḥ', " + options + ")", text); > > works. > > Am 06.02.2020 um 13:45 schrieb Christian Grün: > > I just tried to use the gh1800 test to replicate my problem and it does > not show there. It fails using the GUI. > > I need your help: What does not show there? What fails, what happens? > > > > > Am 06.02.2020 um 13:35 schrieb Christian Grün: > > Hi Omar, > > Yes, that seems to solve the problem partly. Using wildcards now yields the same result as no wildcards. > > Glad to hear. > > But if there is a complex unicode character in the search string, "." for one character looses its meaning. > … > Would you like a PR for the test gh1800 using complex unicode characters? > > A little test case would be helpful indeed. It seems to be a different issue: > > • The first expression is evaluated without the full-text expression. > The reason is that the full-text index algorithms are limited to basic > regular expressions; not all of them can be answered by an index (and > 'r{1,1}' is currently not detected as being identical to `r.`). If I > remember correctly, the index will not be accessed either if a pattern > starts with `.*` (this pattern would lead to a full index scan). > > • The second expression is rewritten for index access. I tried to > build a little command script (test.bxs), but it doesn’t seem to > reflect the case you encountered: > > set ftindex true > create db test <xml>rwḥ</xml> > xquery /*[text() contains text 'r.{1,1}ḥ' using wildcards] > xquery /*[text() contains text 'r.ḥ' using wildcards] > close > > Could you extend this example script a little, such that it > demonstrates what goes wrong? > > Thanks in advance, > Christian

Am 06.02.2020 um 14:40 schrieb Christian Grün: >> I can try. But what you tell me and what I saw with the unit test function I guess the GUI does something that can't be observed when using scripts. > Maybe some options are still active in your GUI that are not reflected > in the script? > Does the script work if you run it on command line? > >> Unfortunately I get new problems with 9.3.2 Beta: > …most probably related to recent typecheck optimizations that we > recently introduced [1]; sorry for that. If you need a very short-term > alternative, you can use string() instead of xs:string(). > > [1] https://github.com/BaseXdb/basex/issues/1801 > > > >> declare function local:maptest() as map(xs:string, xs:string) { >> let $xml := <_ test="test"/> >> return map{xs:string($xml/@test): xs:string('test')} >> }; >> >> local:maptest() >> >> stopped working. >> >> [XPTY0004] Cannot convert map(*) to map(xs:string, xs:string): map { "test": "test" }. >> >> And I get more new type conversion related errors that were not reported before. This is temporary I hope? >> >> Best regards >> >> Omar >> >> Am 06.02.2020 um 14:11 schrieb Christian Grün: >> >> Looks we are getting closer. I have attached a little command script; >> do you think you can adapt it such that it captures the potential bug? >> >> >> On Thu, Feb 6, 2020 at 1:51 PM Omar Siam <Omar.Siam at oeaw.ac.at> wrote: >> >> The following XQuery run in the GUI (pulled from github and built a few minutes ago from source) >> >> ft:tokens('testdata'), >> ft:search('testdata', 'r.ḥ', map {'wildcards': true()})/.., '----------' >> ,collection('testdata')//*[text() contains text 'r.ḥ' using wildcards] >> >> yields >> >> <entry count="4">rwḥ</entry> >> ---------- >> >> with collection('testdata') >> >> <_> >> <gram xmlns="http://www.tei-c.org/ns/1.0" type="root" xml:lang="ar-aeb-x-vicav">rwḥ</gram> >> <gram xmlns="http://www.tei-c.org/ns/1.0" type="root" xml:lang="ar-aeb-x-vicav">rwḥ</gram> >> <gram xmlns="http://www.tei-c.org/ns/1.0" type="root" xml:lang="ar-aeb-x-vicav">rwḥ</gram> >> <gram xmlns="http://www.tei-c.org/ns/1.0" type="root" xml:lang="ar-aeb-x-tunis-vicav">rwḥ</gram> >> </_> >> >> But the gh1800() test changed like this: >> >> final String text = "999 aa 1111 rwḥ"; >> [...] >> query("ft:search('" + NAME + "', 'r.ḥ', " + options + ")", text); >> >> works. >> >> Am 06.02.2020 um 13:45 schrieb Christian Grün: >> >> I just tried to use the gh1800 test to replicate my problem and it does >> not show there. It fails using the GUI. >> >> I need your help: What does not show there? What fails, what happens? >> >> >> >> >> Am 06.02.2020 um 13:35 schrieb Christian Grün: >> >> Hi Omar, >> >> Yes, that seems to solve the problem partly. Using wildcards now yields the same result as no wildcards. >> >> Glad to hear. >> >> But if there is a complex unicode character in the search string, "." for one character looses its meaning. >> … >> Would you like a PR for the test gh1800 using complex unicode characters? >> >> A little test case would be helpful indeed. It seems to be a different issue: >> >> • The first expression is evaluated without the full-text expression. >> The reason is that the full-text index algorithms are limited to basic >> regular expressions; not all of them can be answered by an index (and >> 'r{1,1}' is currently not detected as being identical to `r.`). If I >> remember correctly, the index will not be accessed either if a pattern >> starts with `.*` (this pattern would lead to a full index scan). >> >> • The second expression is rewritten for index access. I tried to >> build a little command script (test.bxs), but it doesn’t seem to >> reflect the case you encountered: >> >> set ftindex true >> create db test <xml>rwḥ</xml> >> xquery /*[text() contains text 'r.{1,1}ḥ' using wildcards] >> xquery /*[text() contains text 'r.ḥ' using wildcards] >> close >> >> Could you extend this example script a little, such that it >> demonstrates what goes wrong? >> >> Thanks in advance, >> Christian

On Thu, Feb 6, 2020 at 3:03 PM Omar Siam <Omar.Siam at oeaw.ac.at> wrote: > > I think I've got it: for reasons I am not 100% sure about right now a usually select diacritics: true() > > So the following does not work as expected: > > <commands> > <xquery><![CDATA[ > let $xml := <_> > <gram xmlns="http://www.tei-c.org/ns/1.0" type="root" xml:lang="ar-aeb-x-vicav">rwḥ</gram> > <gram xmlns="http://www.tei-c.org/ns/1.0" type="root" xml:lang="ar-aeb-x-vicav">rwḥ</gram> > <gram xmlns="http://www.tei-c.org/ns/1.0" type="root" xml:lang="ar-aeb-x-vicav">rwḥ</gram> > <gram xmlns="http://www.tei-c.org/ns/1.0" type="root" xml:lang="ar-aeb-x-tunis-vicav">rwḥ</gram> > </_> > return db:create('testdata', $xml, 'testdata.xml', map { 'ftindex': true(), 'diacritics': true() }) > ]]></xquery> > <xquery> > db:open('testdata')//*[text() contains text 'r.{1,1}ḥ' using wildcards using diacritics sensitive] > </xquery> > <xquery> > db:open('testdata')//*[text() contains text 'r.ḥ' using wildcards using diacritics sensitive] > </xquery> > </commands> > > Am 06.02.2020 um 14:40 schrieb Christian Grün: > > I can try. But what you tell me and what I saw with the unit test function I guess the GUI does something that can't be observed when using scripts. > > Maybe some options are still active in your GUI that are not reflected > in the script? > Does the script work if you run it on command line? > > Unfortunately I get new problems with 9.3.2 Beta: > > …most probably related to recent typecheck optimizations that we > recently introduced [1]; sorry for that. If you need a very short-term > alternative, you can use string() instead of xs:string(). > > [1] https://github.com/BaseXdb/basex/issues/1801 > > > > declare function local:maptest() as map(xs:string, xs:string) { > let $xml := <_ test="test"/> > return map{xs:string($xml/@test): xs:string('test')} > }; > > local:maptest() > > stopped working. > > [XPTY0004] Cannot convert map(*) to map(xs:string, xs:string): map { "test": "test" }. > > And I get more new type conversion related errors that were not reported before. This is temporary I hope? > > Best regards > > Omar > > Am 06.02.2020 um 14:11 schrieb Christian Grün: > > Looks we are getting closer. I have attached a little command script; > do you think you can adapt it such that it captures the potential bug? > > > On Thu, Feb 6, 2020 at 1:51 PM Omar Siam <Omar.Siam at oeaw.ac.at> wrote: > > The following XQuery run in the GUI (pulled from github and built a few minutes ago from source) > > ft:tokens('testdata'), > ft:search('testdata', 'r.ḥ', map {'wildcards': true()})/.., '----------' > ,collection('testdata')//*[text() contains text 'r.ḥ' using wildcards] > > yields > > <entry count="4">rwḥ</entry> > ---------- > > with collection('testdata') > > <_> > <gram xmlns="http://www.tei-c.org/ns/1.0" type="root" xml:lang="ar-aeb-x-vicav">rwḥ</gram> > <gram xmlns="http://www.tei-c.org/ns/1.0" type="root" xml:lang="ar-aeb-x-vicav">rwḥ</gram> > <gram xmlns="http://www.tei-c.org/ns/1.0" type="root" xml:lang="ar-aeb-x-vicav">rwḥ</gram> > <gram xmlns="http://www.tei-c.org/ns/1.0" type="root" xml:lang="ar-aeb-x-tunis-vicav">rwḥ</gram> > </_> > > But the gh1800() test changed like this: > > final String text = "999 aa 1111 rwḥ"; > [...] > query("ft:search('" + NAME + "', 'r.ḥ', " + options + ")", text); > > works. > > Am 06.02.2020 um 13:45 schrieb Christian Grün: > > I just tried to use the gh1800 test to replicate my problem and it does > not show there. It fails using the GUI. > > I need your help: What does not show there? What fails, what happens? > > > > > Am 06.02.2020 um 13:35 schrieb Christian Grün: > > Hi Omar, > > Yes, that seems to solve the problem partly. Using wildcards now yields the same result as no wildcards. > > Glad to hear. > > But if there is a complex unicode character in the search string, "." for one character looses its meaning. > … > Would you like a PR for the test gh1800 using complex unicode characters? > > A little test case would be helpful indeed. It seems to be a different issue: > > • The first expression is evaluated without the full-text expression. > The reason is that the full-text index algorithms are limited to basic > regular expressions; not all of them can be answered by an index (and > 'r{1,1}' is currently not detected as being identical to `r.`). If I > remember correctly, the index will not be accessed either if a pattern > starts with `.*` (this pattern would lead to a full index scan). > > • The second expression is rewritten for index access. I tried to > build a little command script (test.bxs), but it doesn’t seem to > reflect the case you encountered: > > set ftindex true > create db test <xml>rwḥ</xml> > xquery /*[text() contains text 'r.{1,1}ḥ' using wildcards] > xquery /*[text() contains text 'r.ḥ' using wildcards] > close > > Could you extend this example script a little, such that it > demonstrates what goes wrong? > > Thanks in advance, > Christian

On Thu, Feb 6, 2020 at 3:16 PM Christian Grün <christian.gruen at gmail.com> wrote: > > Great! Confirmed and registered [1]. > > [1] https://github.com/BaseXdb/basex/issues/1802 > > > > On Thu, Feb 6, 2020 at 3:03 PM Omar Siam <Omar.Siam at oeaw.ac.at> wrote: > > > > I think I've got it: for reasons I am not 100% sure about right now a usually select diacritics: true() > > > > So the following does not work as expected: > > > > <commands> > > <xquery><![CDATA[ > > let $xml := <_> > > <gram xmlns="http://www.tei-c.org/ns/1.0" type="root" xml:lang="ar-aeb-x-vicav">rwḥ</gram> > > <gram xmlns="http://www.tei-c.org/ns/1.0" type="root" xml:lang="ar-aeb-x-vicav">rwḥ</gram> > > <gram xmlns="http://www.tei-c.org/ns/1.0" type="root" xml:lang="ar-aeb-x-vicav">rwḥ</gram> > > <gram xmlns="http://www.tei-c.org/ns/1.0" type="root" xml:lang="ar-aeb-x-tunis-vicav">rwḥ</gram> > > </_> > > return db:create('testdata', $xml, 'testdata.xml', map { 'ftindex': true(), 'diacritics': true() }) > > ]]></xquery> > > <xquery> > > db:open('testdata')//*[text() contains text 'r.{1,1}ḥ' using wildcards using diacritics sensitive] > > </xquery> > > <xquery> > > db:open('testdata')//*[text() contains text 'r.ḥ' using wildcards using diacritics sensitive] > > </xquery> > > </commands> > > > > Am 06.02.2020 um 14:40 schrieb Christian Grün: > > > > I can try. But what you tell me and what I saw with the unit test function I guess the GUI does something that can't be observed when using scripts. > > > > Maybe some options are still active in your GUI that are not reflected > > in the script? > > Does the script work if you run it on command line? > > > > Unfortunately I get new problems with 9.3.2 Beta: > > > > …most probably related to recent typecheck optimizations that we > > recently introduced [1]; sorry for that. If you need a very short-term > > alternative, you can use string() instead of xs:string(). > > > > [1] https://github.com/BaseXdb/basex/issues/1801 > > > > > > > > declare function local:maptest() as map(xs:string, xs:string) { > > let $xml := <_ test="test"/> > > return map{xs:string($xml/@test): xs:string('test')} > > }; > > > > local:maptest() > > > > stopped working. > > > > [XPTY0004] Cannot convert map(*) to map(xs:string, xs:string): map { "test": "test" }. > > > > And I get more new type conversion related errors that were not reported before. This is temporary I hope? > > > > Best regards > > > > Omar > > > > Am 06.02.2020 um 14:11 schrieb Christian Grün: > > > > Looks we are getting closer. I have attached a little command script; > > do you think you can adapt it such that it captures the potential bug? > > > > > > On Thu, Feb 6, 2020 at 1:51 PM Omar Siam <Omar.Siam at oeaw.ac.at> wrote: > > > > The following XQuery run in the GUI (pulled from github and built a few minutes ago from source) > > > > ft:tokens('testdata'), > > ft:search('testdata', 'r.ḥ', map {'wildcards': true()})/.., '----------' > > ,collection('testdata')//*[text() contains text 'r.ḥ' using wildcards] > > > > yields > > > > <entry count="4">rwḥ</entry> > > ---------- > > > > with collection('testdata') > > > > <_> > > <gram xmlns="http://www.tei-c.org/ns/1.0" type="root" xml:lang="ar-aeb-x-vicav">rwḥ</gram> > > <gram xmlns="http://www.tei-c.org/ns/1.0" type="root" xml:lang="ar-aeb-x-vicav">rwḥ</gram> > > <gram xmlns="http://www.tei-c.org/ns/1.0" type="root" xml:lang="ar-aeb-x-vicav">rwḥ</gram> > > <gram xmlns="http://www.tei-c.org/ns/1.0" type="root" xml:lang="ar-aeb-x-tunis-vicav">rwḥ</gram> > > </_> > > > > But the gh1800() test changed like this: > > > > final String text = "999 aa 1111 rwḥ"; > > [...] > > query("ft:search('" + NAME + "', 'r.ḥ', " + options + ")", text); > > > > works. > > > > Am 06.02.2020 um 13:45 schrieb Christian Grün: > > > > I just tried to use the gh1800 test to replicate my problem and it does > > not show there. It fails using the GUI. > > > > I need your help: What does not show there? What fails, what happens? > > > > > > > > > > Am 06.02.2020 um 13:35 schrieb Christian Grün: > > > > Hi Omar, > > > > Yes, that seems to solve the problem partly. Using wildcards now yields the same result as no wildcards. > > > > Glad to hear. > > > > But if there is a complex unicode character in the search string, "." for one character looses its meaning. > > … > > Would you like a PR for the test gh1800 using complex unicode characters? > > > > A little test case would be helpful indeed. It seems to be a different issue: > > > > • The first expression is evaluated without the full-text expression. > > The reason is that the full-text index algorithms are limited to basic > > regular expressions; not all of them can be answered by an index (and > > 'r{1,1}' is currently not detected as being identical to `r.`). If I > > remember correctly, the index will not be accessed either if a pattern > > starts with `.*` (this pattern would lead to a full index scan). > > > > • The second expression is rewritten for index access. I tried to > > build a little command script (test.bxs), but it doesn’t seem to > > reflect the case you encountered: > > > > set ftindex true > > create db test <xml>rwḥ</xml> > > xquery /*[text() contains text 'r.{1,1}ḥ' using wildcards] > > xquery /*[text() contains text 'r.ḥ' using wildcards] > > close > > > > Could you extend this example script a little, such that it > > demonstrates what goes wrong? > > > > Thanks in advance, > > Christian

Am 09.02.2020 um 11:20 schrieb Christian Grün: > Hi Omar, > > Full-text queries with wildcards should now work with diacritical > strings as well, both with the index and with sequential requests. > > Furthermore, queries with more than 1 dot in the wildcard pattern are > now evaluated by the index if one is available. I’m still not sure if > this is a good decision, as query strings with wildcards might lead to > an extensive number of atomic index requests, and might get slower > than full XML scans. If you observe that your wildcard queries are > processed faster if you disable the full-text index, feel free to > report this back to us. > > A new snapshot is available. Have fun, > Christian > > PS: The xs:string cast bug is fixed as well. > > > > On Thu, Feb 6, 2020 at 3:16 PM Christian Grün <christian.gruen at gmail.com> wrote: >> Great! Confirmed and registered [1]. >> >> [1] https://github.com/BaseXdb/basex/issues/1802 >> >> >> >> On Thu, Feb 6, 2020 at 3:03 PM Omar Siam <Omar.Siam at oeaw.ac.at> wrote: >>> I think I've got it: for reasons I am not 100% sure about right now a usually select diacritics: true() >>> >>> So the following does not work as expected: >>> >>> <commands> >>> <xquery><![CDATA[ >>> let $xml := <_> >>> <gram xmlns="http://www.tei-c.org/ns/1.0" type="root" xml:lang="ar-aeb-x-vicav">rwḥ</gram> >>> <gram xmlns="http://www.tei-c.org/ns/1.0" type="root" xml:lang="ar-aeb-x-vicav">rwḥ</gram> >>> <gram xmlns="http://www.tei-c.org/ns/1.0" type="root" xml:lang="ar-aeb-x-vicav">rwḥ</gram> >>> <gram xmlns="http://www.tei-c.org/ns/1.0" type="root" xml:lang="ar-aeb-x-tunis-vicav">rwḥ</gram> >>> </_> >>> return db:create('testdata', $xml, 'testdata.xml', map { 'ftindex': true(), 'diacritics': true() }) >>> ]]></xquery> >>> <xquery> >>> db:open('testdata')//*[text() contains text 'r.{1,1}ḥ' using wildcards using diacritics sensitive] >>> </xquery> >>> <xquery> >>> db:open('testdata')//*[text() contains text 'r.ḥ' using wildcards using diacritics sensitive] >>> </xquery> >>> </commands> >>> >>> Am 06.02.2020 um 14:40 schrieb Christian Grün: >>> >>> I can try. But what you tell me and what I saw with the unit test function I guess the GUI does something that can't be observed when using scripts. >>> >>> Maybe some options are still active in your GUI that are not reflected >>> in the script? >>> Does the script work if you run it on command line? >>> >>> Unfortunately I get new problems with 9.3.2 Beta: >>> >>> …most probably related to recent typecheck optimizations that we >>> recently introduced [1]; sorry for that. If you need a very short-term >>> alternative, you can use string() instead of xs:string(). >>> >>> [1] https://github.com/BaseXdb/basex/issues/1801 >>> >>> >>> >>> declare function local:maptest() as map(xs:string, xs:string) { >>> let $xml := <_ test="test"/> >>> return map{xs:string($xml/@test): xs:string('test')} >>> }; >>> >>> local:maptest() >>> >>> stopped working. >>> >>> [XPTY0004] Cannot convert map(*) to map(xs:string, xs:string): map { "test": "test" }. >>> >>> And I get more new type conversion related errors that were not reported before. This is temporary I hope? >>> >>> Best regards >>> >>> Omar >>> >>> Am 06.02.2020 um 14:11 schrieb Christian Grün: >>> >>> Looks we are getting closer. I have attached a little command script; >>> do you think you can adapt it such that it captures the potential bug? >>> >>> >>> On Thu, Feb 6, 2020 at 1:51 PM Omar Siam <Omar.Siam at oeaw.ac.at> wrote: >>> >>> The following XQuery run in the GUI (pulled from github and built a few minutes ago from source) >>> >>> ft:tokens('testdata'), >>> ft:search('testdata', 'r.ḥ', map {'wildcards': true()})/.., '----------' >>> ,collection('testdata')//*[text() contains text 'r.ḥ' using wildcards] >>> >>> yields >>> >>> <entry count="4">rwḥ</entry> >>> ---------- >>> >>> with collection('testdata') >>> >>> <_> >>> <gram xmlns="http://www.tei-c.org/ns/1.0" type="root" xml:lang="ar-aeb-x-vicav">rwḥ</gram> >>> <gram xmlns="http://www.tei-c.org/ns/1.0" type="root" xml:lang="ar-aeb-x-vicav">rwḥ</gram> >>> <gram xmlns="http://www.tei-c.org/ns/1.0" type="root" xml:lang="ar-aeb-x-vicav">rwḥ</gram> >>> <gram xmlns="http://www.tei-c.org/ns/1.0" type="root" xml:lang="ar-aeb-x-tunis-vicav">rwḥ</gram> >>> </_> >>> >>> But the gh1800() test changed like this: >>> >>> final String text = "999 aa 1111 rwḥ"; >>> [...] >>> query("ft:search('" + NAME + "', 'r.ḥ', " + options + ")", text); >>> >>> works. >>> >>> Am 06.02.2020 um 13:45 schrieb Christian Grün: >>> >>> I just tried to use the gh1800 test to replicate my problem and it does >>> not show there. It fails using the GUI. >>> >>> I need your help: What does not show there? What fails, what happens? >>> >>> >>> >>> >>> Am 06.02.2020 um 13:35 schrieb Christian Grün: >>> >>> Hi Omar, >>> >>> Yes, that seems to solve the problem partly. Using wildcards now yields the same result as no wildcards. >>> >>> Glad to hear. >>> >>> But if there is a complex unicode character in the search string, "." for one character looses its meaning. >>> … >>> Would you like a PR for the test gh1800 using complex unicode characters? >>> >>> A little test case would be helpful indeed. It seems to be a different issue: >>> >>> • The first expression is evaluated without the full-text expression. >>> The reason is that the full-text index algorithms are limited to basic >>> regular expressions; not all of them can be answered by an index (and >>> 'r{1,1}' is currently not detected as being identical to `r.`). If I >>> remember correctly, the index will not be accessed either if a pattern >>> starts with `.*` (this pattern would lead to a full index scan). >>> >>> • The second expression is rewritten for index access. I tried to >>> build a little command script (test.bxs), but it doesn’t seem to >>> reflect the case you encountered: >>> >>> set ftindex true >>> create db test <xml>rwḥ</xml> >>> xquery /*[text() contains text 'r.{1,1}ḥ' using wildcards] >>> xquery /*[text() contains text 'r.ḥ' using wildcards] >>> close >>> >>> Could you extend this example script a little, such that it >>> demonstrates what goes wrong? >>> >>> Thanks in advance, >>> Christian

On Tue, Feb 11, 2020 at 12:07 PM Omar Siam <Omar.Siam at oeaw.ac.at> wrote: > > Hi Christian, > > I deployed the latest beta and I am quite happy with the performance and > all my problems seem to be fixed. I did not test without index yet. > Maybe I have time to do this later this week with a bigger dataset. > > Best regards > > Omar Siam > > Am 09.02.2020 um 11:20 schrieb Christian Grün: > > Hi Omar, > > > > Full-text queries with wildcards should now work with diacritical > > strings as well, both with the index and with sequential requests. > > > > Furthermore, queries with more than 1 dot in the wildcard pattern are > > now evaluated by the index if one is available. I’m still not sure if > > this is a good decision, as query strings with wildcards might lead to > > an extensive number of atomic index requests, and might get slower > > than full XML scans. If you observe that your wildcard queries are > > processed faster if you disable the full-text index, feel free to > > report this back to us. > > > > A new snapshot is available. Have fun, > > Christian > > > > PS: The xs:string cast bug is fixed as well. > > > > > > > > On Thu, Feb 6, 2020 at 3:16 PM Christian Grün <christian.gruen at gmail.com> wrote: > >> Great! Confirmed and registered [1]. > >> > >> [1] https://github.com/BaseXdb/basex/issues/1802 > >> > >> > >> > >> On Thu, Feb 6, 2020 at 3:03 PM Omar Siam <Omar.Siam at oeaw.ac.at> wrote: > >>> I think I've got it: for reasons I am not 100% sure about right now a usually select diacritics: true() > >>> > >>> So the following does not work as expected: > >>> > >>> <commands> > >>> <xquery><![CDATA[ > >>> let $xml := <_> > >>> <gram xmlns="http://www.tei-c.org/ns/1.0" type="root" xml:lang="ar-aeb-x-vicav">rwḥ</gram> > >>> <gram xmlns="http://www.tei-c.org/ns/1.0" type="root" xml:lang="ar-aeb-x-vicav">rwḥ</gram> > >>> <gram xmlns="http://www.tei-c.org/ns/1.0" type="root" xml:lang="ar-aeb-x-vicav">rwḥ</gram> > >>> <gram xmlns="http://www.tei-c.org/ns/1.0" type="root" xml:lang="ar-aeb-x-tunis-vicav">rwḥ</gram> > >>> </_> > >>> return db:create('testdata', $xml, 'testdata.xml', map { 'ftindex': true(), 'diacritics': true() }) > >>> ]]></xquery> > >>> <xquery> > >>> db:open('testdata')//*[text() contains text 'r.{1,1}ḥ' using wildcards using diacritics sensitive] > >>> </xquery> > >>> <xquery> > >>> db:open('testdata')//*[text() contains text 'r.ḥ' using wildcards using diacritics sensitive] > >>> </xquery> > >>> </commands> > >>> > >>> Am 06.02.2020 um 14:40 schrieb Christian Grün: > >>> > >>> I can try. But what you tell me and what I saw with the unit test function I guess the GUI does something that can't be observed when using scripts. > >>> > >>> Maybe some options are still active in your GUI that are not reflected > >>> in the script? > >>> Does the script work if you run it on command line? > >>> > >>> Unfortunately I get new problems with 9.3.2 Beta: > >>> > >>> …most probably related to recent typecheck optimizations that we > >>> recently introduced [1]; sorry for that. If you need a very short-term > >>> alternative, you can use string() instead of xs:string(). > >>> > >>> [1] https://github.com/BaseXdb/basex/issues/1801 > >>> > >>> > >>> > >>> declare function local:maptest() as map(xs:string, xs:string) { > >>> let $xml := <_ test="test"/> > >>> return map{xs:string($xml/@test): xs:string('test')} > >>> }; > >>> > >>> local:maptest() > >>> > >>> stopped working. > >>> > >>> [XPTY0004] Cannot convert map(*) to map(xs:string, xs:string): map { "test": "test" }. > >>> > >>> And I get more new type conversion related errors that were not reported before. This is temporary I hope? > >>> > >>> Best regards > >>> > >>> Omar > >>> > >>> Am 06.02.2020 um 14:11 schrieb Christian Grün: > >>> > >>> Looks we are getting closer. I have attached a little command script; > >>> do you think you can adapt it such that it captures the potential bug? > >>> > >>> > >>> On Thu, Feb 6, 2020 at 1:51 PM Omar Siam <Omar.Siam at oeaw.ac.at> wrote: > >>> > >>> The following XQuery run in the GUI (pulled from github and built a few minutes ago from source) > >>> > >>> ft:tokens('testdata'), > >>> ft:search('testdata', 'r.ḥ', map {'wildcards': true()})/.., '----------' > >>> ,collection('testdata')//*[text() contains text 'r.ḥ' using wildcards] > >>> > >>> yields > >>> > >>> <entry count="4">rwḥ</entry> > >>> ---------- > >>> > >>> with collection('testdata') > >>> > >>> <_> > >>> <gram xmlns="http://www.tei-c.org/ns/1.0" type="root" xml:lang="ar-aeb-x-vicav">rwḥ</gram> > >>> <gram xmlns="http://www.tei-c.org/ns/1.0" type="root" xml:lang="ar-aeb-x-vicav">rwḥ</gram> > >>> <gram xmlns="http://www.tei-c.org/ns/1.0" type="root" xml:lang="ar-aeb-x-vicav">rwḥ</gram> > >>> <gram xmlns="http://www.tei-c.org/ns/1.0" type="root" xml:lang="ar-aeb-x-tunis-vicav">rwḥ</gram> > >>> </_> > >>> > >>> But the gh1800() test changed like this: > >>> > >>> final String text = "999 aa 1111 rwḥ"; > >>> [...] > >>> query("ft:search('" + NAME + "', 'r.ḥ', " + options + ")", text); > >>> > >>> works. > >>> > >>> Am 06.02.2020 um 13:45 schrieb Christian Grün: > >>> > >>> I just tried to use the gh1800 test to replicate my problem and it does > >>> not show there. It fails using the GUI. > >>> > >>> I need your help: What does not show there? What fails, what happens? > >>> > >>> > >>> > >>> > >>> Am 06.02.2020 um 13:35 schrieb Christian Grün: > >>> > >>> Hi Omar, > >>> > >>> Yes, that seems to solve the problem partly. Using wildcards now yields the same result as no wildcards. > >>> > >>> Glad to hear. > >>> > >>> But if there is a complex unicode character in the search string, "." for one character looses its meaning. > >>> … > >>> Would you like a PR for the test gh1800 using complex unicode characters? > >>> > >>> A little test case would be helpful indeed. It seems to be a different issue: > >>> > >>> • The first expression is evaluated without the full-text expression. > >>> The reason is that the full-text index algorithms are limited to basic > >>> regular expressions; not all of them can be answered by an index (and > >>> 'r{1,1}' is currently not detected as being identical to `r.`). If I > >>> remember correctly, the index will not be accessed either if a pattern > >>> starts with `.*` (this pattern would lead to a full index scan). > >>> > >>> • The second expression is rewritten for index access. I tried to > >>> build a little command script (test.bxs), but it doesn’t seem to > >>> reflect the case you encountered: > >>> > >>> set ftindex true > >>> create db test <xml>rwḥ</xml> > >>> xquery /*[text() contains text 'r.{1,1}ḥ' using wildcards] > >>> xquery /*[text() contains text 'r.ḥ' using wildcards] > >>> close > >>> > >>> Could you extend this example script a little, such that it > >>> demonstrates what goes wrong? > >>> > >>> Thanks in advance, > >>> Christian