Troubles with ft:mark

> The first result of this query is the entry for Austria. I would expect > both of the instances of the word 'German' in that entry to be surrounded > by <mark> tags. However only the first instance is. >

On Thu, 29 Feb 2024, 1:02 am Christian Grün, <christian.gruen at gmail.com> wrote: > Hi Jack, > > Thanks for your observation. > > >> The first result of this query is the entry for Austria. I would expect >> both of the instances of the word 'German' in that entry to be surrounded >> by <mark> tags. However only the first instance is. >> > > I couldn’t reproduce this yet. Here’s a command script that returns 14 > <mark>German</mark> elements: > > <commands> > <set option='ftindex'>true</set> > <create-db name='factbook'>https://files.basex.org/xml/factbook.xml > </create-db> > <xquery><![CDATA[ > let $groups := ('ethnicgroups', 'languages') > let $database := 'factbook' > let $query := 'German' > > let $search := ft:search($database, $query)/parent::* > [name() = $groups]/ancestor::country > let $marked := ft:mark( > $search[.//*[name() = $groups][text() contains text { $query }]] > ) > return $marked//*[text() = 'German'] > ]]></xquery> > </commands> > > Could you check if you get the same result? > > Thanks in advance > Christian > >

> When you say you can't reproduce it, do you mean you get 14 results from

On Thu, Feb 29, 2024 at 12:13 AM Jack Steyn <steynjack at gmail.com> wrote: > Hi Christian, > > When I run your script, I do get 14 elements. > > When I run the following script I just get 12. > > <commands> > <set option='ftindex'>true</set> > <create-db name='factbook'>https://files.basex.org/xml/factbook.xml > </create-db> > <xquery><![CDATA[ > declare function local:search( > $database as xs:string, > $query as xs:string > ) { > let $country-search := ft:search($database, $query)/ancestor::country > let $city-search := ft:search($database, > $query)/ancestor::city/ancestor::country > let $other-search := ft:search($database, $query)/parent::*[name() = > ('ethnicgroups', 'languages')]/ancestor::country > let $country-mark := $country-search[.//name[text() contains text { > $query }]] => ft:mark() > let $city-mark := $city-search[.//city[text() contains text { $query > }]] => ft:mark() > let $other-mark := $other-search[.//*[name() = ('ethnicgroups', > 'languages')][text() contains text { $query }]] => ft:mark() > return ( > $country-mark, > $city-mark, > $other-mark > ) > }; > > local:search('factbook', 'German')//mark > ]]></xquery> > </commands> > > When you say you can't reproduce it, do you mean you get 14 results from > running this script? > > Cheers, > > Jack > > On Thu, 29 Feb 2024, 1:02 am Christian Grün, <christian.gruen at gmail.com> > wrote: > >> Hi Jack, >> >> Thanks for your observation. >> >> >>> The first result of this query is the entry for Austria. I would expect >>> both of the instances of the word 'German' in that entry to be surrounded >>> by <mark> tags. However only the first instance is. >>> >> >> I couldn’t reproduce this yet. Here’s a command script that returns 14 >> <mark>German</mark> elements: >> >> <commands> >> <set option='ftindex'>true</set> >> <create-db name='factbook'>https://files.basex.org/xml/factbook.xml >> </create-db> >> <xquery><![CDATA[ >> let $groups := ('ethnicgroups', 'languages') >> let $database := 'factbook' >> let $query := 'German' >> >> let $search := ft:search($database, $query)/parent::* >> [name() = $groups]/ancestor::country >> let $marked := ft:mark( >> $search[.//*[name() = $groups][text() contains text { $query }]] >> ) >> return $marked//*[text() = 'German'] >> ]]></xquery> >> </commands> >> >> Could you check if you get the same result? >> >> Thanks in advance >> Christian >> >>

On Fri, 1 Mar 2024, 8:12 pm Christian Grün, <christian.gruen at gmail.com> wrote: > Hi Jack, > > > When you say you can't reproduce it, do you mean you get 14 results from > running this script? > > Yes, that’s what I meant. > > The upcoming information will be very technical and specific. You are > welcome to focus on the examples. > > Your updated example was helpful, and I noticed it’s a bunch of issues > that lead to the unexpected results. The core challenge is that ft:mark and > ft:extract only yield expected results if the internally collected > full-text metadata is not lost at some stage during the internal processing > – which can happen at many places hidden to the writer of the query. > > In your specific example, the full-text information gets lost because the local:search > function is too complex to be inlined by the compiler (which enables > further optimizations that eventually allow metadata propagation). You can > tackle this by forcing the compiler to inline your function: > > declare %basex:inline function local:search(...) > > Using '(ethnicgroups, languages)' instead of 'name() = (...)' is another > practical advice; it helps the optimizer to detect at compile time that > metadata will be available at runtime. Another solution is to use > 'local-name()' instead of 'name()' (local-name does not rely on namespace > that may possibly occur in a database, which also affects the way how > full-text queries are evaluated). > > Here’s a variant that should work: > > declare function local:search( > $database as xs:string, > $query as xs:string > ) { > let $country := ft:search($database, $query)/ancestor::country > let $search := function($node) { $node/text() contains text { $query } } > return ( > ft:mark($country[.//name[$search(.)]]), > ft:mark($country[.//city[$search(.)]]), > ft:mark($country[.//(ethnicgroups, languages)[$search(.)]]) > ) > }; > local:search('factbook', 'German') > > …or… > > let $search := function($nodes) { $nodes[text() contains text { $query > }] } > return (ft:mark($country[$search(.//name)]), ... > > From today’s perspective, we would certainly design ft:mark and ft:extract > in a way that the results are always correct. The consequences, however, > would be a much more restricted syntax. > > Hope this helps, > Christian > > > On Thu, Feb 29, 2024 at 12:13 AM Jack Steyn <steynjack at gmail.com> wrote: > >> Hi Christian, >> >> When I run your script, I do get 14 elements. >> >> When I run the following script I just get 12. >> >> <commands> >> <set option='ftindex'>true</set> >> <create-db name='factbook'>https://files.basex.org/xml/factbook.xml >> </create-db> >> <xquery><![CDATA[ >> declare function local:search( >> $database as xs:string, >> $query as xs:string >> ) { >> let $country-search := ft:search($database, $query)/ancestor::country >> let $city-search := ft:search($database, >> $query)/ancestor::city/ancestor::country >> let $other-search := ft:search($database, $query)/parent::*[name() = >> ('ethnicgroups', 'languages')]/ancestor::country >> let $country-mark := $country-search[.//name[text() contains text { >> $query }]] => ft:mark() >> let $city-mark := $city-search[.//city[text() contains text { $query >> }]] => ft:mark() >> let $other-mark := $other-search[.//*[name() = ('ethnicgroups', >> 'languages')][text() contains text { $query }]] => ft:mark() >> return ( >> $country-mark, >> $city-mark, >> $other-mark >> ) >> }; >> >> local:search('factbook', 'German')//mark >> ]]></xquery> >> </commands> >> >> When you say you can't reproduce it, do you mean you get 14 results from >> running this script? >> >> Cheers, >> >> Jack >> >> On Thu, 29 Feb 2024, 1:02 am Christian Grün, <christian.gruen at gmail.com> >> wrote: >> >>> Hi Jack, >>> >>> Thanks for your observation. >>> >>> >>>> The first result of this query is the entry for Austria. I would expect >>>> both of the instances of the word 'German' in that entry to be surrounded >>>> by <mark> tags. However only the first instance is. >>>> >>> >>> I couldn’t reproduce this yet. Here’s a command script that returns 14 >>> <mark>German</mark> elements: >>> >>> <commands> >>> <set option='ftindex'>true</set> >>> <create-db name='factbook'>https://files.basex.org/xml/factbook.xml >>> </create-db> >>> <xquery><![CDATA[ >>> let $groups := ('ethnicgroups', 'languages') >>> let $database := 'factbook' >>> let $query := 'German' >>> >>> let $search := ft:search($database, $query)/parent::* >>> [name() = $groups]/ancestor::country >>> let $marked := ft:mark( >>> $search[.//*[name() = $groups][text() contains text { $query }]] >>> ) >>> return $marked//*[text() = 'German'] >>> ]]></xquery> >>> </commands> >>> >>> Could you check if you get the same result? >>> >>> Thanks in advance >>> Christian >>> >>>

On Wed, Mar 6, 2024 at 5:25 AM Jack Steyn <steynjack at gmail.com> wrote: > Hi Christian, > > Thank you very much for your explanation and variant example. > > In my use case, the local:search function is itself being called (as a > named function reference) from within another function that is the endpoint > of a RESTXQ API. This containing function handles a number of things e.g. > pagination, deduplication, transformation of XML into HTML. > > Even when I rewrite this local:search to match your variant example, > incorrect results are still returned. But when I then add %rest:GET > annotations to turn the local:search function into its own endpoint, the > correct results are returned only when I use that endpoint directly. > > Thus I assume the containing function makes things again too complicated > for metadata to be propagated. > > Does that sound plausible to you? And can you suggest any simple ways > around it? I'm afraid applying %basex:inline hasn't helped. > > Very best, > > Jack > > On Fri, 1 Mar 2024, 8:12 pm Christian Grün, <christian.gruen at gmail.com> > wrote: > >> Hi Jack, >> >> > When you say you can't reproduce it, do you mean you get 14 results >> from running this script? >> >> Yes, that’s what I meant. >> >> The upcoming information will be very technical and specific. You are >> welcome to focus on the examples. >> >> Your updated example was helpful, and I noticed it’s a bunch of issues >> that lead to the unexpected results. The core challenge is that ft:mark and >> ft:extract only yield expected results if the internally collected >> full-text metadata is not lost at some stage during the internal processing >> – which can happen at many places hidden to the writer of the query. >> >> In your specific example, the full-text information gets lost because the local:search >> function is too complex to be inlined by the compiler (which enables >> further optimizations that eventually allow metadata propagation). You can >> tackle this by forcing the compiler to inline your function: >> >> declare %basex:inline function local:search(...) >> >> Using '(ethnicgroups, languages)' instead of 'name() = (...)' is another >> practical advice; it helps the optimizer to detect at compile time that >> metadata will be available at runtime. Another solution is to use >> 'local-name()' instead of 'name()' (local-name does not rely on namespace >> that may possibly occur in a database, which also affects the way how >> full-text queries are evaluated). >> >> Here’s a variant that should work: >> >> declare function local:search( >> $database as xs:string, >> $query as xs:string >> ) { >> let $country := ft:search($database, $query)/ancestor::country >> let $search := function($node) { $node/text() contains text { $query } } >> return ( >> ft:mark($country[.//name[$search(.)]]), >> ft:mark($country[.//city[$search(.)]]), >> ft:mark($country[.//(ethnicgroups, languages)[$search(.)]]) >> ) >> }; >> local:search('factbook', 'German') >> >> …or… >> >> let $search := function($nodes) { $nodes[text() contains text { $query >> }] } >> return (ft:mark($country[$search(.//name)]), ... >> >> From today’s perspective, we would certainly design ft:mark and >> ft:extract in a way that the results are always correct. The consequences, >> however, would be a much more restricted syntax. >> >> Hope this helps, >> Christian >> >> >> On Thu, Feb 29, 2024 at 12:13 AM Jack Steyn <steynjack at gmail.com> wrote: >> >>> Hi Christian, >>> >>> When I run your script, I do get 14 elements. >>> >>> When I run the following script I just get 12. >>> >>> <commands> >>> <set option='ftindex'>true</set> >>> <create-db name='factbook'>https://files.basex.org/xml/factbook.xml >>> </create-db> >>> <xquery><![CDATA[ >>> declare function local:search( >>> $database as xs:string, >>> $query as xs:string >>> ) { >>> let $country-search := ft:search($database, $query)/ancestor::country >>> let $city-search := ft:search($database, >>> $query)/ancestor::city/ancestor::country >>> let $other-search := ft:search($database, $query)/parent::*[name() = >>> ('ethnicgroups', 'languages')]/ancestor::country >>> let $country-mark := $country-search[.//name[text() contains text { >>> $query }]] => ft:mark() >>> let $city-mark := $city-search[.//city[text() contains text { $query >>> }]] => ft:mark() >>> let $other-mark := $other-search[.//*[name() = ('ethnicgroups', >>> 'languages')][text() contains text { $query }]] => ft:mark() >>> return ( >>> $country-mark, >>> $city-mark, >>> $other-mark >>> ) >>> }; >>> >>> local:search('factbook', 'German')//mark >>> ]]></xquery> >>> </commands> >>> >>> When you say you can't reproduce it, do you mean you get 14 results from >>> running this script? >>> >>> Cheers, >>> >>> Jack >>> >>> On Thu, 29 Feb 2024, 1:02 am Christian Grün, <christian.gruen at gmail.com> >>> wrote: >>> >>>> Hi Jack, >>>> >>>> Thanks for your observation. >>>> >>>> >>>>> The first result of this query is the entry for Austria. I would >>>>> expect both of the instances of the word 'German' in that entry to be >>>>> surrounded by <mark> tags. However only the first instance is. >>>>> >>>> >>>> I couldn’t reproduce this yet. Here’s a command script that returns 14 >>>> <mark>German</mark> elements: >>>> >>>> <commands> >>>> <set option='ftindex'>true</set> >>>> <create-db name='factbook'>https://files.basex.org/xml/factbook.xml >>>> </create-db> >>>> <xquery><![CDATA[ >>>> let $groups := ('ethnicgroups', 'languages') >>>> let $database := 'factbook' >>>> let $query := 'German' >>>> >>>> let $search := ft:search($database, $query)/parent::* >>>> [name() = $groups]/ancestor::country >>>> let $marked := ft:mark( >>>> $search[.//*[name() = $groups][text() contains text { $query }]] >>>> ) >>>> return $marked//*[text() = 'German'] >>>> ]]></xquery> >>>> </commands> >>>> >>>> Could you check if you get the same result? >>>> >>>> Thanks in advance >>>> Christian >>>> >>>>