repeatedly full-text marking the same text node
Friday, 8 May 2020 · 9 messages · last reply 12 May 2020
GS
Graydon Saunders8 May 2020 22:52 #
Hello --
Is there some way to iterate full-text matches to mark up in the node every found member of a sequence of phrases?
I have a use case where I have a long list of phrases which may appear in the content set; if they do appear in the content set, these should be marked.
The order of the list is significant; the longest phrases should be marked first.
The idea would be to iterate through the list, marking up the node with any matches.
However, so far as I can tell, full text only works directly on a database node. If I try to pass the node in a function, I get "No database node" errors from attempting full-text operations.
(Making a copy of the DB with a full text index, applying the changes, updating the indexes, and continuing seems possible but also inherently inefficient.)
Is there a straightforward way to do this that I'm missing?
Thanks!
Graydon
LQ
Liam R. E. Quin10 May 2020 11:36 #
Quoted text · Graydon Saunders · 6 lines
On Fri, 2020-05-08 at 14:52 -0400, Graydon Saunders wrote: > > > The idea would be to iterate through the list, marking up the node > with any > matches.
Can you instead use standoff markup? E.g. store positions of start and end as word counts, and then merge them later?
Quoted text · Graydon Saunders · 1 line
>
--
Liam Quin, https://www.delightfulcomputing.com/
Available for XML/Document/Information Architecture/XSLT/
XSL/XQuery/Web/Text Processing/A11Y training, work & consulting. Barefoot Web-slave, antique illustrations: http://www.fromoldbooks.org
G
Graydon10 May 2020 18:12 #
Quoted text · Graydon Saunders · 7 lines
On Sun, May 10, 2020 at 03:35:45AM -0400, Liam R. E. Quin scripsit: > On Fri, 2020-05-08 at 14:52 -0400, Graydon Saunders wrote: > > The idea would be to iterate through the list, marking up the node > > with any matches. > > Can you instead use standoff markup? E.g. store positions of start and > end as word counts, and then merge them later?
In principle, yes. But then I would have to be smart and extract the positions correctly somehow and then get all the positional arithmetic correct.
The attraction of the full-text index was a combination of speed and being able to let some other smarter person handle the "does the match still work if there's a line break? bunches of tabs?" issues.
I now think this just isn't a full-text use case; I was trying to think of a way to use something optimized for single-pass search to support recursion on the changed content and that loses all the attractive optimizations. Nothing says I can't use analyze-string and recursion.
Thanks!
-- Graydon
LC
Loren Cahlander10 May 2020 19:40 #
Take a look at exist-Stanford-nlp in my GitHub. Take a look at the code for the named entity recognition
https://github.com/lcahlander/exist-stanford-nlp/blob/master/src/main/xquery/ner-module.xqm
Loren Cahlander
Sent from my iPhone
Quoted text · Graydon · 8 lines
On May 10, 2020, at 10:13 AM, Graydon <graydonish at gmail.com> wrote: On Sun, May 10, 2020 at 03:35:45AM -0400, Liam R. E. Quin scripsit: >> On Fri, 2020-05-08 at 14:52 -0400, Graydon Saunders wrote: >> The idea would be to iterate through the list, marking up the node >> with any matches. > > Can you instead use standoff markup? E.g. store positions of start and > end as word counts, and then merge them later?
In principle, yes. But then I would have to be smart and extract the positions correctly somehow and then get all the positional arithmetic correct.
The attraction of the full-text index was a combination of speed and being able to let some other smarter person handle the "does the match still work if there's a line break? bunches of tabs?" issues.
I now think this just isn't a full-text use case; I was trying to think of a way to use something optimized for single-pass search to support recursion on the changed content and that loses all the attractive optimizations. Nothing says I can't use analyze-string and recursion.
Thanks!
-- Graydon
LQ
Quoted text · Graydon · 3 lines
On Sun, 2020-05-10 at 10:12 -0400, Graydon wrote: > > I now think this just isn't a full-text use case;
In the past i used a text retrival package i wrote to solve the problem of inserting links automatically, choosing the longest & avoiding overlaps.
I use some multi-threaded procedural code i wrote years ago in Perl to do it on e.g.
https://words.fromoldbooks.org/Chalmers-Biography/w/walsingham-sir-francis.html
Recently i was thinking about rewriting thism perhaps in XSLT and/or XQuery to try and keep the most "relevant" link rather than the longest, with a different UI. The Perl script takes maybe two minutes to run on approx. 200 MBytes of HTML (10,000 files). But i'd need a good definition of relevant.
I regret that my efforts to get more full text researchers interested in joining the XQuery full text work failed - but then i think one of them may have been Sergey Brin, and he had other interests :) - as markup-informed ranking of results ought to be really interesting. On the other hand maybe Full Text would have become even more complex :)
Liam
--
Liam Quin, https://www.delightfulcomputing.com/
Available for XML/Document/Information Architecture/XSLT/
XSL/XQuery/Web/Text Processing/A11Y training, work & consulting. Barefoot Web-slave, antique illustrations: http://www.fromoldbooks.org
CG
Hi Graydon,
Thanks for sharing your use case.
Quoted text · Graydon Saunders · 1 line
> However, so far as I can tell, full text only works directly on a database node. If I try to pass the node in a function, I get "No database node" errors from attempting full-text operations.
You can convert an XML node to the internal “database node”
representation by applying a dummy operation:
let $xml := <xml>hello world</xml> update {}
return ft:mark($xml[text() contains text 'hello'])Does this already help? See the Wiki articles [1,2] for some revised information.
I have already asked myself in the past if we shouldn’t include a function that expose internal result positions to the user?
Suggestions are welcome.
Best,
Christian
[1] https://docs.basex.org/wiki/Full-Text_Module#ft:mark
[2] https://docs.basex.org/wiki/Database_Module#Database_Nodes
CG
Quoted text · Graydon · 1 line
> The thing that I would most want to see is some way to capture multi-word matches using full-text search; "full phrase search", in effect. I can see that as the start and end of a range of internal result positions but will admit to wanting something less at risk of my arithmetic errors.
Providing access to the starts and ends may be difficult due to all the logical operators that can be used (ftor, ftand, ftnot, not in). A simple example:
let $xml := <_>a b c d</_> update {}
return ft:mark($xml[text() contains text 'b c' ftand 'c d'])We could possibly make the full data structures available that need to be internally generated. I fear people wouldn’t really work with it as they are fairly complex (a look into the specification may give you an impression of that [1]).
But thanks for your thoughts, I’ll let them grow.
[1] https://www.w3.org/TR/xpath-full-text-10/#FTOperatorsSemanticsSec
G
Quoted text · Christian Grün · 2 lines
On Mon, May 11, 2020 at 07:51:02PM +0200, Christian Grün scripsit: > Hi Graydon,
Hi Christian --
Quoted text · Christian Grün · 12 lines
> Thanks for sharing your use case. > > > However, so far as I can tell, full text only works directly on a database node. If I try to pass the node in a function, I get "No database node" errors from attempting full-text operations. > > You can convert an XML node to the internal “database node” > representation by applying a dummy operation: > > let $xml := <xml>hello world</xml> update {} > return ft:mark($xml[text() contains text 'hello']) > > Does this already help? See the Wiki articles [1,2] for some revised > information.
That helps and I will check, but not this week. (This part of the current deadline has been addressed by clubbing the problem with a rock, er, xsl:iterate, and that'll do for now. The general pattern is something I need to do a lot so I'll be coming back to this.)
Thank you very much for the docs update!
Quoted text · Christian Grün · 3 lines
> I have already asked myself in the past if we shouldn’t include a > function that expose internal result positions to the user? > Suggestions are welcome.
The thing that I would most want to see is some way to capture multi-word matches using full-text search; "full phrase search", in effect. I can see that as the start and end of a range of internal result positions but will admit to wanting something less at risk of my arithmetic errors.
-- Graydon
LQ
Quoted text · Christian Grün · 3 lines
On Mon, 2020-05-11 at 22:29 +0200, Christian Grün wrote: > Providing access to the starts and ends may be difficult due to all > the logical operators that can be used
A way to go from ($input, $phrases) to a $input autmented with db:milestone elements each containing starts="0 7 23" ends="2 6 18" attributes (where the numbers are positional in the sequene of phrases) might be good. Or the mileston element could iclude the phrase,
I saw his <db:milestone>
<db:start ref="3">naked hooves</db:start>
<db:start ref="6">unshod</db:start>
</db:milestonr>bare<db:milestone>
<db:end ref="6" /></db:mileston> feet....as two problems are (1) overlapping results, and (2) query expansion using a thesaurus and/or stemming.
Liam
Quoted text · Christian Grün · 17 lines
> (ftor, ftand, ftnot, not in). A > simple example: > > let $xml := <_>a b c d</_> update {} > return ft:mark($xml[text() contains text 'b c' ftand 'c d']) > > We could possibly make the full data structures available that need > to > be internally generated. I fear people wouldn’t really work with it > as > they are fairly complex (a look into the specification may give you > an > impression of that [1]). > > But thanks for your thoughts, I’ll let them grow. > > [1] https://www.w3.org/TR/xpath-full-text-10/#FTOperatorsSemanticsSec
--
Liam Quin, https://www.delightfulcomputing.com/
Available for XML/Document/Information Architecture/XSLT/
XSL/XQuery/Web/Text Processing/A11Y training, work & consulting. Barefoot Web-slave, antique illustrations: http://www.fromoldbooks.org
Written for BaseX 9.3. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery