Main Page » Mailing List » September 2012

Full-Text Query Including Attributes

Tuesday, 4 September 2012 · 5 messages · last reply 7 Sep 2012
赤史
赤瀬川 史朗4 Sep 2012 13:31 #
I've just installed BaseX and started using it. I just wonder if the following query is possible with BaseX. I have a corpus in an XML format, where <s> means 'sentence', <w> means 'word'.
<s n="6"><w c5="PNP" hw="she" pos="PRON">She </w><w c5="VVD" hw="say" pos="VERB">said
</w><w c5="PNP" hw="she" pos="PRON">she </w><w c5="VVD" hw="go" pos="VERB">went </w>
<w c5="TO0" hw="to" pos="PREP">to </w><w c5="VVI" hw="buy" pos="VERB">buy </w>
<w c5="PNI" hw="something" pos="PRON">something </w><unclear/>
<w c5="PNX" hw="herself" pos="PRON">herself</w><c c5="PUN">, </c>...</s>
If you create a BaseX database with an option of full-text, you can extract sentences in which the word A and the word B" appear within a designated number of words. For example, the following query will extract the sentence above: //s[. contains text 'buy' ftand 'herself' window 5 words] So this is my question: Is there a way to extract all the sentences in which a word of a particular part of speech (for example, a verb) and another word appear within a designated number of words, like any verb and "herself" appear within a window of 5 words. Thank you in advance. Best, Sam, A.
CG
Christian Grün4 Sep 2012 14:21 #
Hi Sam, sure; I have attached a possible solution below. It's not as compact as the XPath version, though. Hope this helps, Christian _______________________________________
let $word := 'herself'
let $pos := 'VERB'
let $dist := 3
(: read from file/database
let $doc := doc("your-document.xml") :)
let $doc := document {
<s n="6">
  <w c5="PNP" hw="she" pos="PRON">She</w>
  <w c5="VVD" hw="say" pos="VERB">said</w>
  <w c5="PNP" hw="she" pos="PRON">she</w>
  <w c5="VVD" hw="go" pos="VERB">went</w>
  <w c5="TO0" hw="to" pos="PREP">to</w>
  <w c5="VVI" hw="buy" pos="VERB">buy</w>
  <w c5="PNI" hw="something" pos="PRON">something</w>
  <unclear/>
  <w c5="PNX" hw="herself" pos="PRON">herself</w>
  <c c5="PUN">,</c>...
</s> }

for $s    in $doc//s
let $hits-word := $s/w[. contains text { $word }]
let $hits-pos  := $s/w[@pos = $pos]
let $near-pos := (
  $hits-pos/following-sibling::node()[position() <= $dist] union
  $hits-pos/preceding-sibling::node()[position() <= $dist]
)
where ($near-pos intersect $hits-word)
return $s
Quoted text · 赤瀬川 史朗 · 34 lines
On Tue, Sep 4, 2012 at 1:31 PM, 赤瀬川 史朗 <akasan123 at yahoo.co.jp> wrote: > I've just installed BaseX and started using it. I just wonder if the > > following query is possible with BaseX. > > I have a corpus in an XML format, where <s> means 'sentence', <w> means 'word'. > > > <s n="6"><w c5="PNP" hw="she" pos="PRON">She </w><w c5="VVD" hw="say" pos="VERB">said > </w><w c5="PNP" hw="she" pos="PRON">she </w><w c5="VVD" hw="go" pos="VERB">went </w> > <w c5="TO0" hw="to" pos="PREP">to </w><w c5="VVI" hw="buy" pos="VERB">buy </w> > <w c5="PNI" hw="something" pos="PRON">something </w><unclear/> > <w c5="PNX" hw="herself" pos="PRON">herself</w><c c5="PUN">, </c>...</s> > > If you create a BaseX database with an option of full-text, you can > extract sentences in which the word A and the word B" appear > within a designated number of words. For example, the following query > will extract the sentence above: > //s[. contains text 'buy' ftand 'herself' window 5 words] > > So this is my question: Is there a way to extract all the sentences in > which a word of a particular part of speech (for example, a verb) > and another word appear within a designated number of words, > like any verb and "herself" appear within a window of 5 words. > > Thank you in advance. > > Best, > Sam, A. > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
赤史
赤瀬川 史朗6 Sep 2012 06:00 #
Hi, Christian, Many thanks for your reply and code. I'll try your code and learn more about BaseX. Does your code use a full-text index? If not, it is expected to take a lot of time to query a large corpus with more than one million sentences. Is it possible to use full-text index for search attributes? That would be much more suitable for searching a pos-tagged corpus. Best, Sam
CG
Christian Grün6 Sep 2012 08:11 #
Hi Sam,
Quoted text · 赤瀬川 史朗 · 1 line
> Does your code use a full-text index?
I guess that the index should be applied by the "contains text" expression. Feel free to open the InfoView in the GUI to see what's happening internally.
Quoted text · 赤瀬川 史朗 · 1 line
> Is it possible to use full-text index for search attributes?
Others have asked for this before, so chances are increasing that we will eventually add this in a future version. For now, if you should experience bad performance, a popular BaseX approach is to create new databases from all relevant data and create a full-text index for those instances. Hope this helps, Christian
赤史
赤瀬川 史朗7 Sep 2012 08:41 #
Hi, Christian,
Quoted text · Christian Grün · 3 lines
> Others have asked for this before, so chances are > increasing that we will eventually add this in a > future version.
That's great news for me! I'm looking forward to it. Best, Sam

Written for BaseX 7.3. BaseX may have changed since: please check the current documentation.

« API.Command Protocol - more and next functionscan restxq add a document to the db and return an http response? »
⚡Generated with XQuery