performance question

> To make inspection of results easier, I added ft:mark. A collection with > only a dozen of texts of about 71 MB with full text index for German, > optimized, etc. works quite well. However, the example query needs more than > 9s, which is rather slow.

> //(p|l) [text() contains text "Korb geben" using stemming using language > "de"]

>> To make inspection of results easier, I added ft:mark. A collection with >> only a dozen of texts of about 71 MB with full text index for German, >> optimized, etc. works quite well. However, the example query needs more than >> 9s, which is rather slow. > > First of all, it might be interesting to hear what the query compiler > does. Have you looked at the QueryInfo panel to check out if the > full-text index is applied? If yes, you should find something like.. > > Compiling: > - ... > - applying full-text index > - ... > > ..in the info panel.

> Another hint: to enable index optimizations, your query... > >> //(p|l) [text() contains text "Korb geben" using stemming using language >> "de"] > > ..may have to be rewritten as follows: > > //*[text() contains text "Korb geben"][self::p or self::l]

> I will try the Beta 7.1.

Am 13. Januar 2012 16:05 schrieb Cerstin Mahlow <cerstin.mahlow at unibas.ch>: > Dear Christian, > > thanks for your quick answer. > > Zitat von Christian Grün <christian.gruen at gmail.com>: > > To make inspection of results easier, I added ft:mark. A collection with >>> only a dozen of texts of about 71 MB with full text index for German, >>> optimized, etc. works quite well. However, the example query needs more >>> than >>> 9s, which is rather slow. >>> >> >> First of all, it might be interesting to hear what the query compiler >> does. Have you looked at the QueryInfo panel to check out if the >> full-text index is applied? If yes, you should find something like.. >> >> Compiling: >> - ... >> - applying full-text index >> - ... >> >> ..in the info panel. >> > > The interesting thing is: > > for my original query > > //(p|l) [text() contains text "Korb geben" using stemming using language > "de"] > > there is no information on compiling, only information on Timing, Result > (number of results) and Queryplan. > > If I change '(p|l)' to 'p', I get information on Compiling, but only: > > Compiling: > - optimizing descendant-or-self step(s) > Result: root()/descendant::{http://**www.tei-c.org/ns/1.0}p[text()<http://www.tei-c.org/ns/1.0%7Dp%5Btext%28%29>contains text "Korb geben"] > > Apparently, the index is not used. > > Another hint: to enable index optimizations, your query... >> >> //(p|l) [text() contains text "Korb geben" using stemming using language >>> "de"] >>> >> >> ..may have to be rewritten as follows: >> >> //*[text() contains text "Korb geben"][self::p or self::l] >> > > OK, this results in reducing total time by half, but I see only: > > Compiling: > - optimizing descendant-or-self step(s) > Result: root()/descendant::*[text() contains text "Korb geben"][self::{ > http://www.tei-**c.org/ns/1.0}p <http://www.tei-c.org/ns/1.0%7Dp> or > self::{http://www.tei-c.org/**ns/1.0}l <http://www.tei-c.org/ns/1.0%7Dl>] > > Also memory used is reduced a bit, so this definitely helps. However, if I > include 'using stemming using language "de"', total time is almost the same. > > I see no possibility to enforce using the index. I use BaseX 7.0.2, maybe > this is a bug? I will try the Beta 7.1. > > Best regards > > Cerstin > > -- > Dr. phil. Cerstin Mahlow > > Universität Basel > Deutsches Seminar > Nadelberg 4 > 4051 Basel > Schweiz > > Tel: +41 61 267 07 65 > Fax: +41 61 267 34 40 > Mail: cerstin.mahlow at unibas.ch > Web: http://www.oldphras.net > > ------------------------------**------------------------------**---- > This message was sent using IMP, the Internet Messaging Program. > > > ______________________________**_________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-**konstanz.de <BaseX-Talk at mailman.uni-konstanz.de> > https://mailman.uni-konstanz.**de/mailman/listinfo/basex-talk<https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk> >

> And now everything runs smoothly: I had to use the bigger machine for > creating and indexing the collection (now consisting of 677 documents with > 1.9 GB input size resulting in a 2.1 GB collection).

> I would be interested -- mainly for development and debugging -- to access > information concerning query processing (i.e., information displayed in the > "Query Info" buffer in the gui) with the client. I use the Perl API.

Am 14.01.2012 um 00:50 schrieb Christian Grün: >> And now everything runs smoothly: I had to use the bigger machine for >> creating and indexing the collection (now consisting of 677 documents with >> 1.9 GB input size resulting in a 2.1 GB collection). > > Fine; does it mean that the index is now recognized by the optimizer?

>> I would be interested -- mainly for development and debugging -- to access >> information concerning query processing (i.e., information displayed in the >> "Query Info" buffer in the gui) with the client. I use the Perl API. > > That can be done by activating the QUERYINFO option and calling > info(). A little example: > > print $session->execute("set queryinfo on")."\n"; > ... > print $session->execute("xquery 1")."\n"; > print $session->info()."\n"; > ...

> It's a bit strange: I tried to index and re-index and remove the index etc. with BaseX 7.0.2 several times -- not changing my original query -- and one time out of maybe 20 the index was used. Then I switched to BaseX 7.1 and without changing anything, the index is used always. I tried Linux CenOS and Mac OSX with the same result.

>> print $session->execute("set queryinfo on")."\n"; >> ... >> print $session->execute("xquery 1")."\n"; >> print $session->info()."\n"; >> ... > > I have to admit, I don't quite get it and I didn't find relevant things in the examples.

> As a result I would like to see > - the number of results > - the time needed for executing the query and > - the results themselves (preferable one by one for adding additional information, reformating, etc., later I will add a second application to annotate each match and update the collection)

> I tried 'print $xquery->info(), but this seems to work with execute() only, not with more().

> Is there a possibility to store the namespace information somewhere else and not have to write it into every query?

Am 16.01.2012 um 14:04 schrieb Christian Grün: >> As a result I would like to see >> - the number of results >> - the time needed for executing the query and >> - the results themselves (preferable one by one for adding additional information, reformating, etc., later I will add a second application to annotate each match and update the collection) > > The attached perl client may give you the requested results. It > creates the result representation directly within XQuery. This way, > you'll get better performance, and may be more flexible when working > with the evaluated results.

> Next, I'm also using $xquery->bind() for > assigning the query terms, as this reduces the danger of risky query > strings (e.g. including quotes) that could break your query.

>> Is there a possibility to store the namespace information somewhere else and not have to write it into every query? > > Currently, the easiest way is to use wildcards instead of explicitly > specifying the namespace. (... /*:element).

> Thanks. I guess, I cannot do everything directly within XQuery, e.g., extending marked elements to continuous marking, to make "<mark>Korb</mark> <mark>geben</mark>" to be "<mark>Korb geben</mark>" -- it will be more important for queries with ftand or ftor.

> How do I create alternatives for the query? If a user types "A B C" and ticks "process as STRING", the query would be: > > ft:mark(//*[text() contains text "A B C" using stemming using language "de"][self::*:p or self::*:l]) > > If the user ticks "process as AND", the query should be: > > ft:mark(//*[text() contains text ("A" ftand "B" ftand "C") using stemming using language "de" distance at most 10 words][self::*:p or self::*:l])

> I don't know how to process the input-string to create the correct string for 'terms' and how to toggle queries (I would nee the "distance at most 10 words) for discontinuous queries only). Can I handle this by binding other variables, say 'distance' to a value dependent on user input, like: > > ft:mark(//*[text() contains text { \$term } using stemming using language "de" { \$distance }][self::*:p or self::*:l])

> A query could look as follows: > > declare variable $mode := 'STRING'; > declare variable $input := 'This is a c b text'; > declare variable $terms := 'A B C'; > > if($mode = 'STRING') > then $input contains text { $terms } phrase > else if($mode = 'AND') > then $input contains text { $terms } all words > else error((), 'Unknown search mode')

>> Thanks. I guess, I cannot do everything directly within XQuery, >> e.g., extending marked elements to continuous marking, to make >> "<mark>Korb</mark> <mark>geben</mark>" to be "<mark>Korb >> geben</mark>" -- it will be more important for queries with ftand >> or ftor. > > Currently, the ft:mark() and ft:extract() functions are mainly used to > highlight hits in search results, but we are always interested in > extending our XQuery modules with helpful functions/additional > arguments, so feel free to suggest new features (..but I cannot give > any guarantee when a particular request will be implemented). For > example, the latest snapshot contains two new functions ft:tokens() > and ft:tokenize() [1], which have recently been requested.

> I noticed, that it makes no difference for searching if the query is: > > contains text {"A B C"} all words > or > contains text ("A" ftand "B" ftand "C") > > However, applying ft:mark, the first (using "all words") results in marking > only occurences of "C", whereas the second (using "ftand") results in > marking occurences of "A", "B", and "C". Is this a feature/bug of ft:mark?