Main Page » Mailing List » January 2012

index features

Tuesday, 17 January 2012 · 7 messages · last reply 14 Feb 2012

Cerstin Mahlow17 Jan 2012 15:10 #

Hi, I have three questions concerning working with the full-text index: The first question is about "distance" information. Given this query: (1) 'contains text "Kopf Sand Stecken" all words using stemming using language "de"' There is no difference to: (2) 'contains text ("Nase" ftand "Sand" ftand "stecken") using stemming using language "de"' both queries deliver 4 nodes. If I would like to find the query terms within a certain distance, adding 'distance at most 10 words' for (1) I get 2 nodes (a subset of the 4 from the first run), but for (2) I still get all 4 nodes. The information concerning distance doesn't seem to be considered. For my application this is no problem, since I have to go for the "ftand"-variant to get proper marking, but in general this looks strange. The second question is about "ftand" and "ftor". If I try these queries: (3) 'contains text ("Nase" ftand "Sand" ftand "stecken") using stemming using language "de" distance at most 10 words' (4) 'contains text ("Kopf" ftand "Sand" ftand "stecken") using stemming using language "de" distance at most 10 words I get 2 hits for (3) and 11 for (4). So I assumed I would get 13 hits (the ones from (3) and the ones from (4) when changing the query to: (5) 'contains text (("Nase" ftor "Kopf") ftand "Sand" ftand "stecken") using stemming using language "de" distance at most 10 words' However, I get 6 hits -- none of them containing "Nase" (there is no difference, if the query starts with '"Nase" ftor "Kopf"' or with '"Kopf" ftor "Nase"'). Did I mess something up? The third question is about the full-text index itself. When applying fuzzy search or using wildcards, the full-text index is not applied -- resulting in a time out on my website, I need 341859.09 ms in the GUI for applying 'ft:mark (//*[text() contains text ("Korb" ftand "geben") using fuzzy][self::*:p or self::*:l])' to my 3 GB collection. The information at the "Full-Text" tab says: - Structure: Trie - Stemming: ON - Case Sensitivity: ON - Diacritics: ON - Language: German - Size: 1 GB - Entries: 1743744 I created the full-text index with the option "Support Wildcards", too, but this information is not shown in the Database properties. When creating the index, "SET WILDCARDS true" is shown. I used stemming, casesensitivity, diacritics, and wildcards -- is this an unrecommended combination? Thank you very much in advance Cerstin

-- Dr. phil. Cerstin Mahlow Universität Basel Deutsches Seminar Nadelberg 4 4051 Basel Schweiz Tel: +41 61 267 07 65 Fax: +41 61 267 34 40 Mail: cerstin.mahlow at unibas.ch Web: http://www.oldphras.net ---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program.

Christian Grün18 Jan 2012 14:24 #

Cerstin, sorry for delaying the answer; be sure I'll give you detailed feedback as soon as I've resolved some other open issues.

Quoted text · Cerstin Mahlow · 98 lines

On Tue, Jan 17, 2012 at 3:10 PM, Cerstin Mahlow <cerstin.mahlow at unibas.ch> wrote: > Hi, > > I have three questions concerning working with the full-text index: > > The first question is about "distance" information. Given this query: > > (1) 'contains text "Kopf Sand Stecken" all words using stemming using > language "de"' > > There is no difference to: > > (2) 'contains text ("Nase" ftand "Sand" ftand "stecken") using stemming > using language "de"' > > both queries deliver 4 nodes. > > If I would like to find the query terms within a certain distance, adding > > 'distance at most 10 words' > > for (1) I get 2 nodes (a subset of the 4 from the first run), but for (2) I > still get all 4 nodes. The information concerning distance doesn't seem to > be considered. For my application this is no problem, since I have to go for > the "ftand"-variant to get proper marking, but in general this looks > strange. > > > > The second question is about "ftand" and "ftor". If I try these queries: > > (3) 'contains text ("Nase" ftand "Sand" ftand "stecken") using stemming > using language "de" distance at most 10 words' > (4) 'contains text ("Kopf" ftand "Sand" ftand "stecken") using stemming > using language "de" distance at most 10 words > > I get 2 hits for (3) and 11 for (4). So I assumed I would get 13 hits (the > ones from (3) and the ones from (4) when changing the query to: > > (5) 'contains text (("Nase" ftor "Kopf") ftand "Sand" ftand "stecken") using > stemming using language "de" distance at most 10 words' > > However, I get 6 hits -- none of them containing "Nase" (there is no > difference, if the query starts with '"Nase" ftor "Kopf"' or with '"Kopf" > ftor "Nase"'). > > Did I mess something up? > > > The third question is about the full-text index itself. When applying fuzzy > search or using wildcards, the full-text index is not applied -- resulting > in a time out on my website, I need 341859.09 ms in the GUI for applying > > 'ft:mark (//*[text() contains text ("Korb" ftand "geben") using > fuzzy][self::*:p or self::*:l])' > > to my 3 GB collection. The information at the "Full-Text" tab says: > > - Structure: Trie > - Stemming: ON > - Case Sensitivity: ON > - Diacritics: ON > - Language: German > - Size: 1 GB > - Entries: 1743744 > > I created the full-text index with the option "Support Wildcards", too, but > this information is not shown in the Database properties. When creating the > index, "SET WILDCARDS true" is shown. I used stemming, casesensitivity, > diacritics, and wildcards -- is this an unrecommended combination? > > Thank you very much in advance > > Cerstin > > -- > Dr. phil. Cerstin Mahlow > > Universität Basel > Deutsches Seminar > Nadelberg 4 > 4051 Basel > Schweiz > > Tel: +41 61 267 07 65 > Fax: +41 61 267 34 40 > Mail: cerstin.mahlow at unibas.ch > Web: http://www.oldphras.net > > ---------------------------------------------------------------- > This message was sent using IMP, the Internet Messaging Program. > > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk

Christian Grünin reply to Cerstin Mahlow, 17 Jan 2012, 15:1018 Jan 2012 23:41 #

Quoted text · Cerstin Mahlow · 19 lines

> (1) 'contains text "Kopf Sand Stecken" all words using stemming using > language "de"' > > There is no difference to: > > (2) 'contains text ("Nase" ftand "Sand" ftand "stecken") using stemming > using language "de"' > > both queries deliver 4 nodes. > > If I would like to find the query terms within a certain distance, adding > > 'distance at most 10 words' > > for (1) I get 2 nodes (a subset of the 4 from the first run), but for (2) I > still get all 4 nodes. The information concerning distance doesn't seem to > be considered. For my application this is no problem, since I have to go for > the "ftand"-variant to get proper marking, but in general this looks > strange.

This may well be correct, as (unfortunately) the internal data model of XQuery Full Text is pretty complex and leads to frequent misunderstandings. To give more information, I'll have to look at the actual data; do you think you can provide me with a little document that exemplifies your observation?

Quoted text · Cerstin Mahlow · 2 lines

> The second question is about "ftand" and "ftor". If I try these queries: > ...

Once more, it might be helpful to have the actual data at hand..

Quoted text · Cerstin Mahlow · 3 lines

> The third question is about the full-text index itself. When applying fuzzy > search or using wildcards, the full-text index is not applied -- resulting > in a time out on my website, I need 341859.09 ms in the GUI for applying

Currently, the choice has to be made between efficient fuzzy or wildcard matching (the latter being based on a Trie index structure). Some information on that can be found in our Wiki [1] (btw, feel free to edit the Wiki if you feel it's incomplete!). We are working on a new index structure that will unify both index structures, improve performance, and support incremental updates. We may even eliminate the explicit choice of some other full-text options, such that those options can be dynamically chosen without the need to reindex the database. More feature requests regarding the full-text index are welcome. Christian [1] http://docs.basex.org/wiki/Full-Text

Cerstin Mahlow22 Jan 2012 11:14 #

Hi Christian, I come back to some previously discussed questions: Zitat von Christian Grün <christian.gruen at gmail.com>: [...]

Quoted text · Christian Grün · 3 lines

> To give more information, I'll have to look at the > actual data; do you think you can provide me with a little document > that exemplifies your observation?

As I am not sure, if the behavior has something to do with my actual data, I didn't create an example, but put a sample of my collection consisting of 4 smaller documents online: <http://oldphras.unibas.ch/test.tgz> //*[text() contains text ('Kopf' ftand 'Sand' ftand 'stecken') using stemming using language "de"][self::*:p or self::*:l] gives 3 hits (in Wille, Suttner, and Cervantes) //*[text() contains text ('Kopf' ftand 'Sand' ftand 'stecken') using stemming using language "de" distance at most 10 words][self::*:p or self::*:l] gives 2 hits (in Wille and Suttner) //*[text() contains text "Kopf Sand stecken" all words using stemming using language "de" distance at most 10 words][self::*:p or self::*:l] gives 3 hits (in Wille, Suttner, and Cervantes), the "distance" option seems to be ignored.

Quoted text · Christian Grün · 1 line

>> The second question is about "ftand" and "ftor".

//*[text() contains text ('Kopf' ftand 'Sand' ftand 'stecken') using stemming using language "de" distance at most 10 words][self::*:p or self::*:l] gives 2 hits (in Wille and Suttner) //*[text() contains text ('Nase' ftand 'Sand' ftand 'stecken') using stemming using language "de" distance at most 10 words][self::*:p or self::*:l] gives 1 hit (in Müllenhoff) Therefore, for //*[text() contains text ( ('Nase' ftor 'Kopf') ftand 'Sand' ftand 'stecken') using stemming using language "de" distance at most 10 words][self::*:p or self::*:l] I would expect to get all 3 hits, but actually get only 1 (the one in Wille). It makes no difference, if I put ('Nase' ftor 'Kopf') or ('Kopf' ftor 'Nase'). Additionally, the highlighting is strange. In the end, I would like to search for something like this to speed up annotating the data: ( Nase | Kopf | Hals ) & ( Sand | Schlinge ) & ( ziehen | stecken )

Quoted text · Christian Grün · 6 lines

>> The third question is about the full-text index itself. When applying fuzzy >> search or using wildcards, the full-text index is not applied -- resulting >> in a time out on my website, I need 341859.09 ms in the GUI for applying > > Currently, the choice has to be made between efficient fuzzy or > wildcard matching (the latter being based on a Trie index structure).

So I can have fuzzy OR stemming and wildcard. For searching it's OK, I copied the collection and created the other index for the copy, but as I wan't to update the collection after searching, I would have to update both collections and re-index them after updating one. Is this correct? Best regards Cerstin

Christian Grün30 Jan 2012 01:47 #

Dear Cerstin, some quick feedback on your last e-mail, to get sure it doesn't get lost..

Quoted text · Cerstin Mahlow · 5 lines

> //*[text() contains text "Kopf Sand stecken" all words using stemming using > language "de" distance at most 10 words][self::*:p or self::*:l] > > gives 3 hits (in Wille, Suttner, and Cervantes), the "distance" option seems > to be ignored.

True, this is a bug, which I've documented in a new GitHub issue [1]. Currently, this query returns different results if the full-text is activated or not, while it should always return the same results. Thanks for the analysis.

Quoted text · Cerstin Mahlow · 4 lines

> So I can have fuzzy OR stemming and wildcard. For searching it's OK, I > copied the collection and created the other index for the copy, but as I > wan't to update the collection after searching, I would have to update both > collections and re-index them after updating one. Is this correct?

Yep, that's correct. Another tracker entry [2] addresses this: in future, we will only provide one single full-text index that will support both fuzzy and wildcard searches. Ideally, the new index will already support incremental update, such that there won't be any need to re-index your data. Christian [1] https://github.com/BaseXdb/basex/issues/359 [2] https://github.com/BaseXdb/basex/issues/346

Cerstin Mahlow9 Feb 2012 16:04 #

Hi, I just downloaded the 7.1 stable version.

Quoted text · Christian Grün · 8 lines

Am 30.01.2012 um 01:47 schrieb Christian Grün: >> //*[text() contains text "Kopf Sand stecken" all words using stemming using >> language "de" distance at most 10 words][self::*:p or self::*:l] >> >> gives 3 hits (in Wille, Suttner, and Cervantes), the "distance" option seems >> to be ignored. > > True, this is a bug, which I've documented in a new GitHub issue [1].

It seems that the "ordered" option is ignored as well.

Quoted text · Christian Grün · 1 line

> [1] https://github.com/BaseXdb/basex/issues/359

Best regards Cerstin

Christian Grün14 Feb 2012 19:37 #

Hi Cerstin, thanks; this issue seems in fact to be related to the distance issue [1], which I've just extended with a small comment. Keeping you updated, Christian [1] https://github.com/BaseXdb/basex/issues/359

Quoted text · Cerstin Mahlow · 47 lines

On Thu, Feb 9, 2012 at 4:04 PM, Cerstin Mahlow <cerstin.mahlow at unibas.ch> wrote: > Hi, > > I just downloaded the 7.1 stable version. > > Am 30.01.2012 um 01:47 schrieb Christian Grün: > >>> //*[text() contains text "Kopf Sand stecken" all words using stemming using >>> language "de" distance at most 10 words][self::*:p or self::*:l] >>> >>> gives 3 hits (in Wille, Suttner, and Cervantes), the "distance" option seems >>> to be ignored. >> >> True, this is a bug, which I've documented in a new GitHub issue [1]. > > It seems that the "ordered" option is ignored as well. > >> [1] https://github.com/BaseXdb/basex/issues/359 > > Best regards > > Cerstin > > -- > Dr. phil. Cerstin Mahlow > > Universität Basel > Deutsches Seminar > Nadelberg 4 > 4051 Basel > Schweiz > > Tel: +41 61 267 07 65 > Fax: +41 61 267 34 40 > Mail: cerstin.mahlow at unibas.ch > Web: http://www.oldphras.net > > > > > > > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk

Written for BaseX 7.0, 7.1. BaseX may have changed since: please check the current documentation.

⚡Generated with XQuery

`J`	Select the next message
`K`	Select the previous message
`N`	Next thread
`P`	Previous thread
`U`	Go up one level
`/`	Jump to the search field