multi-language full-text indexing

-----Ursprüngliche Nachricht----- Von: Christian Grün [mailto:christian.gruen at gmail.com] Gesendet: Mittwoch, 22. April 2015 11:03 An: Goetz Heller Cc: BaseX Betreff: Re: [basex-talk] multi-language full-text indexing > It is desirable to have > documents indexed by locale-specific parts, e.g. I can see that this would absolutely make sense, but it would be quite some effort to realize it. There are also various conceptul issues related to XQuery Full Text: If you don't specify the language in the query, we'd need to dynamically decide what stemmers to use for the query strings, depending on the nodes that are currently targeted. This would pretty much blow up the existing architecture. As there are so many other types of index structures that could be helpful, depending on the particular use case, we usually recommend users to create additional BaseX databases, which can then serve as indexes. This can all be done in XQuery. I remember there have been various examples for this on this mailing list (see e.g. [1,2]). Hope this helps, Christian [1] https://www.mail-archive.com/basex-talk at mailman.uni-konstanz.de/msg04837.html [2] https://www.mail-archive.com/basex-talk at mailman.uni-konstanz.de/msg06089.html > > > > CREATE FULL-TEXT INDEX ON DATABASE XY STARTING WITH ( > > (path_a)/LOCALIZED_PART_A[@LANG=$lang], > > (path_b)/LOCALIZED_PART_B[@LG=$lang],… > > ) FOR LANGUAGE $lang IN ( > > BG, > > DN, > > DE WITH STOPWORDS filepath_de WITH STEM = YES, > > EN WITH STOPWORDS filepath_en, > > FR, … > > ) [USING language_code_map] > > and then to write full-text retrieval queries with a clause such as > ‘FOR LANGUAGE BG’, for example. The index parts would be much smaller > and full-text retrieval therefore much faster. The language codes > would be mapped somehow to standard values recognized by BaseX in the > language_code_map file. > > Are there any efforts towards such a feature?

On Wed, Apr 22, 2015 at 11:35 AM, Goetz Heller <heller at hellerim.de> wrote: > Here's another addendum: Even if multi-language full-text indexing is not going tob e implemented in the near future, it still would be a useful feature to be able to restrict full-text indexing to parts of a document, e.g. > > CREATE FULL-TEXT INDEX ON DATABASE XY STARTING WITH ( > (path_a)/PART_A, > (path_b)/ PART_B,… > ) > > Kind regards, > > Goetz > > -----Ursprüngliche Nachricht----- > Von: Christian Grün [mailto:christian.gruen at gmail.com] > Gesendet: Mittwoch, 22. April 2015 11:03 > An: Goetz Heller > Cc: BaseX > Betreff: Re: [basex-talk] multi-language full-text indexing > >> It is desirable to have >> documents indexed by locale-specific parts, e.g. > > I can see that this would absolutely make sense, but it would be quite some effort to realize it. There are also various conceptul issues related to XQuery Full Text: If you don't specify the language in the query, we'd need to dynamically decide what stemmers to use for the query strings, depending on the nodes that are currently targeted. > This would pretty much blow up the existing architecture. > > As there are so many other types of index structures that could be helpful, depending on the particular use case, we usually recommend users to create additional BaseX databases, which can then serve as indexes. This can all be done in XQuery. I remember there have been various examples for this on this mailing list (see e.g. [1,2]). > > Hope this helps, > Christian > > [1] https://www.mail-archive.com/basex-talk at mailman.uni-konstanz.de/msg04837.html > [2] https://www.mail-archive.com/basex-talk at mailman.uni-konstanz.de/msg06089.html > > > > >> >> >> >> CREATE FULL-TEXT INDEX ON DATABASE XY STARTING WITH ( >> >> (path_a)/LOCALIZED_PART_A[@LANG=$lang], >> >> (path_b)/LOCALIZED_PART_B[@LG=$lang],… >> >> ) FOR LANGUAGE $lang IN ( >> >> BG, >> >> DN, >> >> DE WITH STOPWORDS filepath_de WITH STEM = YES, >> >> EN WITH STOPWORDS filepath_en, >> >> FR, … >> >> ) [USING language_code_map] >> >> and then to write full-text retrieval queries with a clause such as >> ‘FOR LANGUAGE BG’, for example. The index parts would be much smaller >> and full-text retrieval therefore much faster. The language codes >> would be mapped somehow to standard values recognized by BaseX in the >> language_code_map file. >> >> Are there any efforts towards such a feature? >

On Wed, Apr 22, 2015 at 11:35:48AM +0200, Goetz Heller wrote: > Here's another addendum: Even if multi-language full-text indexing is not going tob e implemented in the near future, it still would be a useful feature to be able to restrict full-text indexing to parts of a document, e.g. > > CREATE FULL-TEXT INDEX ON DATABASE XY STARTING WITH ( > (path_a)/PART_A, > (path_b)/ PART_B,… > ) > > Kind regards, > > Goetz > > -----Ursprüngliche Nachricht----- > Von: Christian Grün [mailto:christian.gruen at gmail.com] > Gesendet: Mittwoch, 22. April 2015 11:03 > An: Goetz Heller > Cc: BaseX > Betreff: Re: [basex-talk] multi-language full-text indexing > > > It is desirable to have > > documents indexed by locale-specific parts, e.g. > > I can see that this would absolutely make sense, but it would be quite some effort to realize it. There are also various conceptul issues related to XQuery Full Text: If you don't specify the language in the query, we'd need to dynamically decide what stemmers to use for the query strings, depending on the nodes that are currently targeted. > This would pretty much blow up the existing architecture. > > As there are so many other types of index structures that could be helpful, depending on the particular use case, we usually recommend users to create additional BaseX databases, which can then serve as indexes. This can all be done in XQuery. I remember there have been various examples for this on this mailing list (see e.g. [1,2]). > > Hope this helps, > Christian > > [1] https://www.mail-archive.com/basex-talk at mailman.uni-konstanz.de/msg04837.html > [2] https://www.mail-archive.com/basex-talk at mailman.uni-konstanz.de/msg06089.html > > > > > > > > > > > > CREATE FULL-TEXT INDEX ON DATABASE XY STARTING WITH ( > > > > (path_a)/LOCALIZED_PART_A[@LANG=$lang], > > > > (path_b)/LOCALIZED_PART_B[@LG=$lang],… > > > > ) FOR LANGUAGE $lang IN ( > > > > BG, > > > > DN, > > > > DE WITH STOPWORDS filepath_de WITH STEM = YES, > > > > EN WITH STOPWORDS filepath_en, > > > > FR, … > > > > ) [USING language_code_map] > > > > and then to write full-text retrieval queries with a clause such as > > ‘FOR LANGUAGE BG’, for example. The index parts would be much smaller > > and full-text retrieval therefore much faster. The language codes > > would be mapped somehow to standard values recognized by BaseX in the > > language_code_map file. > > > > Are there any efforts towards such a feature? >

> Wed, Apr 22, 2015 at 11:20 PM, Chris Yocum <cyocum at gmail.com> wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > Hi, > > I just want to say that for the dictionary that I used BaseX for, > having a multi-lingual full text would have been very nice. Bar that > a partial index based on certain rules the user supplies would have > also been nice. For instance, being able to distinguish between á, a, > and ā in a word. In early Irish textual criticism, length marks are > often added by text editors with a macron to denote a long vowel that > has been idenified by the editor but is not in the original text. > Being able to say: "build an index with á and a but not ā" would be > helpful. > > I would suggest as a first pass building the index by using xml:lang > attributes to determine what stemmer to use, etc. If the document has > supplied them, you could use them to build the indices differently > based on it. > > All the best, > Chris > > On Wed, Apr 22, 2015 at 11:35:48AM +0200, Goetz Heller wrote: >> Here's another addendum: Even if multi-language full-text indexing is not going tob e implemented in the near future, it still would be a useful feature to be able to restrict full-text indexing to parts of a document, e.g. >> >> CREATE FULL-TEXT INDEX ON DATABASE XY STARTING WITH ( >> (path_a)/PART_A, >> (path_b)/ PART_B,… >> ) >> >> Kind regards, >> >> Goetz >> >> -----Ursprüngliche Nachricht----- >> Von: Christian Grün [mailto:christian.gruen at gmail.com] >> Gesendet: Mittwoch, 22. April 2015 11:03 >> An: Goetz Heller >> Cc: BaseX >> Betreff: Re: [basex-talk] multi-language full-text indexing >> >> > It is desirable to have >> > documents indexed by locale-specific parts, e.g. >> >> I can see that this would absolutely make sense, but it would be quite some effort to realize it. There are also various conceptul issues related to XQuery Full Text: If you don't specify the language in the query, we'd need to dynamically decide what stemmers to use for the query strings, depending on the nodes that are currently targeted. >> This would pretty much blow up the existing architecture. >> >> As there are so many other types of index structures that could be helpful, depending on the particular use case, we usually recommend users to create additional BaseX databases, which can then serve as indexes. This can all be done in XQuery. I remember there have been various examples for this on this mailing list (see e.g. [1,2]). >> >> Hope this helps, >> Christian >> >> [1] https://www.mail-archive.com/basex-talk at mailman.uni-konstanz.de/msg04837.html >> [2] https://www.mail-archive.com/basex-talk at mailman.uni-konstanz.de/msg06089.html >> >> >> >> >> > >> > >> > >> > CREATE FULL-TEXT INDEX ON DATABASE XY STARTING WITH ( >> > >> > (path_a)/LOCALIZED_PART_A[@LANG=$lang], >> > >> > (path_b)/LOCALIZED_PART_B[@LG=$lang],… >> > >> > ) FOR LANGUAGE $lang IN ( >> > >> > BG, >> > >> > DN, >> > >> > DE WITH STOPWORDS filepath_de WITH STEM = YES, >> > >> > EN WITH STOPWORDS filepath_en, >> > >> > FR, … >> > >> > ) [USING language_code_map] >> > >> > and then to write full-text retrieval queries with a clause such as >> > ‘FOR LANGUAGE BG’, for example. The index parts would be much smaller >> > and full-text retrieval therefore much faster. The language codes >> > would be mapped somehow to standard values recognized by BaseX in the >> > language_code_map file. >> > >> > Are there any efforts towards such a feature? >> > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1 > > iF4EAREIAAYFAlU4EIcACgkQDjE+CSbP7HrWyAEAsW698gwkbKrtEb9Vkv7S1aV/ > r5YyIp/UKv9k8gYxqboA/0+oesCxM6K2dZGEfIolUJG+x3vhxMfocY+QGpwjtLhy > =Zq8Q > -----END PGP SIGNATURE-----

On Wed, Apr 22, 2015 at 10:24 PM, Christian Grün <christian.gruen at gmail.com> wrote: > Chris, > > Thanks for your feedback. > > Yes, I see that there is a lot of demand for a more customizable > full-text index. Did you already try to build some additional index > databases, based on the rules you were listing here? It's not as > comfortable as a tightly coupled full-text index, but the more use > case I get to hear of, the more I wonder if we could at all manage to > satisfy everyone's needs.. > > Cheers, > Christian > > >> Wed, Apr 22, 2015 at 11:20 PM, Chris Yocum <cyocum at gmail.com> wrote: >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA256 >> >> Hi, >> >> I just want to say that for the dictionary that I used BaseX for, >> having a multi-lingual full text would have been very nice. Bar that >> a partial index based on certain rules the user supplies would have >> also been nice. For instance, being able to distinguish between á, a, >> and ā in a word. In early Irish textual criticism, length marks are >> often added by text editors with a macron to denote a long vowel that >> has been idenified by the editor but is not in the original text. >> Being able to say: "build an index with á and a but not ā" would be >> helpful. >> >> I would suggest as a first pass building the index by using xml:lang >> attributes to determine what stemmer to use, etc. If the document has >> supplied them, you could use them to build the indices differently >> based on it. >> >> All the best, >> Chris >> >> On Wed, Apr 22, 2015 at 11:35:48AM +0200, Goetz Heller wrote: >>> Here's another addendum: Even if multi-language full-text indexing is not going tob e implemented in the near future, it still would be a useful feature to be able to restrict full-text indexing to parts of a document, e.g. >>> >>> CREATE FULL-TEXT INDEX ON DATABASE XY STARTING WITH ( >>> (path_a)/PART_A, >>> (path_b)/ PART_B,… >>> ) >>> >>> Kind regards, >>> >>> Goetz >>> >>> -----Ursprüngliche Nachricht----- >>> Von: Christian Grün [mailto:christian.gruen at gmail.com] >>> Gesendet: Mittwoch, 22. April 2015 11:03 >>> An: Goetz Heller >>> Cc: BaseX >>> Betreff: Re: [basex-talk] multi-language full-text indexing >>> >>> > It is desirable to have >>> > documents indexed by locale-specific parts, e.g. >>> >>> I can see that this would absolutely make sense, but it would be quite some effort to realize it. There are also various conceptul issues related to XQuery Full Text: If you don't specify the language in the query, we'd need to dynamically decide what stemmers to use for the query strings, depending on the nodes that are currently targeted. >>> This would pretty much blow up the existing architecture. >>> >>> As there are so many other types of index structures that could be helpful, depending on the particular use case, we usually recommend users to create additional BaseX databases, which can then serve as indexes. This can all be done in XQuery. I remember there have been various examples for this on this mailing list (see e.g. [1,2]). >>> >>> Hope this helps, >>> Christian >>> >>> [1] https://www.mail-archive.com/basex-talk at mailman.uni-konstanz.de/msg04837.html >>> [2] https://www.mail-archive.com/basex-talk at mailman.uni-konstanz.de/msg06089.html >>> >>> >>> >>> >>> > >>> > >>> > >>> > CREATE FULL-TEXT INDEX ON DATABASE XY STARTING WITH ( >>> > >>> > (path_a)/LOCALIZED_PART_A[@LANG=$lang], >>> > >>> > (path_b)/LOCALIZED_PART_B[@LG=$lang],… >>> > >>> > ) FOR LANGUAGE $lang IN ( >>> > >>> > BG, >>> > >>> > DN, >>> > >>> > DE WITH STOPWORDS filepath_de WITH STEM = YES, >>> > >>> > EN WITH STOPWORDS filepath_en, >>> > >>> > FR, … >>> > >>> > ) [USING language_code_map] >>> > >>> > and then to write full-text retrieval queries with a clause such as >>> > ‘FOR LANGUAGE BG’, for example. The index parts would be much smaller >>> > and full-text retrieval therefore much faster. The language codes >>> > would be mapped somehow to standard values recognized by BaseX in the >>> > language_code_map file. >>> > >>> > Are there any efforts towards such a feature? >>> >> -----BEGIN PGP SIGNATURE----- >> Version: GnuPG v1 >> >> iF4EAREIAAYFAlU4EIcACgkQDjE+CSbP7HrWyAEAsW698gwkbKrtEb9Vkv7S1aV/ >> r5YyIp/UKv9k8gYxqboA/0+oesCxM6K2dZGEfIolUJG+x3vhxMfocY+QGpwjtLhy >> =Zq8Q >> -----END PGP SIGNATURE-----