full text search collation

> I am trying to use it with > Greek texts and the default collation assumes that ά != α

> As a side question i noticed that the stemmers used from lucne are quite > outdated. 3.6.0 also includes a Greek stemmer. I tried to include the 3.6.0 > stemmers instead but language codes seem to be hardcoded in > util/ft/Language.java

> If you'd manage to provide me with some appropriate tables for Greek > characters, I'll be glad to extend this mapping.

> Do you have a direct reference to your prefered Greek stemmer class?

On 2012-06-22, Charles Kowalski <alxarch at gmail.com> wrote: > I am concerned though because this is not always the desired behavior. > Sometimes (ie in an academic context) I could see the need for > accent-sensitive searches. > The optimal scenario would be to have these mappings in an easily > parsable text file (just like stopword list behaves). > OTOH this would be reinventing the collation wheel (an oversimplified > version of it) > For example the mapping above only covers modern Greek text. > Ancient/Polytonic Greek has a lot more mappings > that are not needed for modern greek. Also I'm pretty sure other > languages have such needs too. > > I'd like to hear your thoughts on this.

> If i understood correctly i used the unicode codes from > http://unicode.org/charts/PDF/U0370.pdf > to produce the following mapping: > [...]

> I am concerned though because this is not always the desired behavior. > Sometimes (ie in an academic context) I could see the need for > accent-sensitive searches.

> OTOH this would be reinventing the collation wheel (an oversimplified > version of it)

> Yes, this is reinventing the wheel, and my impression is that Token.java > is also a reinvented wheel, and I'm sorry to say that it doesn't look > very round to me.

"Christian Grün" <christian.gruen at gmail.com> wrote:

> If i understood correctly i used the unicode codes from > http://unicode.org/charts/PDF/U0370.pdf > to produce the following mapping: > [...]

> I am concerned though because this is not always the desired behavior. > Sometimes (ie in an academic context) I could see the need for > accent-sensitive searches.

> OTOH this would be reinventing the collation wheel (an oversimplified > version of it)

> The stemmer OTOH does not seem to be working. > I think it needs to be integrated in the same way that the other lucene > stemmers are integrated, > using the whole lucene-analyzers-3.6.0.jar instead of the > lucene-stemmers-3.4.0.jar.

> Thanks for your feedback; I already guessed that this might take a > little bit more time. Could you provide us with some simple example > queries and their expected result? Similar to.. > > "ά" contains text "α" → true > "..." contains text "..." using stemming using language "el" → ... > > Thanks in advance, > Christian

On Sat, Jun 23, 2012 at 2:24 PM, Christian Grün <christian.gruen at gmail.com> wrote: > Hi Αλέξανδρος, > >> The stemmer OTOH does not seem to be working. >> I think it needs to be integrated in the same way that the other lucene >> stemmers are integrated, >> using the whole lucene-analyzers-3.6.0.jar instead of the >> lucene-stemmers-3.4.0.jar. > > Thanks for your feedback; I already guessed that this might take a > little bit more time. Could you provide us with some simple example > queries and their expected result? Similar to.. > > "ά" contains text "α" → true > "..." contains text "..." using stemming using language "el" → ... > > Thanks in advance, > Christian

On 2012-06-23, Christian Grün <christian.gruen at gmail.com> wrote: >> Yes, this is reinventing the wheel, and my impression is that Token.java >> is also a reinvented wheel, and I'm sorry to say that it doesn't look >> very round to me. > > I completely agree; the more languages we support, the more reasonable > it seems to resort to Java’s existing Unicode framework.

> Performance was the main reason why we initially introduced our own > mappings and algorithms. As we internally keep all strings as UTF8 > byte arrays (i.e., in the same way as they are stored on disk), we can > often avoid converting bytes to Strings and vice versa, as this can > get quite costly if it's done millions of times. This is also one of > the reason why our existing full-text algorithms are more efficient > than some competing implementations.

> However, due to the increasing demands BaseX is confronted with, we'll > probably need to make some more compromises here, or – even better – > find a hybrid solution (e.g., avoid expensive algorithms whenever a > string is known to have only standard characters). Our existing > full-text architecture actually provides more than enough features > when it comes to the majority of our existing projects, but it's > rather poor from a linguistic perspective.

>> Performance was the main reason why we initially introduced our own >> mappings and algorithms. As we internally keep all strings as UTF8 >> byte arrays (i.e., in the same way as they are stored on disk), we can >> often avoid converting bytes to Strings and vice versa, as this can >> get quite costly if it's done millions of times. This is also one of >> the reason why our existing full-text algorithms are more efficient >> than some competing implementations. > > Well, one may also say you're cutting corners... As a computer > scientist, I think BaseX is great, and that's also why I'm following the > mailing list even though I'm not currently using BaseX. Why not? > Because I'm also a digital humanities researcher. I'm working with > narrative-oriented documents, but BaseX currently has a very strong bias > towards record-oriented Latin-1 documents, so strong in fact, that it > is, IMHO, not suited for narrative-oriented documents. > > AFAIK, many of the issues that make BaseX currently unsuited for > narrative-oriented documents (and thus the TEI community in general) are > explicit design decisions, for example, chopping whitespace by default, > not supporting the full-text search Ignore option, and using a limited > implementation of Unicode. > > From my point of view, this is unfortunate: if the full-text search > doesn't work for my documents, it doesn't help that it is fast. The > same goes for Unicode support: if BaseX doesn't support combining > characters (which are not only needed for our historical texts), it > doesn't help it's fast for some restricted set of characters. It's nice > "α" now matches "ά" (U+03AC) as precomposed character--but it still > doesn't match "ά" (U+03B1 + U+0301) using a combining accent, even > though the two encodings are canonically equivalent as per the Unicode > Standard. The same goes for "ä" and "ä" and an infinite number of > further combinations. When you create collections from different > sources, you will probably end up with documents using precomposed > characters *and* combining characters. The difference is only the > encoding, it will look identical when displayed to users, who expect > (and rightly so) to find all instances of "alpha with tonos" or "a with > diaeresis," regardless of the encoding. > >> However, due to the increasing demands BaseX is confronted with, we'll >> probably need to make some more compromises here, or – even better – >> find a hybrid solution (e.g., avoid expensive algorithms whenever a >> string is known to have only standard characters). Our existing >> full-text architecture actually provides more than enough features >> when it comes to the majority of our existing projects, but it's >> rather poor from a linguistic perspective. > > I'd say, the problem is not so much the linguistics, but the > language-independent Unicode support, which would give you a basic level > of functionality for all languages. > > Best regards > > -- > Dr.-Ing. Michael Piotrowski, M.A. <mxp at cl.uzh.ch> > Institute of Computational Linguistics, University of Zurich > Phone +41 44 63-54313 | OpenPGP public key ID 0x1614A044 > * OUT NOW: Systems and Frameworks for Computational Morphology > * <http://www.springeronline.com/978-3-642-23137-7> > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk

On 2012-06-23, Christian Grün <christian.gruen at gmail.com> wrote: > Thanks for your feedback. In a nutshell: yes, it's quite a challenge > to satisfy the wild range of scenarios BaseX is used for, which is why > we need to set priorities, and can't do justice to all users.

> As a matter of fact, the project is Open Source, and contributions are > welcome and needed (as long as features are not financially > sponsored).

> Btw, what's your opinion on the Lucene tokenizers and stemmers? As you > may know, they also focus on performance. They also bypass Java's > Unicode normalization algorithms and do everything by themselves, > which is we they may be more relevant to us than the standard Java > libraries.

>> Thanks for your feedback. In a nutshell: yes, it's quite a challenge >> to satisfy the wild range of scenarios BaseX is used for, which is why >> we need to set priorities, and can't do justice to all users. > > I agree. I believe, however, that everybody would benefit from good > Unicode support :-) > >> As a matter of fact, the project is Open Source, and contributions are >> welcome and needed (as long as features are not financially >> sponsored). > > I'm an open-source author myself, so I know what you mean. > Unfortunately, I currently don't have any capacities left for > contributing code to BaseX. > >> Btw, what's your opinion on the Lucene tokenizers and stemmers? As you >> may know, they also focus on performance. They also bypass Java's >> Unicode normalization algorithms and do everything by themselves, >> which is we they may be more relevant to us than the standard Java >> libraries. > > My understanding is that this is mostly for historical reasons, not > (primarily) for performance reasons; and some of them are probably hacks > and workarounds. Lucene's support for Unicode has a number of problems, > and I think they're now moving towards the use of ICU [1], see, e.g., > > <http://2010.lucene-eurocon.org/sessions-track1-day1.html#2> > > Some of this is already available in contrib: > > <https://issues.apache.org/jira/browse/LUCENE-1488> > <http://lucene.apache.org/core/3_6_0/api/contrib-icu/index.html> > > I think it would be a good idea for BaseX to have a look at ICU. > > Best regards > > > Footnotes: > [1] <http://site.icu-project.org/> > > -- > Dr.-Ing. Michael Piotrowski, M.A. <mxp at cl.uzh.ch> > Institute of Computational Linguistics, University of Zurich > Phone +41 44 63-54313 | OpenPGP public key ID 0x1614A044 > * OUT NOW: Systems and Frameworks for Computational Morphology > * <http://www.springeronline.com/978-3-642-23137-7> > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk