Can the full text tokenizer be instructed not to tokenize at hyphens?
Tuesday, 13 July 2021 · 3 messages
IL
Imsieke, Gerrit, le-tex13 Jul 2021 07:28 #
A customer wants to include composite terms such as 'third-generation' as single tokens so that they may be offered in a completion list. I don’t think this is configurable, or is it?
Gerrit
CG
Christian Grün13 Jul 2021 08:27 #
Hi Gerrit,
I’m sorry there’s currently no way to adjust that. We’d probably think of how this goes hand in hand with other XQFT features that rely on single-word tokens (such as stemming).
For now, a little extra index could be generated instead, which contains all terms the that are supposed to occur in the completion list.
Cheers,
Christian
Quoted text · le-tex · 8 lines
On Tue, Jul 13, 2021 at 7:28 AM Imsieke, Gerrit, le-tex <gerrit.imsieke at le-tex.de> wrote: > > A customer wants to include composite terms such as 'third-generation' > as single tokens so that they may be offered in a completion list. I > don’t think this is configurable, or is it? > > Gerrit
IL
Imsieke, Gerrit, le-tex13 Jul 2021 08:47 #
That’s a feasible workaround, thank you.
Quoted text · Christian Grün · 23 lines
On 13.07.2021 08:27, Christian Grün wrote: > Hi Gerrit, > > I’m sorry there’s currently no way to adjust that. We’d probably think > of how this goes hand in hand with other XQFT features that rely on > single-word tokens (such as stemming). > > For now, a little extra index could be generated instead, which > contains all terms the that are supposed to occur in the completion > list. > > Cheers, > Christian > > > On Tue, Jul 13, 2021 at 7:28 AM Imsieke, Gerrit, le-tex > <gerrit.imsieke at le-tex.de> wrote: >> >> A customer wants to include composite terms such as 'third-generation' >> as single tokens so that they may be offered in a completion list. I >> don’t think this is configurable, or is it? >> >> Gerrit
Written for BaseX 9.5. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery