db:create, language values
Wednesday, 25 May 2016 · 7 messages · last reply 26 May 2016
CG
Quoted text · France Baril · 1 line
> Is it possible to add the list of supported values in the doc for LANGUAGE at: http://docs.basex.org/wiki/Options#Indexing.
The list depends on your local Java environment. You can get a list via:
declare namespace locale = "java:java.util.Locale"; (locale:getAvailableLocales() ! locale:getLanguage(.)) => distinct-values() => sort()
I have added this example to the documentation.
Quoted text · France Baril · 12 lines
> LANGUAGE > > SignatureLANGUAGE [lang] > Defaulten > SummaryThe specified language will influence the way how an input text will be tokenized. This option is mainly important if tokens are to be stemmed, or if the tokenization of a language differs from Western languages. See Full-Text Index for more details. > > Thanks! > > -- > France Baril > Architecte documentaire / Documentation architect > france.baril at architextus.com
FB
France Baril25 May 2016 20:58 #
Hi,
Is it possible to add the list of supported values in the doc for LANGUAGE at: http://docs.basex.org/wiki/Options#Indexing.
LANGUAGE
*Signature* LANGUAGE [lang]
*Default* en
*Summary* The specified language will influence the way how an input text will be tokenized. This option is mainly important if tokens are to be stemmed, or if the tokenization of a language differs from Western languages. See Full-Text Index
<http://docs.basex.org/wiki/Indexes#Full-Text_Index> for more details.
Thanks!
--
France Baril
Architecte documentaire / Documentation architect
france.baril at architextus.com
KK
Probably the list of available locales is not the same as the list of languages that can be stemmed. I understood the question was about tokenization and full-text indexing in particular and not locales in general.
Maybe I got it wrong, but I would still appreciate hints to technical docs about supported languages with stemming. What components are used for this?
Cheers Kristian K
25.05.2016 20:21 Christian Grün kirjutas:
Quoted text · Christian Grün · 24 lines
>> Is it possible to add the list of supported values in the doc for LANGUAGE at: http://docs.basex.org/wiki/Options#Indexing. > The list depends on your local Java environment. You can get a list via: > > declare namespace locale = "java:java.util.Locale"; > (locale:getAvailableLocales() ! locale:getLanguage(.)) > => distinct-values() > => sort() > > I have added this example to the documentation. > > > >> LANGUAGE >> >> SignatureLANGUAGE [lang] >> Defaulten >> SummaryThe specified language will influence the way how an input text will be tokenized. This option is mainly important if tokens are to be stemmed, or if the tokenization of a language differs from Western languages. See Full-Text Index for more details. >> >> Thanks! >> >> -- >> France Baril >> Architecte documentaire / Documentation architect >> france.baril at architextus.com
CG
Christian Grün25 May 2016 22:29 #
Hi Kristian,
I have slightly updated our Wiki section on language support in [1]. For more information, I invite you to have a look at the related Java classes (e.g. [2,3]) or ask some more questions.
Cheers, Christian
[1] http://docs.basex.org/wiki/Full-Text#Languages [2] https://github.com/BaseXdb/basex/blob/master/basex-core/src/main/java/org/basex/util/ft/Language.java [3] https://github.com/BaseXdb/basex/blob/master/basex-core/src/main/java/org/basex/util/ft/WesternTokenizer.java
Quoted text · Kristian Kankainen · 47 lines
On Wed, May 25, 2016 at 9:27 PM, Kristian Kankainen <kristian at keeleleek.ee> wrote: > Probably the list of available locales is not the same as the list of > languages that can be stemmed. I understood the question was about > tokenization and full-text indexing in particular and not locales in > general. > > Maybe I got it wrong, but I would still appreciate hints to technical docs > about supported languages with stemming. What components are used for this? > > Cheers > Kristian K > > > 25.05.2016 20:21 Christian Grün kirjutas: >>> >>> Is it possible to add the list of supported values in the doc for >>> LANGUAGE at: http://docs.basex.org/wiki/Options#Indexing. >> >> The list depends on your local Java environment. You can get a list via: >> >> declare namespace locale = "java:java.util.Locale"; >> (locale:getAvailableLocales() ! locale:getLanguage(.)) >> => distinct-values() >> => sort() >> >> I have added this example to the documentation. >> >> >> >>> LANGUAGE >>> >>> SignatureLANGUAGE [lang] >>> Defaulten >>> SummaryThe specified language will influence the way how an input text >>> will be tokenized. This option is mainly important if tokens are to be >>> stemmed, or if the tokenization of a language differs from Western >>> languages. See Full-Text Index for more details. >>> >>> Thanks! >>> >>> -- >>> France Baril >>> Architecte documentaire / Documentation architect >>> france.baril at architextus.com > >
FB
Thanks!
Quoted text · Christian Grün · 32 lines
On Wed, May 25, 2016 at 10:21 AM, Christian Grün <christian.gruen at gmail.com> wrote: > > Is it possible to add the list of supported values in the doc for > LANGUAGE at: http://docs.basex.org/wiki/Options#Indexing. > > The list depends on your local Java environment. You can get a list via: > > declare namespace locale = "java:java.util.Locale"; > (locale:getAvailableLocales() ! locale:getLanguage(.)) > => distinct-values() > => sort() > > I have added this example to the documentation. > > > > > LANGUAGE > > > > SignatureLANGUAGE [lang] > > Defaulten > > SummaryThe specified language will influence the way how an input text > will be tokenized. This option is mainly important if tokens are to be > stemmed, or if the tokenization of a language differs from Western > languages. See Full-Text Index for more details. > > > > Thanks! > > > > -- > > France Baril > > Architecte documentaire / Documentation architect > > france.baril at architextus.com >
--
France Baril
Architecte documentaire / Documentation architect
france.baril at architextus.com
FB
I was also interested in stemming. Awesome. I assume the codes for the lucene supported languages are the standard 2 letter codes for the listed language!?
Quoted text · Christian Grün · 69 lines
On Wed, May 25, 2016 at 1:29 PM, Christian Grün <christian.gruen at gmail.com> wrote: > Hi Kristian, > > I have slightly updated our Wiki section on language support in [1]. > For more information, I invite you to have a look at the related Java > classes (e.g. [2,3]) or ask some more questions. > > Cheers, > Christian > > [1] http://docs.basex.org/wiki/Full-Text#Languages > [2] > https://github.com/BaseXdb/basex/blob/master/basex-core/src/main/java/org/basex/util/ft/Language.java > [3] > https://github.com/BaseXdb/basex/blob/master/basex-core/src/main/java/org/basex/util/ft/WesternTokenizer.java > > > > On Wed, May 25, 2016 at 9:27 PM, Kristian Kankainen > <kristian at keeleleek.ee> wrote: > > Probably the list of available locales is not the same as the list of > > languages that can be stemmed. I understood the question was about > > tokenization and full-text indexing in particular and not locales in > > general. > > > > Maybe I got it wrong, but I would still appreciate hints to technical > docs > > about supported languages with stemming. What components are used for > this? > > > > Cheers > > Kristian K > > > > > > 25.05.2016 20:21 Christian Grün kirjutas: > >>> > >>> Is it possible to add the list of supported values in the doc for > >>> LANGUAGE at: http://docs.basex.org/wiki/Options#Indexing. > >> > >> The list depends on your local Java environment. You can get a list via: > >> > >> declare namespace locale = "java:java.util.Locale"; > >> (locale:getAvailableLocales() ! locale:getLanguage(.)) > >> => distinct-values() > >> => sort() > >> > >> I have added this example to the documentation. > >> > >> > >> > >>> LANGUAGE > >>> > >>> SignatureLANGUAGE [lang] > >>> Defaulten > >>> SummaryThe specified language will influence the way how an input text > >>> will be tokenized. This option is mainly important if tokens are to be > >>> stemmed, or if the tokenization of a language differs from Western > >>> languages. See Full-Text Index for more details. > >>> > >>> Thanks! > >>> > >>> -- > >>> France Baril > >>> Architecte documentaire / Documentation architect > >>> france.baril at architextus.com > > > > >
--
France Baril
Architecte documentaire / Documentation architect
france.baril at architextus.com
CG
Christian Grün26 May 2016 08:12 #
Quoted text · France Baril · 3 lines
> I was also interested in stemming. Awesome. I assume the codes for the > lucene supported languages are the standard 2 letter codes for the listed > language!?
You can specify either the language codes or the names of the languages (Bulgarian, Catalan, etc.). Here is yet another query you can call to get all names supported on your system (I assume that all Lucene languages should be included):
declare namespace locale = "java:java.util.Locale";
distinct-values(
locale:getAvailableLocales() !
locale:getDisplayLanguage(., locale:ENGLISH())
)Quoted text · France Baril · 79 lines
> > On Wed, May 25, 2016 at 1:29 PM, Christian Grün <christian.gruen at gmail.com> > wrote: >> >> Hi Kristian, >> >> I have slightly updated our Wiki section on language support in [1]. >> For more information, I invite you to have a look at the related Java >> classes (e.g. [2,3]) or ask some more questions. >> >> Cheers, >> Christian >> >> [1] http://docs.basex.org/wiki/Full-Text#Languages >> [2] >> https://github.com/BaseXdb/basex/blob/master/basex-core/src/main/java/org/basex/util/ft/Language.java >> [3] >> https://github.com/BaseXdb/basex/blob/master/basex-core/src/main/java/org/basex/util/ft/WesternTokenizer.java >> >> >> >> On Wed, May 25, 2016 at 9:27 PM, Kristian Kankainen >> <kristian at keeleleek.ee> wrote: >> > Probably the list of available locales is not the same as the list of >> > languages that can be stemmed. I understood the question was about >> > tokenization and full-text indexing in particular and not locales in >> > general. >> > >> > Maybe I got it wrong, but I would still appreciate hints to technical >> > docs >> > about supported languages with stemming. What components are used for >> > this? >> > >> > Cheers >> > Kristian K >> > >> > >> > 25.05.2016 20:21 Christian Grün kirjutas: >> >>> >> >>> Is it possible to add the list of supported values in the doc for >> >>> LANGUAGE at: http://docs.basex.org/wiki/Options#Indexing. >> >> >> >> The list depends on your local Java environment. You can get a list >> >> via: >> >> >> >> declare namespace locale = "java:java.util.Locale"; >> >> (locale:getAvailableLocales() ! locale:getLanguage(.)) >> >> => distinct-values() >> >> => sort() >> >> >> >> I have added this example to the documentation. >> >> >> >> >> >> >> >>> LANGUAGE >> >>> >> >>> SignatureLANGUAGE [lang] >> >>> Defaulten >> >>> SummaryThe specified language will influence the way how an input text >> >>> will be tokenized. This option is mainly important if tokens are to be >> >>> stemmed, or if the tokenization of a language differs from Western >> >>> languages. See Full-Text Index for more details. >> >>> >> >>> Thanks! >> >>> >> >>> -- >> >>> France Baril >> >>> Architecte documentaire / Documentation architect >> >>> france.baril at architextus.com >> > >> > > > > > > -- > France Baril > Architecte documentaire / Documentation architect > france.baril at architextus.com
Written for BaseX 8.4. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery