Changes

|{{Func|ft:tokenize|$input as xs:string|xs:string*}}<br/>{{Func|ft:tokenize|$input as xs:string, $options as item()|xs:string*}}

|Tokenizes the given {{Code|$input}} string, using the current default full-text optionsor the {{Code|$options}} specified as second argument.The following options are available:* {{Code|case}}: determines how character case is processed. Allowed values are {{Code|insensitive}}, {{Code|sensitive}}, {{Code|upper}} and {{Code|lower}}. By default, search is case insensitive.* {{Code|diacritics}}: determines how diacritical characters are processed. Allowed values are {{Code|insensitive}} and {{Code|sensitive}}. By default, search is diacritical insensitive.* {{Code|stemming}}: determines is tokens are stemmed. Allowed values are {{Code|true}} and {{Code|false}}. By default, stemming is turned off.* {{Code|language}}: determines the language. This option is relevant for stemming tokens. All language codes are supported. The default language is {{Code|en}}.The {{Code|$options}} argument can be used to control full-text processing. Options can be either specified<br/>* as children of an {{Code|&lt;options/&gt;}} element, e.g.:<pre class="brush:xml"><options> <key1 value='value1'/> ...</options></pre>* as map, which contains all key/value pairs:<pre class="brush:xml">map { "key1": "value1", ... }</pre>

* {{Code|<code>ft:tokenize("No Doubt")}} </code> returns the two strings {{Code|no}} and {{Code|doubt}}.* <code>ft:tokenize("École", map { 'diacritics': 'sensitive' })</code> returns the string {{Code|école}}.* <code>declare ft-option using stemming; ft:tokenize("GIFTS")}} </code> returns a single string {{Code|gift}}.