Changes

=Introduction=

This is a simple example for a basic full-text expression:

"This is YOUR World" contains text "your world"

It yields {{Code|true}}, because the search string is ''tokenized'' before it is compared with the tokenized input string. In the tokenization process, several normalizations take place. Many of those steps can hardly be simulated with plain XQuery: as an example, upper/lower case and diacritics (umlauts, accents, etc.) are removed and an optional, language-dependent stemming algorithm is applied. Beside that, special characters such as whitespaces and punctuation marks will be ignored. Thus, this query also yields true:

"Well... Done!" contains text "well, done"

The {{Code|occurs}} keyword comes into play when more than one occurrence of a token is to be found:

"one and two and three" contains text "and" occurs at least 2 times

==Combining Results==

In the given example, curly braces are used to combine multiple keywords:

for $country in doc('factbook')//country

where $country//religions[text() contains text { 'Sunni', 'Shia' } any]

return $country/name

The query will output the names of all countries with a religion element containing {{Code|sunni}} or {{Code|shia}}. The {{Code|any}} keyword is optional; it can be replaced with:

The keywords {{Code|ftand}}, {{Code|ftor}} and {{Code|ftnot}} can also be used to combine multiple query terms. The following query yields the same result as the last one does:

doc('factbook')//country[descendant::religions contains text 'sunni' ftor 'shia']/name

The keywords {{Code|not in}} are special: they are used to find tokens which are not part of a longer token sequence:

for $text in ("New York", "new conditions")

return $text contains text "New" not in "New York"

Due to the complex data model of the XQuery Full Text spec, the usage of {{Code|ftand}} may lead to a high memory consumption. If you should encounter problems, simply use the {{Code|all}} keyword:

doc('factbook')//country[descendant::religions contains text { 'Christian', 'Jewish'} all]/name

==Positional Filters==

A popular retrieval operation is to filter texts by the distance of the searched words. In this query…

<xml>

<text>There is some reason why ...</text>

<text>The reason why some people ...</text>

</xml>//text[. contains text { "some", "reason" } all ordered distance at most 3 words]

…the two first texts will be returned as result, because there are at most three words between {{Code|some}} and {{Code|reason}}. Additionally, the {{Code|ordered}} keyword ensures that the words are found in the specified order, which is why the third text is excluded. Note that {{Code|all}} is required here to guarantee that only those hits will be accepted that contain all searched words.

The {{Code|window}} keyword is related: it accepts those texts in which all keyword occur within the specified number of tokens. Can you guess what is returned by the following query?

("A C D", "A B C D E")[. contains text { "A", "E" } all window 3 words]

Sometimes it is interesting to only select texts in which all searched terms occur in the {{Code|same sentence}} or {{Code|paragraph}} (you can even filter for {{Code|different}} sentences/paragraphs). This is obviously not the case in the following example:

'Mary told me, “I will survive!”.' contains text { 'will', 'told' } all words same sentence

By the way: In some examples above, the {{Code|words}} unit was used, but {{Code|sentences}} and {{Code|paragraphs}} would have been valid alternatives.

* If {{Code|case}} is insensitive, no distinction is made between characters in upper and lower case. By default, the option is {{Code|insensitive}}; it can also be set to {{Code|sensitive}}:

"Respect Upper Case" contains text "Upper" using case sensitive

* If {{Code|diacritics}} is insensitive, characters with and without diacritics (umlauts, characters with accents) are declared as identical. By default, the option is {{Code|insensitive}}; it can also be set to {{Code|sensitive}}:

* If {{Code|stemming}} is activated, words are shortened to a base form by a language-specific stemmer:

"catch" contains text "catches" using stemming

* With the {{Code|stop words}} option, a list of words can be defined that will be ignored when tokenizing a string. This is particularly helpful if the full-text index takes too much space (a standard stopword list for English texts is provided in the directory {{Code|etc/stopwords.txt}} in the full distributions of BaseX, and available online at http://files.basex.org/etc/stopwords.txt):

"You and me" contains text "you or me" using stop words ("and", "or"),

"You and me" contains text "you or me" using stop words at "http://files.basex.org/etc/stopwords.txt"

* Related terms such as synonyms can be found with the sophisticated [[#Thesaurus|Thesaurus]] option.

* <code>.{min,max}</code> matches ''min''–''max'' number of characters.

"This may be interesting in the year 2000" contains text { "interest.*", "2.{3,3}" } using wildcards

This was a quick introduction to XQuery Full Text; you are invited to explore the numerous other features of the language!

A list of all language codes that are available on your system can be retrieved as follows:

declare namespace locale = "java:java.util.Locale";

distinct-values(locale:getAvailableLocales() ! locale:getLanguage(.))

By default, unless the languages codes <code>ja</code>, <code>ar</code>, <code>ko</code>, <code>th</code>, or <code>zh</code> are specified, a tokenizer for Western texts is used:

* Paragraph delimiters are newlines (<code>&amp;#xa;</code>).

The JAR files are included in the ZIP and EXE distributions of BaseX.

The following two queries, which both return <code>true</code>, demonstrate that stemming depends on the selected language:

"Indexing" contains text "index" using stemming,

"häuser" contains text "haus" using stemming using language "German"

==Scoring==

The scoring model of BaseX takes into consideration the number of found terms, their frequency in a text, and the length of a text. The shorter the input text is, the higher scores will be:

(: Score values: 1 0.62 0.45 :)

for $text score $score in ("A", "A B", "A B C")[. contains text "A"]

order by $score descending

return <hit score='{ format-number($score, "0.00") }'>{ $text }</hit>

This simple approach has proven to consistently deliver good results, and in particular when little is known about the structure of the queried XML documents.

Please note that scores will only be computed if a parent expression requests them:

(: Computes and returns a scoring value. :)

let score $score := <x>Hello Universe</x> contains text "hello"

let score $score := $result

return $score

Scores will be propagated by the {{Code|and}} and {{Code|or}} expressions and in predicates. In the following query, all returned scores are equal:

let $text := "A B C"

let score $s1 := $text contains text "A" ftand "B C"

let score $s5 := $text[. contains text "A"][. contains text "B C"]

return ($s1, $s2, $s3, $s4, $s5)

==Thesaurus==

BaseX supports full-text queries using thesauri, but it does not provide a default thesaurus. This is why queries such as:

'computers' contains text 'hardware'

using thesaurus default

will return <code>false</code>. However, if the thesaurus is specified, then the result will be <code>true</code>:

'computers' contains text 'hardware'

using thesaurus at 'XQFTTS_1_0_4/TestSources/usability2.xml'

==Fuzzy Querying==

'''Document 'doc.xml'''':

<doc>

<a>house</a>

<a>haus</a>

</doc>

'''Query:'''

//a[text() contains text 'house' using fuzzy]

'''Result:'''

<a>house</a>

<a>hous</a>

Fuzzy search is also supported by the full-text index.

=Mixed Content=

<p>This is only an illustrative <hi>example</hi>, not a <q>real</q> text.</p>

Note that the node structure is ignored by the full-text tokenizer: The {{Code|contains text}} expression applies all full-text operations to the ''string value'' of its left operand. As a consequence, the <code>ft:mark</code> and <code>ft:extract</code> functions (see [[Full-Text Module|Full-Text Functions]]) will only yield useful results if they are applied to single text nodes, as the following example demonstrates:

(: Structure is ignored; no highlighting: :)

ft:mark(//p[. contains text 'real'])

(: Single text nodes are addressed: results will be highlighted: :)

ft:mark(//p[.//text() contains text 'real'])

let $docs := db:open('docs')

return db:create(

map { 'ftindex': true() }

)

=Functions=

|-

| {{Code|decomposition}}

|}

* If a default collation is specified, it applies to all collation-dependent string operations in the query. The following expression yields <code>true</code>:

declare default collation 'http://basex.org/collation?lang=de;strength=secondary';

'Straße' = 'Strasse'

* Collations can also be specified in {{Code|order by}} and {{Code|group by}} clauses of FLWOR expressions. This query returns {{Code|à plutôt! bonjour!}}:

for $w in ("bonjour!", "à plutôt!") order by $w collation "?lang=fr" return $w

* Various string function exists that take an optional collation as argument: The following functions give us {{Code|a}} and {{Code|1 2 3}} as results:

distinct-values(("a", "á", "à"), "?lang=it-IT;strength=primary"),

index-of(("a", "á", "à"), "a", "?lang=it-IT;strength=primary")

=Changelog=

; Version 8.0: