Full-Text Functions

This module extends the Full-Text features of BaseX: The index can be directly accessed, full-text results can be marked with additional elements, or the relevant parts can be extracted.

The tokenization of Japanese input is described in Full-Text Japanese.

Conventions

All functions and errors are in the http://basex.org/modules/ft namespace, to which the ft prefix is statically bound.

ft:search and ft:tokens access the full-text index of a database. All other functions operate on arbitrary strings and nodes; what they have in common is that they are aware of the language of the input, and that they tokenize, normalize and stem it accordingly.

The String Module complements this module: it measures how similar two arbitrary strings are. Its similarity functions accept the same options as ft:tokenize, and apply them to their input. See the String Module for a comparison of the two.

Full-text options that are declared in the query prolog with declare ft-option are inherited by ft:search, ft:contains, ft:tokenize and ft:normalize. Options that are supplied in the $options argument of a function call take precedence.

Database Functions

Updated: REBUILD REQUIRED. See Full-Text Index for more details.

ft:search

Updated: New stop-words option; full-text options declared in the query prolog are inherited.

ft:search(  $database  as xs:string,  $terms     as item()*,  $options   as map(*)?  := {}) as text()*

ft:search('DB', 'QUERY')

ft:search('DB', ('2010', '2020'), { 'mode': 'all' })

ft:search('db', ('A', 'B'), {
  'mode': 'all words',
  'distance': { 'max': 5, 'unit': 'words' }
})

let $terms := 'Hello Worlds'
let $fuzzy := true()
for $db in 1 to 3
let $dbname := 'DB' || $db
return ft:search($dbname, $terms, { 'fuzzy': $fuzzy })/..

ft:tokens

Updated: New fuzzy and errors options for looking up tokens with a bounded number of errors.

ft:tokens(  $database  as xs:string,  $prefix    as xs:string?  := (),  $options   as map(*)?  := {}) as element(entry)*

let $term := ft:tokenize('Exercises')
return number(ft:tokens('db', $term)[. = $term]/@count)

ft:tokens('db', 'rembrandt', { 'fuzzy': true(), 'errors': 2 })

let $candidates := ft:tokens('db', 'rembrandt', { 'fuzzy': true(), 'errors': 2 })
return string:closest('rembrandt', $candidates, { 'limit': 3 })

Highlighting

ft:mark

Updated: Constructed nodes are accepted; the name of the marker element must be an NCName.

ft:mark(  $nodes  as node()*,  $name   as xs:string?  := ()) as node()*

ft:mark(db:get('DB')//*[text() contains text 'hello'])

let $start := 1
let $end   := 10
let $term  := 'welcome'
let $test  := fn($node) { $node/text() contains text { $term } }
for $ft in (db:get('DB')//*[$test(.)])[position() = $start to $end]
return ft:mark($ft[$test(.)])

ft:mark(<xml>hello world</xml>[text() contains text 'world'], 'b')

ft:extract

Updated: Constructed nodes are accepted; the name of the marker element must be an NCName.

ft:extract(  $nodes   as node()*,  $name    as xs:string?  := (),  $length  as xs:integer?  := ()) as node()*

ft:extract(db:get('DB')//*[text() contains text 'hello'], 'b', 1)

Tokenization

Updated: Letters that denote multiple characters are expanded (ß → ss, æ → ae, …).

ft:tokenize

ft:tokenize(  $value    as xs:string?,  $options  as map(*)?  := {}) as xs:string*

ft:tokenize('No Doubt')

ft:tokenize('École', { 'diacritics': 'sensitive' })

declare ft-option using stemming; ft:tokenize('GIFTS')

ft:tokenize("Vincent van Gogh's Sunflowers (1888)")

ft:normalize

ft:normalize(  $value    as xs:string?,  $options  as map(*)?  := {}) as xs:string

ft:normalize('Häuser am Meer', { 'case': 'sensitive' })

ft:normalize('Rübens Ærø Łódź straße Đurađ ıstanbul')

General Functions

ft:contains

Updated: New stop-words and occurs options; full-text options declared in the query prolog are inherited.

ft:contains(  $input    as item()*,  $terms    as item()*,  $options  as map(*)?  := {}) as xs:boolean

ft:contains('John Doe', ('jack', 'john'), { 'mode': 'any' })

for $s in (true(), false())
return ft:contains('Häuser', 'Haus', { 'stemming': $s, 'language':'de' })

ft:contains('to be or not to be', 'be', { 'occurs': { 'min': 2 } })

ft:contains('sunflowers', 'the sunflowers',
  { 'mode': 'all words', 'stop-words': 'the' })

ft:count

Updated: Constructed nodes are accepted.

ft:count(  $nodes  as node()*) as xs:integer

ft:count(//*[text() contains text 'QUERY'])

ft:score

ft:score(  $input  as item()*) as xs:double*

ft:score('a' contains text 'a')

ft:thesaurus

ft:thesaurus(  $node     as node(),  $term     as xs:string,  $options  as map(*)?  := {}) as xs:string*

ft:thesaurus(
  <thesaurus>
    <entry>
      <term>happy</term>
      <synonym>
        <term>lucky</term>
        <relationship>RT</relationship>
      </synonym>
    </entry>
  </thesaurus>,
  'happy'
)

Errors

Changelog

• Updated: Database Functions: REBUILD REQUIRED
• Updated: ft:search, ft:contains: new stop-words option; ft:contains: new occurs option.
• Updated: ft:search and ft:contains inherit the full-text options that are declared in the query prolog.
• Updated: ft:mark, ft:extract: the name of the marker element must be an NCName.
• Updated: ft:mark, ft:extract and ft:count also accept constructed nodes.
• Updated: ft:tokens: new fuzzy and errors options for looking up index tokens with a bounded number of errors.
• Updated: Expand letters to multiple character (ß → ss, æ → ae, …).

• Added: ft:thesaurus
• Updated: ft:search, ft:contains: new errors option.

• Updated: ft:tokenize and ft:normalize can be called with empty sequence.

• Updated: error codes updated; errors now use the module namespace

• Added: ft:contains, ft:normalize
• Updated: Options added to ft:tokenize

• Added: ft:contains
• Updated: Options added to ft:search

• Updated: the functions no longer accept database nodes as reference. Instead, the name of a database must now be specified.

• Updated: ft:search (second argument generalized, third parameter added)

• Added: ft:tokens, ft:tokenize