Bureaucrats, editor, reviewer, Administrators
13,550
edits
Changes
Jump to navigation
Jump to search
Matching The indexed text nodes can be accessed directly requested from the index with the XQuery function {{Function|Database|db:text}}. The index contents indexed string values can be accessed looked up via {{Function|Index|index:text}}.
Text nodes can directly be retrieved from the index via the XQuery function With {{Function|Database|db:text-range}}, you can access all text nodes whose values are between a minimum and maximum value.
Attributes ''Attribute nodes'' with a matching value (containing at least one from a set of given tokens ) can be directly retrieved from the index with the XQuery function {{Function|Database|db:token}}. The index contents (''token strings'') can be accessed with {{Function|Index|index:tokens}}.
no edit summary
The query compiler tries to optimize and speed up queries by applying the index whenever it is possible and seems promising. To see how a query is rewritten, and if an index is used, you can turn on the [[GUI#Visualizations|Info View]] in the GUI or use the [[Command-Line Options#BaseX_Standalone|-V flag]] on the command line:
* A message like <code>Applying apply text index for "Japan"</code> indicates that the text index is applied to speed up the search of the shown string. The following message…* <code>Removing path with no index results</code> indicates that a string in a path expression will never yield results. Because of that, the path does not need to be evaluated at all.* If you cannot find any index optimization hints in the info output, it often helps if you rewrite and simplify your query.<br/> Additional examples for index rewritings are presented in our article on [[XQuery Optimizations]].
=Structural Indexes=
The name index is e.g. applied to discard location steps that will never yield results:
<pre classsyntaxhighlight lang="brush:xquery">
(: will be rewritten to an empty sequence :)
/non-existing-name
</presyntaxhighlight>
The contents of the name indexes can be directly accessed with the XQuery functions [[Index Module#index:element-names|index:element-names]] and [[Index Module#index:attribute-names|index:attribute-names]].
==Path Index==
The path index (which is also called ''path summary'' or ''data guide'') stores all distinct paths of the documents in the database. It contains additional statistical information, such as the number of occurrence of a path, its distinct string values, and the minimum/maximum of numeric values. The maximum number of distinct values to store per name can be changed via {{Option|MAXCATS}}. Since {{Version|8.6}}, the distinct Distinct values are also stored for elements and attributes of numeric type.
Various queries will be evaluated much faster if an up-to-date path index is available (as can be observed when opening the [[GUI#Visualizations|Info View]]):
* Descendant steps will be rewritten to multiple child steps. Child steps are evaluated faster, as fewer nodes have to be traversed:
<pre classsyntaxhighlight lang="brush:xquery">
doc('factbook.xml')//province,
(: ...will be rewritten to... :)
doc('factbook.xml')/mondial/country/province
</presyntaxhighlight>
* The {{Code|fn:count}} function will be pre-evaluated by looking up the number in the index:
<pre classsyntaxhighlight lang="brush:xquery">
count(doc('factbook')//country)
</presyntaxhighlight>
* The distinct values of elements or attributes can be looked up in the index as well:
<pre classsyntaxhighlight lang="brush:xquery">
distinct-values(db:open('factbook')//religions)
</presyntaxhighlight>
The contents of the path index can be directly accessed with the XQuery function [[Index Module#index:facets|index:facets]].
With XQuery, index structures can be created and dropped via {{Function|Database|db:optimize}}:
<pre classsyntaxhighlight lang="brush:xquery">
(: Optimize specified database, create full-text index for texts of the specified elements :)
db:optimize(
map { 'ftindex': true(), 'ftinclude': 'p div' }
)
</presyntaxhighlight>
==Text Index==
===Exact Queries===
This index speeds up string-based equality tests on references text nodesof documents. It will be utilized to accelerate string comparisons in path expressions. The following queries will all be rewritten for index access:
<pre classsyntaxhighlight lang="brush:xquery"> (: 1st example 1 :)
//*[text() = 'Germany'],
(: 2nd example 2 :)
doc('factbook.xml')//name[. = 'Germany'],
(: 3rd example 3 :)
for $c in db:open('factbook')//country
where $c//city/name = 'Hanoi'
return $c/name
</presyntaxhighlight> Before the actual index rewriting takes places, some preliminary optimizations are applied:* In example 2, the context item expression {{Code|.}} will be replaced with a {{Code|text()}} step.* In example 3, the {{Code|where}} clause will be rewritten to a predicate and attached to the first path expression.
The {{Option|UPDINDEX}} option can be activated enabled to keep this index up-to-date, for example:
<pre classsyntaxhighlight lang="brush:xquery">
db:optimize(
'mydb',
map { 'updindex':true(), 'textindex': true(), 'textinclude':'id' }
)
</presyntaxhighlight>
===Range Queries===
The text index also supports range queries based on string comparisons:
<pre classsyntaxhighlight lang="brush:xquery"> (: 1st example 1 :)
db:open('Library')//Medium[Year >= '2011' and Year <= '2016'],
(: 2nd example 2 :)
let $min := '2014-04-16T00:00:00'
let $max := '2014-04-19T23:59:59'
return db:open('news')//entry[date-time > $min and date-time < $max]
</presyntaxhighlight>
Please note that the current index structures do not support queries for numbers and dates.
==Attribute Index==
Similar to the text index, this index speeds up string-based equality and range tests comparisons on attribute values. Additionally, the XQuery function {{Code|fn:id}} takes advantage of the index whenever possible. The following queries will all be rewritten for index access:
<pre classsyntaxhighlight lang="brush:xquery">
(: 1st example :)
//country[@car_code = 'J'],
(: 4th example :)
fn:id('f0_119', db:open('factbook'))
</presyntaxhighlight>
''Attribute nodes '' (which you can use as starting points of navigation) can directly be retrieved from the index with the XQuery functions {{Function|Database|db:attribute}} and {{Function|Database|db:attribute-range}}. The index contents (''strings'') can be accessed with {{Function|Index|index:attributes}}.
The {{Option|UPDINDEX}} option can be activated to keep this index up-to-date.
In many XML dialects, such as HTML or DITA, multiple tokens are stored in attribute values. The token index can be created to speed up the retrieval of these tokens. The XQuery functions {{Code|fn:contains-token}}, {{Code|fn:tokenize}} and {{Code|fn:idref}} are rewritten for index access whenever possible. If a token index exists, it will e.g. be utilized for the following queries:
<pre classsyntaxhighlight lang="brush:xquery">
(: 1st example :)
//div[contains-token(@class, 'row')],
(: 3rd example :)
doc('graph.xml')/idref('edge8')
</presyntaxhighlight>
==Full-Text Index==
The [[Full-Text]] index contains the normalized tokens of text nodes of a document. It is utilized to speed up queries with the {{Code|contains text}} expression, and it is capable of processing wildcard and fuzzy search operations. Three evaluation strategies are available: the standard sequential database scan, a full-text index based evaluation and a hybrid one, combining both strategies (see [httphttps://wwwfiles.infbasex.uni-konstanz.deorg/gkpublications/pubsys/publishedFiles/GrGaHo09Gruen%20et%20al.%20%5B2009%5D,%20XQuery%20Full%20Text%20Implementation%20in%20BaseX.pdf XQuery Full Text implementation in BaseX]).
If the full-text index exists, the following queries will all be rewritten for index access:
<pre classsyntaxhighlight lang="brush:xquery">
(: 1st example :)
//country[name/text() contains text 'and'],
//religions[.//text() contains text { 'Catholic', 'Roman' }
using case insensitive distance at most 2 words]
</presyntaxhighlight>
The index provides support for the following full-text features (the values can be changed in the GUI or via the {{Command|SET}} command):
The options that have been used for creating the full-text index will also be applied to the optimized full-text queries. However, the defaults can be overwritten if you supply options in your query. For example, if words were stemmed in the index, and if the query can be rewritten for index access, the query terms will be stemmed as well, unless stemming is not explicitly disabled. This is demonstrated in the following [[Commands#Command_Scripts|Command Script]]:
<pre classsyntaxhighlight lang="brush:xml">
<commands>
<!-- Create database with stemmed full-text index -->
<xquery> /text[. contains text { 'houses' } using no stemming] </xquery>
</commands>
</presyntaxhighlight>
Text nodes can be directly requested from the index via the XQuery function {{Function|Full-Text|ft:search}}. The index contents can be accessed with {{Function|Full-Text|ft:tokens}}.
; Commands
<pre classsyntaxhighlight lang="brush:xml">
SET ATTRINCLUDE id,name
CREATE DB factbook http://files.basex.org/xml/factbook.xml'
# Restore default
SET ATTRINCLUDE
</presyntaxhighlight>
; XQuery
<pre classsyntaxhighlight lang="brush:xquery">
db:create('factbook', 'http://files.basex.org/xml/factbook.xml', '',
map { 'attrinclude': 'id,name' })
</presyntaxhighlight>
With {{Command|CREATE INDEX}} and {{Function|Database|db:optimize}}, new selective indexing options will ba applied to an existing database.
==Enforce Rewritings==
In various cases, existing index structures will not be utilized by the query optimizer. This is usually the case if the name of the database is not a static string (e.g., because it is bound to a variable or passed on as argument of a function call). Furthermore, several candidates for index rewritings may exist, and the query optimizer may decide for a rewriting that turns out to be suboptimal.
With the {{Option|ENFORCEINDEX}} option, certain index rewritings can be enforced. While the option can be globally enabled, it is usually better to supply it as [[XQuery Extensions#Pragmas|Pragma]]. Two examples:
* In the query below, 10 databases will be addressed. If it is known in advance that these databases contain an up-to-date text index, the index rewriting can be enforced as follows:
<syntaxhighlight lang="xquery">
(# db:enforceindex #) {
for $n in 1 to 10
let $db := 'persons' || $n
return db:open($db)//person[name/text() = 'John']
}
</syntaxhighlight>
* The following query contains two predicates that may both be rewritten for index access. If the automatically chosen rewriting is known not to be optimal, another index rewriting can enforced by surrounding the specific expression with the pragma:
<syntaxhighlight lang="xquery">
db:open('factbook')//country
[(# db:enforceindex #) {
@population > '10000000' and
@population < '10999999'
}]
[religions/text() = 'Protestant']
</syntaxhighlight>
The option can also be assigned to predicates with dynamic values. In the following example the comparison of the first comparison will be rewritten for index access. Without the pragma expression, the second comparison is preferred and chosen for the rewriting, because the statically known string allows for an exact cost estimation:
<syntaxhighlight lang="xquery">
for $name in ('Germany', 'Italy')
for $country in db:open('factbook')//country
where (# db:enforceindex #) { $country/name = $name }
where $country/religions/text() = 'Protestant'
return $country
</syntaxhighlight>
Please note that:
* The option should only be enabled if the addressed databases exist, have all required index structures and are up-to-date (otherwise, you will be given an error message).
* If you address the full-text index, and if you use non-default indexing options, you will have to specify them in your query (via {{Code|using stemming}}, {{Code|using language 'de'}}, etc).
* If you have more than one enforce pragma in a single path expression, only the first will be considered.
* In general, there are always expressions that cannot be rewritten for index access. If you enforce rewritings, you will have no guarantee that an index will be used.
=Custom Index Structures=
With XQuery, it is comparatively easy to create your own, custom index structures. The following query demonstrate how you can create a {{Code|factbook-index}} database, which contains all texts of the original database in lower case:
<pre classsyntaxhighlight lang="brush:xquery">
let $db := 'factbook'
return db:create($db || '-index', $index, $db || '-index.xml')
</presyntaxhighlight>
In the following query, a text string is searched, and the text nodes of the original database are retrieved:
<pre classsyntaxhighlight lang="brush:xquery">
let $db := 'factbook'
let $text := 'italian'
for $id in db:open($db || '-index')//*[@string = $text]/id
return db:open-id($db, $id)/..
</presyntaxhighlight>
With some extra effort, and if {{Option|UPDINDEX}} is enabled for both your original and your index database (see below), your index database will support updates as well (try it, it’s fun!).
If {{Option|DEBUG}} is enabled, the command-line output might help you to find a good split size. The following example shows the output for creating a database for an XMark document with 1 GB, and with 128 MB assigned to the JVM:
<presyntaxhighlight>
> basex -d -c"SET FTINDEX ON; SET TOKENINDEX ON; CREATE DB xmark 1gb.xml"
Creating Database...
Indexing Full-Text...
..|.|.|.|...|...|..|.|..| 116.33 M operations, 138740.94 ms (106 MB). Recommended SPLITSIZE: 12.
</presyntaxhighlight>
The output can be interpreted as follows:
=Changelog=
;Version 9.1
* Updated: [[#Enforce Rewritings|Enforce Rewritings]], support for comparisons with dynamic values.
;Version 9.0
* Added: [[#Enforce Rewritings|Enforce Rewritings]]
;Version 8.4
