Changes

The name index contains all element and attribute references to the names of a database, all elements and the fixed-size index ids are stored attributes in the main database table. If a database is updated, new names are automatically added. Furthermore, the index is enriched with It contains some basic statistical information, such as the distinct (categorical) or minimum and maximum values number of its elements and attributes. The maximum number occurrence of categories to store per a name can be changed via [[Options#MAXCATS|MAXCATS]]. The index are discarded after [[#Updates|database updates]].

The name index is e.g. applied to pre-evaluate discard location steps that will never yield results:

<pre classsyntaxhighlight lang="brush:xquery">

</presyntaxhighlight>

The path index (which is also called ''path summary'' or ''data guide'') stores all distinct paths of the documents in the database. It contains similar additional statistical information , such as the number of occurrence of a path, its distinct string values, and the minimum/maximum of numeric values. The maximum number of distinct values to store per name indexcan be changed via {{Option|MAXCATS}}. The index Distinct values are discarded after [[#Updates|database updates]]also stored for elements and attributes of numeric type.

The Various queries will be evaluated much faster if an up-to-date path index is applied to rewrite descendant steps to multiple child steps. Child steps available (as can be evaluated faster, as fewer nodes have to be accessedobserved when opening the [[GUI#Visualizations|Info View]]):

* Descendant steps will be rewritten to multiple child steps. Child steps are evaluated faster, as fewer nodes have to be traversed: <pre classsyntaxhighlight lang="brush:xquery">

</presyntaxhighlight>

* The paths statistics are e.g. used to pre-evaluate the {{Code|fn:count}} functionwill be pre-evaluated by looking up the number in the index:

<pre classsyntaxhighlight lang="brush:xquery"> count(: will doc('factbook')//country)</syntaxhighlight> * The distinct values of elements or attributes can be rewritten and pre-evaluated by looked up in the path index as well:)count<syntaxhighlight lang="xquery">distinct-values( docdb:open('factbook')//country religions)</presyntaxhighlight>

In the GUI, index structures can be managed in the dialog windows for creating new databases or displaying the database properties. On command-line, the commands <code>[[Commands#CREATE INDEX{{Command|CREATE INDEX]]</code> }} and <code>[[Commands#DROP INDEX{{Command|DROP INDEX]]</code> }} are used to create and drop index structures. With <code>[[Commands#INFO INDEX{{Command|INFO INDEX]]</code>}}, you get some insight into the contents of an index structure, and <code>[[Commands#SET{{Command|SET]]</code> }} allows you to change the index defaults for new databases:

With XQuery, index structures can be created and dropped via [[{{Function|Database Module#db:optimize|db:optimize]]}}:

<pre classsyntaxhighlight lang="brush:xquery">

</presyntaxhighlight>

This index speeds up string-based equality tests on references text nodesof documents. It will be utilized to accelerate string comparisons in path expressions. The following queries will all be rewritten for index access:

<pre classsyntaxhighlight lang="brush:xquery"> (: 1st example 1 :)

(: 2nd example 2 :)

(: 3rd example 3 :)

</presyntaxhighlight>

Matching text nodes can be directly requested from Before the actual index rewriting takes places, some preliminary optimizations are applied:* In example 2, the context item expression {{Code|.}} will be replaced with the XQuery function [[Database Module#db:texta {{Code|db:text]]()}} step. The index contents can * In example 3, the {{Code|where}} clause will be accessed via [[Index Module#index:texts|index:texts]]rewritten to a predicate and attached to the first path expression.

The <code>[[Options#UPDINDEXindexed text nodes can be accessed directly with the XQuery function {{Function|Database|db:text}}. The indexed string values can be looked up via {{Function|Index|index:text}}. The {{Option|UPDINDEX]]</code> }} option can be activated enabled to keep this index up-to-date.: <syntaxhighlight lang="xquery">db:optimize( 'mydb', true(), map { 'updindex':true(), 'textindex': true(), 'textinclude':'id' })</syntaxhighlight>

<pre classsyntaxhighlight lang="brush:xquery"> (: 1st example 1 :)

(: 2nd example 2 :)

</presyntaxhighlight>

Text nodes can directly be retrieved from the index via the XQuery function [[With {{Function|Database Module#|db:text-range|db:}}, you can access all text-range]]nodes whose values are between a minimum and maximum value.

Please note that the current index structures do not support queries for numbers and dates.

Similar to the text index, this index speeds up string-based equality and range tests comparisons on attribute values. Additionally, the XQuery function {{Code|fn:id}} takes advantage of the index whenever possible. The following queries will all be rewritten for index access:

<pre classsyntaxhighlight lang="brush:xquery">

</presyntaxhighlight>

''Attribute nodes '' (which you can use as starting points of navigation) can directly be retrieved from the index with the XQuery functions [[{{Function|Database Module#db:attribute|db:attribute]] }} and [[{{Function|Database Module#db:attribute-range|db:attribute-range]]}}. The index contents (''strings'') can be accessed with [[{{Function|Index Module#index:attributes|index:attributes]]}}.

The [[Options#UPDINDEX{{Option|UPDINDEX]] }} option can be activated to keep this index up-to-date.

<pre classsyntaxhighlight lang="brush:xquery">

</presyntaxhighlight>

Attributes ''Attribute nodes'' with a matching value (containing at least one from a set of given tokens ) can be directly retrieved from the index with the XQuery function [[{{Function|Database Module#db:token|db:token]]}}. The index contents (''token strings'') can be accessed with [[{{Function|Index Module#index:tokens|index:tokens]]}}.

The [[Full-Text]] index contains the normalized tokens of text nodes of a document. It is utilized to speed up queries with the {{Code|contains text}} expression, and it is capable of processing wildcard and fuzzy search operations. Three evaluation strategies are available: the standard sequential database scan, a full-text index based evaluation and a hybrid one, combining both strategies (see [httphttps://wwwfiles.infbasex.uni-konstanz.deorg/gkpublications/pubsys/publishedFiles/GrGaHo09Gruen%20et%20al.%20%5B2009%5D,%20XQuery%20Full%20Text%20Implementation%20in%20BaseX.pdf XQuery Full Text implementation in BaseX]).

<pre classsyntaxhighlight lang="brush:xquery">

</presyntaxhighlight>

The index provides support for the following full-text features (the values can be changed in the GUI or via the <code>[[Commands#SET{{Command|SET]]</code> }} command):

* '''Stemming''': tokens are stemmed before being indexed (see <code>[[Options#STEMMINGoption: {{Option|STEMMING]]</code>}})* '''Case Sensitive''': tokens are indexed in case-sensitive mode (see <code>[[Options#CASESENSoption: {{Option|CASESENS]]</code>}})* '''Diacritics''': diacritics are indexed as well (see <code>[[Options#DIACRITICSoption: {{Option|DIACRITICS]]</code>}})* '''Stopword List''': a stop word list can be defined to reduce the number of indexed tokens (see <code>[[Options#STOPWORDSoption: {{Option|STOPWORDS]]</code>}})* '''Language''': see [[Full-Text#Languages|Languages]] for more details (see <code>[[Options#LANGUAGEoption: {{Option|LANGUAGE]]</code>}})

<pre classsyntaxhighlight lang="brush:xml">

</presyntaxhighlight>

Text nodes can be directly requested from the index via the XQuery function [[{{Function|Full-Text Module#ft:search|ft:search]]}}. The index contents can be accessed with [[{{Function|Full-Text Module#ft:tokens|ft:tokens]]}}.

{{Mark|Updated with Version 8.4:}} {{Code|TOKENINCLUDE}} option added Value indexing can be restricted to specific elements and attributes. The nodes to be indexed can be restricted via the <code>[[Options#TEXTINCLUDE{{Option|TEXTINCLUDE]]</code>}}, <code>[[Options#ATTRINCLUDE{{Option|ATTRINCLUDE]]</code>}}, <code>[[Options#TOKENINCLUDE{{Option|TOKENINCLUDE]]</code> }} and <code>[[Options#FTINCLUDE{{Option|FTINCLUDE]]</code> }} options. The options take a list of name patterns, which are separated by commas. The following name patterns are supported:

The options can either be specified via the <code>[[Commands#SET{{Command|SET]]</code> }} command or via XQuery. With the following operations, an attribute index is created for all {{Code|id}} and {{Code|name}} attributes:

<pre classsyntaxhighlight lang="brush:xml">

</presyntaxhighlight>

<pre classsyntaxhighlight lang="brush:xquery">

</presyntaxhighlight> With {{Command|CREATE INDEX}} and {{Function|Database|db:optimize}}, new selective indexing options will ba applied to an existing database. ==Enforce Rewritings==

In various cases, existing index structures will not be utilized by the query optimizer. This is usually the case if the name of the database is not a static string (e.g., because it is bound to a variable or passed on as argument of a function call). Furthermore, several candidates for index rewritings may exist, and the query optimizer may decide for a rewriting that turns out to be suboptimal. With <code>the {{Option|ENFORCEINDEX}} option, certain index rewritings can be enforced. While the option can be globally enabled, it is usually better to supply it as [[CommandsXQuery Extensions#CREATE INDEXPragmas|CREATE INDEXPragma]]. Two examples: * In the query below, 10 databases will be addressed. If it is known in advance that these databases contain an up-to-date text index, the index rewriting can be enforced as follows: </codesyntaxhighlight lang="xquery"> and [[Database Module(# db:enforceindex #) { for $n in 1 to 10 let $db:optimize= 'persons' || $n return db:open($db)//person[name/text() = 'John']}</syntaxhighlight> * The following query contains two predicates that may both be rewritten for index access. If the automatically chosen rewriting is known not to be optimal, another index rewriting can enforced by surrounding the specific expression with the pragma: <syntaxhighlight lang="xquery">db:open('factbook')//country [(# db:optimizeenforceindex #) { @population > '10000000' and @population < '10999999' }] [religions/text() = 'Protestant']</syntaxhighlight> The option can also be assigned to predicates with dynamic values. In the following example the comparison of the first comparison will be rewritten for index access. Without the pragma expression, the second comparison is preferred and chosen for the rewriting, because the statically known string allows for an exact cost estimation: <syntaxhighlight lang="xquery">for $name in ('Germany', 'Italy')for $country in db:open('factbook')//countrywhere (# db:enforceindex #) { $country/name = $name }where $country/religions/text() = 'Protestant'return $country</syntaxhighlight> Please note that: * The option should only be enabled if the addressed databases exist, have all required index structures and are up-to-date (otherwise, new selective you will be given an error message).* If you address the full-text index, and if you use non-default indexing options , you will ba applied have to specify them in your query (via {{Code|using stemming}}, {{Code|using language 'de'}}, etc).* If you have more than one enforce pragma in a single path expression, only the first will be considered.* In general, there are always expressions that cannot be rewritten for index access. If you enforce rewritings, you will have no guarantee that an existing databaseindex will be used.

<pre classsyntaxhighlight lang="brush:xquery">

</presyntaxhighlight>

<pre classsyntaxhighlight lang="brush:xquery">

</presyntaxhighlight>

With some extra effort, and if {{Option|UPDINDEX}} is set enabled for both your original and your index database (see below), your index database will support updates as well (try it, it’s fun!).

<presyntaxhighlight>

</presyntaxhighlight>

Updates in BaseX Generally, update operations are generally very fastin BaseX. By default, because the index structures will be invalidated by updates. As ; as a result, subsequent queries that benefit from index structures may be executed more slowly than before the updateslow down after updates. There are different alternatives to cope with this:

* After the execution of one or more update operations, the {{Command|OPTIMIZE}} command or the [[{{Function|Database Module#db:optimize|db:optimize]] }} function can be called to rebuild the index structures.* The {{Option|UPDINDEX}} option can be activated before creating or optimizing the database. As a result, the text, attribute and token indexes will be incrementally updated after each database update. Please note that incremental updates are not available for the full-text index and database statistics. This is also explains why the up-to-date UPTODATE flag, which is e.g. displayed via {{Command|INFO DB}} or [[Database_Module#db:info{{Function|Database|db:info]]}}, will be set to {{Code|false}} until the database will be optimized again(various optimizations won’t be triggered. For example, count(//item) can be extremely fast if all meta data is up-to-date.* The {{Option|AUTOOPTIMIZE}} option can be enabled before creating or optimizing the database. All outdated index structures and statistics will then be recreated after each database update. This option should only be used done for small and medium-sized databases.* Both options can be used side by side: {{Option|UPDINDEX}} will take care that the value index structures will be updated as part of the actual update operation. {{Option|AUTOOPTIMIZE}} will update the remaining data structures (full-text index, database statistics).