Bureaucrats, editor, reviewer, Administrators
13,550
edits
Changes
Jump to navigation
Jump to search
The Since {{Version|8.6}}, the distinct values are also stored for elements and attributes of numeric type. Various queries will be evaluated much faster if an up-to-date path index is applied to rewrite descendant available (as can be observed when opening the [[GUI#Visualizations|Info View]]): * Descendant steps will be rewritten to multiple child steps. Child steps can be are evaluated faster, as fewer nodes have to be accessedtraversed:
{{Mark|Introduced with Version 8.4:}}
{{Mark|Updated with Version 8.4:}} {{Code|TOKENINCLUDE}} option added Value indexing can be restricted to specific elements and attributes. The nodes to be indexed can be restricted via the <code>[[Options#TEXTINCLUDE{{Option|TEXTINCLUDE]]</code>}}, <code>[[Options#ATTRINCLUDE{{Option|ATTRINCLUDE]]</code>}}, <code>[[Options#TOKENINCLUDE{{Option|TOKENINCLUDE]]</code> }} and <code>[[Options#FTINCLUDE{{Option|FTINCLUDE]]</code> }} options. The options take a list of name patterns, which are separated by commas. The following name patterns are supported:
{{Mark|Updated with Version 8In various cases, existing index structures will not be utilized by the query optimizer. This is usually the case if the name of the database is not a static string (e.g., because it is bound to a variable or passed on as argument of a function call). Furthermore, several candidates for index rewritings may exist, and the query optimizer may decide for a rewriting that turns out to be suboptimal.4:}}
If main memory runs out while creating a value index, the current index structures will be partially written to disk and eventually merged. If the memory heuristics fail for some reason (i.eWith {{Version|9.0}}, because multiple index operations run at the same time, or because the applied JVM does not support explicit garbage collections), a fixed index split sizes may be chosen via the {{Option|SPLITSIZEENFORCEINDEX}} optionwas introduced, which allows you to enforce certain index rewritings. While the option can be globally enabled, it is usually better to supply it as [[XQuery Extensions#Pragmas|Pragma]].Two examples:
* The vertical bar <codepre class="brush:xquery">|<db:open('factbook')//codecountry [(# db:enforceindex #) { @population > indicates that a partial index structure was written to disk.'10000000' and @population < '10999999'* The mean value of the recommendations can be assigned to the {{Option|SPLITSIZE} } option. Please note that the recommendation is only a vague proposal, so try different values if you get main-of-memory errors or indexing gets too slow. Greater values will require more main memory.]* In the example, the full- [religions/text index was split 12 times. 116 million tokens were indexed, processing time was 2,5 minutes, and final main memory consumption (after writing the index to disk) was 76 MB. A good value for the split size option could be {{Code|15}}.= 'Protestant']</pre>
=Updates=Please note that:
Updates in BaseX are generally very fast, because the index structures will be invalidated by updates. As a result, subsequent queries may be executed more slowly than before the update. There are different alternatives to cope with this: * After the execution of one or more update operations, the {{Command|OPTIMIZE}} command or the [[Database Module#db:optimize|db:optimize]] function can be called to rebuild the index structures.* The {{Option|UPDINDEX}} option can should only be activated before creating or optimizing enabled if the database. As a resultaddressed databases exist, the text, attribute have all required index structures and token indexes will be incrementally updated after each database update. Please note that incremental updates are not available for the full-text index and database statistics. This is also explains why the up-to-date flag, which is e.g. displayed via {{Command|INFO DB}} or [[Database_Module#db:info|db:info]](otherwise, you will be set to {{Code|false}} until the database will be optimized againgiven an error message).* The {{Option|AUTOOPTIMIZE}} option can enforcement of index rewritings gives you no guarantee that an index will be enabled before creating or optimizing the databaseused. All outdated There are many expressions that cannot be rewritten for index structures and statistics will then be recreated after each database updateaccess. This option should * If you have more than one enforce pragma in a single path expression, only the first will be used for small and medium-sized databasesconsidered.
→Enforce Rewritings
==Name Index==
The name index contains all element and attribute references to the names of a database, all elements and the fixed-size index ids are stored attributes in the main database table. If a database is updated, new names are automatically added. Furthermore, the index is enriched with It contains some basic statistical information, such as the distinct (categorical) or minimum and maximum values number of its elements and attributes. The maximum number occurrence of categories to store per a name can be changed via [[Options#MAXCATS|MAXCATS]]. The index are discarded after [[#Updates|database updates]].
The name index is e.g. applied to pre-evaluate discard location steps that will never yield results:
<pre class="brush:xquery">
The contents of the name indexes can be directly accessed with the XQuery functions [[Index Module#index:element-names|index:element-names]] and [[Index Module#index:attribute-names|index:attribute-names]].
If a database is updated, new names will be added incrementally, but the statistical information will get out-dated.
==Path Index==
The path index (which is also called ''path summary'' or ''data guide'') stores all distinct paths of the documents in the database. It contains similar additional statistical information , such as the name indexnumber of occurrence of a path, its distinct string values, and the minimum/maximum of numeric values. The index are discarded after [[#Updatesmaximum number of distinct values to store per name can be changed via {{Option|database updates]]MAXCATS}}.
<pre class="brush:xquery">
</pre>
* The paths statistics are e.g. used to pre-evaluate the {{Code|fn:count}} functionwill be pre-evaluated by looking up the number in the index: <pre class="brush:xquery"> count(doc('factbook')//country)</pre> * The distinct values of elements or attributes can be looked up in the index as well:
<pre class="brush:xquery">
distinct-values(db: will be rewritten and pre-evaluated by the path index :)count( docopen('factbook')//country religions)
</pre>
The contents of the path index can be directly accessed with the XQuery function [[Index Module#index:facets|index:facets]].
If a database is updated, the statistics in the path index will be invalidated.
==Document Index==
Value indexes can be created and dropped by the user. Four types of values indexes are available: a text and attribute index, and an optional token and full-text index. By default, the text and attribute index will automatically be created.
In the GUI, index structures can be managed in the dialog windows for creating new databases or displaying the database properties. On command-line, the commands <code>[[Commands#CREATE INDEX{{Command|CREATE INDEX]]</code> }} and <code>[[Commands#DROP INDEX{{Command|DROP INDEX]]</code> }} are used to create and drop index structures. With <code>[[Commands#INFO INDEX{{Command|INFO INDEX]]</code>}}, you get some insight into the contents of an index structure, and <code>[[Commands#SET{{Command|SET]]</code> }} allows you to change the index defaults for new databases:
* <code>OPEN factbook; CREATE INDEX fulltext</code>: Open database; create full-text index
* <code>SET ATTRINDEX true; SET ATTRINCLUDE id name; CREATE DB factbook.xml</code>: Enable attribute index; only index 'id' and 'name' attributes; create database
With XQuery, index structures can be created and dropped via [[{{Function|Database Module#db:optimize|db:optimize]]}}:
<pre class="brush:xquery">
</pre>
Matching text nodes can be directly requested from the index with the XQuery function [[{{Function|Database Module#db:text|db:text]]}}. The index contents can be accessed via [[{{Function|Index Module#index:texts|index:texts]]text}}.
The <code>[[Options#UPDINDEX{{Option|UPDINDEX]]</code> }} option can be activated to keep this index up-to-date., for example: <pre class="brush:xquery">db:optimize( 'mydb', true(), map { 'updindex':true(), 'textindex': true(), 'textinclude':'id' })</pre>
===Range Queries===
</pre>
Text nodes can directly be retrieved from the index via the XQuery function [[{{Function|Database Module#db:text-range|db:text-range]]}}.
Please note that the current index structures do not support queries for numbers and dates.
</pre>
Attribute nodes can directly be retrieved from the index with the XQuery functions [[{{Function|Database Module#db:attribute|db:attribute]] }} and [[{{Function|Database Module#db:attribute-range|db:attribute-range]]}}. The index contents can be accessed with [[{{Function|Index Module#index:attributes|index:attributes]]}}.
The [[Options#UPDINDEX{{Option|UPDINDEX]] }} option can be activated to keep this index up-to-date.
==Token Index==
In many XML dialects, such as HTML or DITA, multiple tokens are stored in attribute values. The token index can be created to speed up the retrieval of these tokens. The XQuery functions {{Code|fn:contains-token}}, {{Code|fn:tokenize}} and {{Code|fn:idref}} are rewritten for index access whenever possible. If a token index exists, it will e.g. be utilized for the following queries:
</pre>
Attributes with tokens can be directly retrieved from the index with the XQuery function [[{{Function|Database Module#db:token|db:token]]}}. The index contents can be accessed with [[{{Function|Index Module#index:tokens|index:tokens]]}}.
==Full-Text Index==
</pre>
The index provides support for the following full-text features (the values can be changed in the GUI or via the <code>[[Commands#SET{{Command|SET]]</code> }} command):
* '''Stemming''': tokens are stemmed before being indexed (see <code>[[Options#STEMMINGoption: {{Option|STEMMING]]</code>}})* '''Case Sensitive''': tokens are indexed in case-sensitive mode (see <code>[[Options#CASESENSoption: {{Option|CASESENS]]</code>}})* '''Diacritics''': diacritics are indexed as well (see <code>[[Options#DIACRITICSoption: {{Option|DIACRITICS]]</code>}})* '''Stopword List''': a stop word list can be defined to reduce the number of indexed tokens (see <code>[[Options#STOPWORDSoption: {{Option|STOPWORDS]]</code>}})* '''Language''': see [[Full-Text#Languages|Languages]] for more details (see <code>[[Options#LANGUAGEoption: {{Option|LANGUAGE]]</code>}})
The options that have been used for creating the full-text index will also be applied to the optimized full-text queries. However, the defaults can be overwritten if you supply options in your query. For example, if words were stemmed in the index, and if the query can be rewritten for index access, the query terms will be stemmed as well, unless stemming is not explicitly disabled. This is demonstrated in the following [[Commands#Command_Scripts|Command Script]]:
</pre>
Text nodes can be directly requested from the index via the XQuery function [[{{Function|Full-Text Module#ft:search|ft:search]]}}. The index contents can be accessed with [[{{Function|Full-Text Module#ft:tokens|ft:tokens]]}}.
==Selective Indexing==
* <code>*</code>: all names
* <code>Q{uri}name</code>: elements or attributes called <code>name</code> in the <code>uri</code> namespace
The options can either be specified via the <code>[[Commands#SET{{Command|SET]]</code> }} command or via XQuery. With the following operations, an attribute index is created for all {{Code|id}} and {{Code|name}} attributes:
; Commands
</pre>
With <code>[[Commands#CREATE INDEX{{Command|CREATE INDEX]]</code> }} and [[{{Function|Database Module#db:optimize|db:optimize]]}}, new selective indexing options will ba applied to an existing database.
=Performance=Enforce Rewritings==
* In the query below, 10 databases will be addressed. If {{Option|DEBUG}} it is enabled, the commandknown in advance that these databases contain an up-line output might help you to find a good split size. The following example shows the output for creating a database for an XMark document with 1 GB-date text index, and with 128 MB assigned to the JVMindex rewriting can be enforced as follows:
<pre>> basex -d -cclass="SET FTINDEX ON; SET TOKENINDEX ON; CREATE DB xmark 1gb.xmlbrush:xquery">Creating Database................................... 76559.99 ms (29001 KB# db:enforceindex #){Indexing Text... for $n in 1 to 10.... let $db := 'persons' |...|...|.....|. 9.81 M operations, 18576.92 ms (13523 KB). Recommended SPLITSIZE$n return db: 20.Indexing Attribute Values............|....... 3.82 M operations, 7151.77 ms open(6435 KB$db). Recommended SPLITSIZE: 20.Indexing Tokens..........|..|.....|.. 3.82 M operations, 9636.73 ms //person[name/text(10809 KB). Recommended SPLITSIZE: 10.= 'John']Indexing Full-Text.....|.|.|.|...|...|..|.|..| 116.33 M operations, 138740.94 ms (106 MB). Recommended SPLITSIZE: 12.}
</pre>
* The output following query contains two predicates that may both be rewritten for index access. If the automatically chosen rewriting is known not to be optimal, another index rewriting can be interpreted as followsenforced by surrounding the specific expression with the pragma:
=Custom Index Structures=
With XQuery, it is very comparatively easy to create your own, custom index structures. The following query demonstrate how you can create a {{Code|factbook-index}} database, which contains all texts of the original database in lower case:
<pre class="brush:xquery">
</pre>
With some extra effort, and if {{Option|UPDINDEX}} is set enabled for both your original and your index database(see below), your index database will support updates as well (try it, it’s fun!). =Performance= If main memory runs out while creating a value index, the current index structures will be partially written to disk and eventually merged. If the memory heuristics fail for some reason (i.e., because multiple index operations run at the same time, or because the applied JVM does not support explicit garbage collections), a fixed index split sizes may be chosen via the {{Option|SPLITSIZE}} option. If {{Option|DEBUG}} is enabled, the command-line output might help you to find a good split size. The following example shows the output for creating a database for an XMark document with 1 GB, and with 128 MB assigned to the JVM: <pre>> basex -d -c"SET FTINDEX ON; SET TOKENINDEX ON; CREATE DB xmark 1gb.xml"Creating Database................................... 76559.99 ms (29001 KB)Indexing Text.......|...|...|.....|. 9.81 M operations, 18576.92 ms (13523 KB). Recommended SPLITSIZE: 20.Indexing Attribute Values............|....... 3.82 M operations, 7151.77 ms (6435 KB). Recommended SPLITSIZE: 20.Indexing Tokens..........|..|.....|.. 3.82 M operations, 9636.73 ms (10809 KB). Recommended SPLITSIZE: 10.Indexing Full-Text.....|.|.|.|...|...|..|.|..| 116.33 M operations, 138740.94 ms (106 MB). Recommended SPLITSIZE: 12.</pre> The output can be interpreted as follows: * The vertical bar <code>|</code> indicates that a partial index structure was written to disk.* The mean value of the recommendations can be assigned to the {{Option|SPLITSIZE}} option. Please note that the recommendation is only a vague proposal, so try different values if you get main-of-memory errors or indexing gets too slow. Greater values will require more main memory.* In the example, the full-text index was split 12 times. 116 million tokens were indexed, processing time was 2,5 minutes, and final main memory consumption (after writing the index to disk) was 76 MB. A good value for the split size option could be {{Code|15}}. =Updates= Generally, update operations are very fast in BaseX. By default, the index structures will be invalidated by updates; as a result, queries that benefit from index structures may slow down after updates. There are different alternatives to cope with this: * After the execution of one or more update operations, the {{Command|OPTIMIZE}} command or the {{Function|Database|db:optimize}} function can be called to rebuild the index structures.* The {{Option|UPDINDEX}} option can be activated before creating or optimizing the database. As a result, the text, attribute and token indexes will be incrementally updated after each database update. Please note that incremental updates are not available for the full-text index and database statistics. This is also explains why the UPTODATE flag, which is e.g. displayed via {{Command|INFO DB}} or {{Function|Database|db:info}}, will be set to {{Code|false}} until the database will be optimized again (various optimizations won’t be triggered. For example, count(//item) can be extremely fast if all meta data is up-to-date.* The {{Option|AUTOOPTIMIZE}} option can be enabled before creating or optimizing the database. All outdated index structures and statistics will then be recreated after each database update. This option should only be done for small and medium-sized databases.* Both options can be used side by side: {{Option|UPDINDEX}} will take care that the value index structures will be updated as part of the actual update operation. {{Option|AUTOOPTIMIZE}} will update the remaining data structures (full-text index, database statistics).
=Changelog=
;Version 9.0
* Added: [[#Enforce Rewritings|Enforce Rewritings]]
;Version 8.4
* Updated: [[#Name Index|Name Index]], [[#Path Index|Path Index]]
;Version 8.4
