administrator, editor
33
edits
Changes
Jump to navigation
Jump to search
=Version: 7This article is part of the [[Advanced User's Guide]]. It presents some low-level details on how data is stored in the database files.0=
==Data Types==* <code>Num</code>: compressed integer (1-5 bytes)* <code>Token</code>: length (<code>Num</code>) and bytes of UTF8 byte representation* <code>double</code>: number, stored as token* <code>boolean</code>: boolean (1 byte, <code>00</code> or <code>01</code>)
| valign='top' | '''2. Main memory indexesb) Path Index'''| Pairs of key/value tokens1. Flag for path definition ({{Type|Boolean}}, suffixed by empty key:always {{Code|true}}; legacy)<br />• 2. PathNode:<codebr/>TAGS2.1. Name reference ({{Type|Num}})<br/code> → Tag Index2.2. Node kind ({{Type|Num}})<br />• <code>ATTS2.3. Number of occurrences ({{Type|Num}})<br/code> → Attribute Index2.4. Number of children ({{Type|Num}})<br />&bull2.5. {{Type|Double}}; legacy, can be reused or discarded<code>PATH<br/code> 2.6. Recursive generation of child nodes (→ Path Index<br 2)| [https://github.com/BaseXdb/basex/blob/master/basex-core/src/main/java/org/basex/index/>• <code>NS<path/code> → NamespacesPathSummary.java PathSummary()]<br />• <code>DOCS</code> → Document Index| valign='top' | [https://github.com/BaseXdb/basex/blob/master/basex-core/src/main/java/org/basex/dataindex/path/DiskDataPathNode.java DiskDataPathNode()]
| valign='top' | '''2.1. Name Indexc) Namespaces'''<br/>Element/attribute names| 1. Token set, enriched with statistical information:storing prefixes ({{Type|TokenSet}})<br />12. Token set: key array (<code>Tokens</code>), next/bucket/size arrays storing URIs (<code>Nums</code>{{Type|TokenSet}})<br />23. Content kind (<code>Num</code>)NSNode:<br />23.1. Number: min/max pre value (<code>Doubles</code>{{Type|Num}})<br />23.2. Category: number of entries (<code>Num<References to prefix/code>), entries URI pairs (<code>Tokens</code>{{Type|Nums}})<br />23.3. Number of entries children (<code>{{Type|Num</code>}})<br />23.4. Leaf flag Recursive generation of child nodes (<code>Boolean</code>)<br />2.5. Maximum text length (<code>Double</code>; legacy, could be <code>Num</code>→ 3)| valign='top' | [https://github.com/BaseXdb/basex/blob/master/basex-core/src/main/java/org/basex/indexdata/NamesNamespaces.java NamesNamespaces()]<br/>[https://github.com/BaseXdb/basex/blob/master/src/main/java/org/basex/util/hash/TokenSet.java TokenSet.read()]<br/>[https://github.com/BaseXdb/basex/blob/master-core/src/main/java/org/basex/indexdata/StatsKeyNSNode.java StatsKeyNSNode()]
| valign='top' | '''2.2. Path d) Document Index'''| 1. Flag for path definition (<code>Boolean</code>, always <code>true</code>; legacy)<br/>2. PathNode:<br/>2.1. Name reference (<code>Num</code>)<br/>2.2. Node kind (<code>Num</code>)<br/>2.3. Number of occurrences (<code>Num</code>)<br/>2.4. Number Array of children (<code>Num</code>)<br/>2.5. <code>Double</code>; legacyintegers, can be reused or discarded<br/>2.6. Recursive generation of child nodes representing the distances between all document pre values (→ 2{{Type|Nums}})<code>| [https://github.com/BaseXdb/basex/blob/master/basex-core/src/main/java/org/basex/index/path/PathSummaryDocIndex.java PathSummaryDocIndex()]
Added information regarding pth.basex and idp.basex
The following data types are used for specifying the storage layout: {| class="wikitable" width=inf"100%"|-! Type! Description! Example (native → hex integers)|-| {{Type|Num}}| Compressed integer (1-5 bytes), specified in [https://github.com/BaseXdb/basex/blob/master/basex-core/src/main/java/org/basex/util/Num.java Num.java]| {{Code|15}} → {{Code|0F}}; {{Code|511}} → {{Code|41 FF}}<br/>|-| {{Type|Token}}| Length ({{Type|Num}}) and bytes of UTF8 byte representation| {{Code|Hello}} → {{Code|05 48 65 6c 6c 6f}}|-| {{Type|Double}}| Number, stored as token| {{Code|123}} → {{Code|03 31 32 33}}|-| {{Type|Boolean}}| Boolean (1 byte, {{Code|00}} or {{Code|01}})| {{Code|true}} → {{Code|01}}|-| {{Type|Nums}}, {{Type|Tokens}}, {{Type|Doubles}}| Arrays of values, introduced with the number of entries| {{Code|1,2}} → {{Code|02 01 31 01 32}}|-| {{Type|TokenSet}}| Key array ({{Type|Tokens}}), next/bucket/size arrays (3x {{Type|Nums}})||} =Database Files= The following tables illustrate the layout of the BaseX database files. All files are suffixed with {{Code|.basex}}. ==Meta Data, Name/Path/Doc Indexes: {{Code|inf}}==
{| class="wikitable" width="100%"
! Method
|-
| valign='top' | '''Disk 1. Meta Data'''| Database meta information1. Key/value pairs, in no particular order ({{Type|Token}}/{{Type|Token}}):<br/> • Examples: {{Code|FNAME}}, {{Code|TIME}}, {{Code|SIZE}}, ...<br /> • {{Code|PERM}} → Number of users ({{Type|Num}}), and name/password/permission values for each user ({{Type|Token}}/{{Type|Token}}/{{Type|Num}})<br/>2. Empty key as finalizer| valign=[https://github.com/BaseXdb/basex/blob/master/basex-core/src/main/java/org/basex/data/DiskData.java DiskData()]<br/>[https://github.com/BaseXdb/basex/blob/master/basex-core/src/main/java/org/basex/data/MetaData.java MetaData()]<br/>[https://github.com/BaseXdb/basex/blob/master/basex-core/src/main/java/org/basex/core/Users.java Users()]|-| '''top2. Main memory indexes' ''| 1. Key/value pairs, in no particular order ({{Type|Token}}/{{Type|Token}}):<br /> • {{Code|TAGS}} → Element Name Index<br /> • {{Code|ATTS}} → Attribute Name Index<br /> • {{Code|PATH}} → Path Index<br /> • {{Code|NS}} → Namespaces<br /> • {{Code|DOCS}} → Document Index<br/>2. Empty key as finalizer| [https://github.com/BaseXdb/basex/blob/master/basex-core/src/main/java/org/basex/data/DiskData.java DiskData()]
|-
| valign='top' | '2 a) Name Index'''<br/>Element/attribute names| 1. Token set, storing all names ({{Type|TokenSet}})<br />2. One StatsKey instance per entry:<br/>2.1. Meta Data'''Content kind ({{Type|Num}}):<br />2.1.1. Number: min/max ({{Type| Pairs Doubles}})<br />2.1.2. Category: number of key/value tokensentries ({{Type|Num}}), suffixed by empty key:entries ({{Type|Tokens}})<br />• 2.2. Number of entries ({{Type|Num}})<codebr />PERM2.3. Leaf flag ({{Type|Boolean}})<br /code> → User Permissions2.4. Maximum text length ({{Type|Double}}; legacy, could be {{Type|Num}})| valign='top' | [https://github.com/BaseXdb/basex/blob/master/basex-core/src/main/java/org/basex/index/Names.java Names()]<br/>[https://github.com/BaseXdb/basex/blob/master/basex-core/src/main/java/org/basex/datautil/hash/MetaDataTokenSet.java MetaDataTokenSet.read()]<br/>[https://github.com/BaseXdb/basex/blob/master/basex-core/src/main/java/org/basex/coreindex/UsersStatsKey.java Users.readStatsKey()]
|-
|-
|-
|}
==Node Table: {{Code|tbl}}, {{Code|tbli}}==
* {{Code|tbl}}: Main database table, stored in blocks.
* {{Code|tbli}}: Database directory, organizing the database blocks.
Some more information on the [[Node Storage|node storage]] is available.
==Texts: {{Code|txt}}, {{Code|atv}}==
* {{Code|txt}}: Heap file for text values (document names, string values of texts, comments and processing instructions)
* {{Code|atv}}: Heap file for attribute values.
==Value Indexes: {{Code|txtl}}, {{Code|txtr}}, {{Code|atvl}}, {{Code|atvr}}==
'''Text Index:'''
* {{Code|txtl}}: Heap file with ID lists.
* {{Code|txtr}}: Index file with references to ID lists.
The '''Attribute Index''' is contained in the files {{Code|atvl}} and {{Code|atvr}}, the '''Token Index''' in {{Code|tokl}} and {{Code|tokr}}. All have the same layout.
For a more detailed discussion and examples of these file formats please see [[Index File Structure]].
==Document Path Index: {{Code|pth}}==
Provides an index of all the document paths in the database. For databases with a large number of paths this file can be quite large so it is only generated the first time a function requesting a path lookup is run. For databases where path lookups are never used this file will not exist.
'''Note:''' On Windows/Mac systems this file is case insensitive (all paths are lower case). On UNIX-like systems this file is case sensitive. The behaviour of path look ups will vary between systems. Copying this file between system types may lead to unexpected behaviour.
==ID/Pre Mapping: {{Code|idp}}==
This file is only created if incremental indexing (UPDINDEX) is enabled for a database. It is used to provide a quick look up of the pre value for a database node id.
==Full-Text Fuzzy Index: {{Code|ftxx}}, {{Code|ftxy}}, {{Code|ftxz}}==
...may soon be reimplemented.
