BaseX 13

This article presents the highlights of BaseX 13. Changes that may require adjustments when migrating existing projects from BaseX 12 are summarized in Upgrading to BaseX 13, and all additions and updates are listed in the Changelog.

BaseX 13 keeps all indexes up to date, implements the latest drafts of XQuery 4.0, and comes with a redesigned editor and DBA.

Storage

• Up-to-date indexes: All index structures, including the full-text index, are kept up to date after updates (UPDINDEX is now enabled by default).
• Faster maintenance: Value indexes have a segmented layout, the node table is read in parallel, and OPTIMIZE only compacts updated indexes and skips unchanged ones.
• More index access: Integer comparisons are rewritten for index access, and inputs larger than main memory are moved to disk.
• Concurrency: Updating queries only acquire read locks for databases that they do not update, so more queries can run concurrently (see Transaction Management).
• Analysis: db:counts counts distinct values for faceted search, and db:inspect checks storage structures for inconsistencies.

XQuery 4.0

• JNodes: JSON and other map and array structures can be navigated with XPath, e.g. json-doc(…)//city.
• Destructuring and records: Destructuring assigns the components of sequences, arrays and maps to variables, records are updated with but with, and the trace clause emits diagnostic output per tuple.
• New functions: More standard functions of XQuery 4.0 are available, e.g. fn:map-to-element for converting JSON to XML, fn:xsd-validator for schema validation, fn:scan, fn:drop-while, fn:pad-string and fn:location.
• Conveniences: The local prefix of function declarations can be omitted, map constructors accept arbitrary expressions, QName literals can have prefixes, and there are new axes, node comparisons and the Unicode operators × and ÷.

In the following query, the entries of a map are bound to variables, and the names of all cities are selected with a path expression:

let $country := {
  'capital': 'Paris',
  'cities': [ { 'name': 'Lyon' }, { 'name': 'Marseille' } ]
}
let ${ $capital } := $country
return ($capital, data($country//name))

Full-Text

• Updatable index: The full-text index is kept up to date after updates, and fuzzy queries are faster.
• Mixed content: Elements with mixed content can be indexed, and their matches are highlighted.
• Thesauri: Thesauri can be stored and indexed in databases, and SKOS thesauri are supported.
• Functions: ft:search and ft:contains accept stop words, ft:tokens looks up index tokens with a bounded number of errors, and ft:languages lists the available stemmers and tokenizers.

Data and Formats

• HTTP Client: The new HTTP Client has a map-based interface (http:get, http:post, …) with typed options, record results and cookie support.
• Formats: JSON Lines, the W3 mapping of JSON, CSV comments, and TAR, Zstandard and XZ archives are supported.
• Serialization: Results can be serialized as canonical XML and as XQuery expressions, and the CSV output method is now part of the standard.
• Resources: db:add accepts multiple inputs, and binary and value resources can be added and stored as initial contents of db:create.
• Modules: Fuzzy string matching (string:closest, string:levenshtein-distance, string:token-set-ratio, …), new Binary Functions and the Geo Functions with GML 3 support are available.

Editor

• Speed: The GUI renders much faster.
• Coding aids: The editor offers code completion, function signatures, marked occurrences, multiple cursors, go to declaration, code formatting and Markdown highlighting.
• Projects: Files can be searched and replaced project-wide, with backup and undo (Project View), and files changed on disk are reloaded.

Web

• DBA and WebDAV: The DBA has been redesigned, and WebDAV is back as an XQuery application.
• RESTXQ: Applications can be deployed as ZIP archives and installed by name from the Repository.
• QUERY method: The HTTP QUERY method is supported by REST and RESTXQ.
• WebSockets: WebSockets support path templates, permission checks and sub-protocols.
• Deployment: Official Docker images for linux/amd64 and linux/arm64 are published in the GitHub Container Registry (ghcr.io/basexdb/basex) and rebuilt weekly, and BaseX runs in Jakarta EE 10 servlet containers.

Evaluation and Jobs

• Function items: xquery:eval, job:eval, job:execute and ws:eval accept function items instead of query strings: no strings need to be assembled, and no code can be injected.
• Scheduling: Jobs can be scheduled with the Cron Syntax and restricted with timeout, memory and permission options.
• Parallelism: xquery:for-each and xquery:reduce distribute iterations and aggregations over all processor cores, and xquery:fork-any returns the first available result.
• Caching: Memoization with %basex:memo caches function results, %basex:eval evaluates functions in other queries, and Cache Functions keep values across queries.

Security

Security has been a major focus of this release: authentication, permissions and the processing of external input were reviewed and hardened throughout.

• Hardening: External entities and XIncludes are only resolved for administrators, archive entries cannot escape their target paths, and session cookies are protected against cross-site requests (SameSite).
• Passwords: Authentication is salted, and passwords can be stored as PBKDF2 hashes (see User Management).
• Untrusted code: Code can be untrusted, and external resources are only fetched if TRUSTEXTERNAL is enabled.
• Permissions: Permissions were revised: external resources require admin permission, and permissions are also checked for opened databases and WebSocket handshakes.
• Logins: Pending and repeated logins are limited (see Login Protection).
• Cryptography: crypto:encrypt supports AES-GCM and RSA, crypto:generate-signature supports RSA and DSA with SHA-256 and SHA-512, and crypto:pbkdf2 derives keys from passwords; outdated algorithms (MD5, DES) were removed.

Performance

• Joins: If a sequence is filtered repeatedly with equality or range comparisons, it is indexed on the fly, and each comparison becomes a lookup (see Comparisons).
• Paths: Paths in loops are cached as long as their root and the variables they reference do not change.
• Memory: Maps, constructed elements and integer sequences are stored in compact representations and need much less main memory.
• Node construction: Enclosed nodes are no longer copied if the result is only serialized: e.g. a database can be wrapped in an element and written to a file without being duplicated in main memory (see Node Construction).