Main Page » Getting Started » BaseX 13

BaseX 13

This article presents the highlights of BaseX 13 and summarizes the changes that may require adjustments when migrating existing projects from BaseX 12. All additions and updates are listed in the Changelog.

Highlights

XQuery

Storage

  • All index structures, including the full-text index, are kept up to date after updates (UPDINDEX is now enabled by default).
  • Fuzzy full-text queries are faster, integer comparisons are rewritten for index access, and thesauri can be indexed.
  • Inputs larger than main memory are moved to disk.
  • db:inspect checks storage structures for inconsistencies.

Editor

  • The GUI renders much faster.
  • Code completion, function signatures, go to declaration and code formatting.
  • Project-wide search and replace with undo; files changed on disk are reloaded.

Web

  • The DBA has been redesigned, and WebDAV is back as an XQuery application.
  • RESTXQ applications can be deployed as ZIP archives.
  • WebSockets support path templates and permission checks.
  • BaseX runs in Jakarta EE 10 containers.

Security

  • Authentication is salted, and passwords can be stored as PBKDF2 hashes (see User Management).
  • Code can be untrusted, and external resources are only fetched if TRUSTEXTERNAL is enabled.
  • Permissions are also checked for opened databases and WebSocket handshakes.
  • Outdated algorithms (MD5, DES) were removed.

Critical Changes

The following changes can prevent BaseX from starting or clients from connecting, change results without raising an error, or affect many existing applications. Most other changes raise errors that point to the required adjustment.

  • Java: Java 21 or later is required (see Prerequisites).
  • Configuration: Existing .basex files are no longer rewritten, old values of PARALLEL and HTTPLOCAL remain effective, and a .basex file in an ancestor directory may now determine the home directory (see Configuration).
  • Server: basexhttp no longer starts a database server on port 1984 (see Database Server).
  • Clients: Clients of BaseX 8 to 12 and third-party bindings without the salted handshake cannot connect, users with digest hashes can no longer log in, and HTTP Digest authentication was removed (see Authentication).
  • Web application: A Jakarta EE 10 servlet container is required, and custom web.xml and jetty.xml files must be checked. RESTXQ may dispatch requests to different functions, and WebSocket errors are only sent to clients by %ws:error functions (see Web Server, RESTXQ and WebSockets).
  • Downgrades: Databases and values written by BaseX 13 may be rejected by BaseX 12 (see Compatibility).
  • Full-text: Terms with characters such as ß or æ are only found after the index has been rebuilt (see Full-Text Index).
  • Whitespace: STRIPWS preserves whitespace in mixed content (see Options).
  • Options: Boolean options of standard functions must be booleans: fn:serialize($data, { 'indent': 'no' }) raises an error (see Standard Functions).
  • External resources: DTDs, entities, XInclude documents and schemas are only fetched if TRUSTEXTERNAL is enabled, and modules loaded with fn:load-xquery-module are untrusted (see Standard Functions).
  • Results: ft:thesaurus traverses one level and omits the queried term, http:send-request returns lower-case header names and follows fewer redirects, fn:decode-from-uri and fn:parse-uri keep plus signs, file:delete ignores missing paths, quotes in unquoted CSV fields are kept, HTML is serialized as version 5.0, trace output is dropped for users without CREATE permission, and dates without timezone may shift by an hour (see Functions, Language and Serialization).

Prerequisites

BaseX 13 requires Java 21 or later (see Startup).

Configuration

The .basex configuration file is no longer rewritten when BaseX is started:

  • Entries of an existing file take precedence over the defaults. To adopt the new defaults of PARALLEL and HTTPLOCAL, remove the entries, or delete the file to have a new one created.
  • Removed options (e.g. FAIRLOCK, SPLITSIZE) are reported as unknown on every start until they are removed from the file.
  • Local options no longer need to be placed after the Local Options comment.
  • The ancestors of the current user directory are also searched for a .basex or .basexhome file, which may result in a different home directory (see Configuration).

Databases

Compatibility

Databases created with BaseX 12 can be opened with BaseX 13, including their index structures. A database created or updated with BaseX 13 remains readable by BaseX 12 if…

  1. it has not been updated since it was created,
  2. UPDINDEX is disabled (it is now enabled by default),
  3. it was optimized with OPTIMIZE or db:optimize, or
  4. the text, attribute, token and full-text indexes are all disabled.

Otherwise, BaseX 12 rejects it. Small databases (up to 100,000 nodes) remain readable after updates (see Updates).

BaseX 12 cannot read a database at all if…

  • Mixed content: the new FTMIXED option is enabled and a full-text index exists. Drop the index, or disable the option and optimize the database.
  • Namespaces: its namespace structure has more than 4,096 entries. Such structures are stored in a compact format.

XQuery values are stored in a new format. Value resources (see db:put-value) and stores written by BaseX 12 can be read by BaseX 13, and they are converted to the new format when they are written next. Values written by BaseX 13 are rejected by BaseX 12.

Full-Text Index

The token normalization of the full-text index was standardized: characters that denote multiple letters are expanded (ß → ss, æ → ae, …), and missing and incorrect mappings were fixed. Full-text indexes created with older versions remain usable, but terms with affected characters are only found after the index has been rebuilt, e.g. with CREATE INDEX or OPTIMIZE ALL.

Options

  • UPDINDEX: Enabled by default: value indexes of new databases are kept up-to-date after updates. Disable it to keep updated databases readable by BaseX 12 (see Compatibility).
  • STRIPWS: Only whitespace-only text nodes are discarded; whitespace in mixed content is preserved.
  • PARALLEL: The default was raised to 32.
  • MIXUPDATES was removed: use job:execute to run updating and non-updating queries side by side.
  • LSERROR was removed: use using fuzzy N errors or the errors option of the Full-Text Functions.
  • FAIRLOCK and SPLITSIZE were removed: non-fair locking is always used, and index structures are always split based on the available main memory.
  • TAILCALLS was removed: tail calls are always eliminated.
  • ADDCACHE was removed: large inputs are moved to disk automatically. Remove it from command scripts and database options.
  • FORCECREATE was removed: use db:create to create databases.
  • INTPARSE: Unresolvable entity references are replaced with U+FFFD instead of ?.
  • db:system, db:info, INFO: Section names are no longer localized.
  • TEXTINCLUDE, ATTRINCLUDE, TOKENINCLUDE, FTINCLUDE: Invalid name patterns were ignored before; now, index creation fails.

Server and Clients

Database Server

  • With the new default of HTTPLOCAL, the HTTP server no longer starts a Database Server instance. Start basexhttp with -L (see Command-Line Options) or disable the option if Clients need to connect on port 1984.
  • Servers are stopped via the network instead of a stop file, and only from the same host. Stop servers with the binaries of the same version.

Authentication

The login handshake of the Server Protocol has been changed from digest to salted authentication, and the unsalted digest (MD5) password algorithm was removed (see User Management):

  • Clients of versions 8 to 12 can no longer connect. Third-party bindings must implement the salted handshake; the bindings in the BaseX repository have been updated.
  • Clients of Version 13 cannot connect to older servers.
  • HTTP Digest authentication was removed: the Digest value of AUTHMETHOD is no longer supported, and digest entries in AUTHALGORITHMS are ignored. Use Basic authentication via HTTPS instead. Digest authentication of the HTTP Client is still supported.
  • digest hashes in users.xml are ignored and dropped when the file is written next.
  • The client-side fallback to cram-md5 authentication, the outdated clients and the XML:DB API were removed.
  • Permissions are also checked for databases that have already been opened.
  • CREATE permission is required to access external resources, including modules imported with at, stop-word and thesaurus files, serialization parameter documents, and paths or URLs passed to ADD, PUT, BINARY PUT and db:put-binary. RESTXQ modules are exempt.

Web Server

  • BaseX is based on the Jetty EE10 environment. It can be deployed in any Jakarta EE 10 servlet container (Servlet 6.0 and WebSocket 2.1: Apache Tomcat 10.1, Jetty 12, WildFly 27 or later). Containers based on the javax.servlet API are not supported (see Servlet Container).
  • Static resources are served by a servlet of BaseX; the directory is specified via the path parameter. If you use a custom web.xml file, compare it with the one of the distribution.
  • The HTTP port is specified via HTTPPORT or -h. jetty.xml is no longer part of the distribution: an existing file in WEB-INF is still used, and -h overrides its port. Remove it unless it contains custom settings.
  • WebDAV runs as a RESTXQ application. Locks are kept in main memory, and the ~webdav database is not used anymore.
  • Permission errors of REST queries are answered with 403 instead of 400.

RESTXQ

  • Server-side forwarding (the rest:forward element and web:forward) was dropped. Use job:execute to combine multiple steps in a single request.
  • Functions with method annotations take precedence over functions without, and HEAD requests are dispatched to GET functions first. Automatic OPTIONS and HEAD responses follow RFC 9110.
  • Requests that match no function are answered with a specific status code (e.g. 415 if the content type is missing), and permission errors with 403 instead of 500 (see Response).
  • Request bodies of type application/xquery are converted to strings.

WebSockets

  • The %ws:header-param annotation was removed: use the Request Functions instead, which can now be called in WebSocket handlers.
  • Without a %ws:error function, no error message is sent to the client.
  • The result of ws:eval is sent as a single message, and errors are reported to the client.

Language

The implementation follows the latest drafts of XQuery 4.0. The following language features were changed or dropped:

  • Record types: A record has a fixed set of fields. Optional fields (field?) and the wildcard (*) were removed, field access is strict, and subtyping is width-invariant.
  • Record declarations: Declared record types are nominative. Declarations with identical fields define distinct types.
  • Record constructors: Built-in record types (e.g. fn:division-record, fn:parsed-csv-structure-record) have no constructor functions anymore.
  • Methods: The %method annotation was replaced by the =?> operator.
  • The map and array filter (e.g. [ 1, 2, 3 ]?[. != 2]) was removed.
  • Node tests: node() matches XML nodes and JNodes, and xnode() matches XML nodes only. gnode() was removed: use node() instead, and replace node() with xnode() where JNodes must be excluded.
  • Casting: Casting from strings follows XSD 1.1; casts to xs:anyURI never fail.
  • The nondeterministic prefix for dynamic function calls was removed: calls of functions that are statically unknown are treated as nondeterministic (see XQuery Extensions).
  • Dates and times without timezone are adjusted with the implicit timezone of the query, a fixed offset, instead of the daylight-saving-aware offset of the system zone. Results may shift by an hour, e.g. in convert:dateTime-to-integer or SQL parameters.
  • order by: NaN precedes all numbers. Results only change with empty greatest.
  • Without ICU, fn:collation returns a UCA collation URI.
  • The BaseX-specific regular expression flags j, ! and ; were removed.

Functions

Renamed and Removed

The following functions were renamed or removed:

Description BaseX 13 BaseX 12
Insert separator between items fn:insert-separator fn:sequence-join, fn:intersperse
Parse CSV rows fn:parse-csv?rows fn:csv-to-arrays
Return function identity removed fn:function-identity
Return subsequence between matching items fn:drop-while, fn:take-while fn:subsequence-where
Return key-value pairs of a map map:entries (singleton maps) map:pairs
Build map from key-value pairs map:build map:of-pairs
Return keys of matching entries map:keys, map:filter map:keys-where
Functions: Higher-order functions removed Higher-Order Functions Module
Return intermediate results of a fold fn:scan (results wrapped in arrays) hof:scan-left
Fold non-empty sequence without seed fn:fold-left hof:fold-left1
Return greatest items fn:sort-by, fn:sort-with hof:top-k-by, hof:top-k-with
Pad string fn:pad-string string:format
Convert between strings and binary data Binary Functions convert:binary-to-string, convert:string-to-base64, convert:string-to-hex
Convert between durations and numbers fn:seconds, fn:duration-to-seconds (seconds instead of milliseconds) convert:integer-to-dayTime, convert:dayTime-to-integer
Forward request removed web:forward

Standard Functions

  • Boolean options must be supplied as booleans. For example, fn:serialize($data, { 'indent': 'no' }) raises an error; use { 'indent': false() } instead.
  • fn:doc, fn:parse-xml, fn:parse-xml-fragment: The strip-space option accepts all, none or conditional instead of a boolean. The entity-expansion-limit and allow-external-entities options were removed.
  • fn:doc, fn:doc-available, fn:parse-xml, fn:parse-xml-fragment: External DTDs and entities, XInclude documents and referenced schemas are only fetched if the trust-external option or TRUSTEXTERNAL is enabled.
  • fn:load-xquery-module: Loaded modules are untrusted by default and cannot access external resources. Enable the trusted option to grant them the permissions of the calling code.
  • fn:deep-equal: The false-on-error and normalize-space options were removed.
  • fn:replace: The captured groups are passed to replacement functions as a map instead of a sequence ($groups?1 instead of $groups[1]). The old form is still supported if the parameter is declared as xs:untypedAtomic*.
  • fn:decode-from-uri: Plus signs are no longer decoded as spaces. fn:parse-uri: Plus signs are only decoded as spaces in query parameters.
  • array:join: The $separator parameter was removed.
  • array:members, array:of-members: JNodes are used instead of value records.

Data Formats

  • CSV Functions: field-delimiter was renamed back to separator, and quotes in unquoted fields are adopted as literal characters. row-delimiter, skip-empty, strict-quoting and the xquery and w3-arrays formats (use w3) were removed.
  • JSON Functions: number-parser was replaced with number-format. The xquery and basic formats (use w3 and w3-xml) were removed.
  • HTML Functions: If no HTML parser is available, the input is wrapped in an html element.
  • Archive Functions: Duplicate entry names raise an error (before, the last entry was kept), and archive:create-from rejects multiple entries for gzip. Error codes are reported with the archive prefix (before: admin).

Databases and Full-Text

  • db:put-binary: $input is restricted to binary items and strings, which are interpreted as URIs.
  • db:put, db:put-binary, db:put-value, PUT, BINARY PUT: Only the resource at the specified path is replaced; resources below the path are kept. A binary and a value resource can no longer share a path. Storing a binary or value resource at the path of a directory raises an error. PUT and REST PUT requests with XML input replace binary resources with XML documents (before, the input was written into the binary resource).
  • Store Functions: All opened stores are kept in main memory, and a newly opened store no longer overwrites the default store.
  • ft:thesaurus, Thesaurus: By default, one level is traversed instead of all levels. Specify the levels option or a level range (e.g. at most 10 levels) to traverse more levels. The queried term is no longer returned.
  • ft:mark, ft:extract: The name of the marker element must be an NCName.

Files and Processes

  • file:read-text-lines: The $offset and $length parameters were removed.
  • file:delete: Non-existing paths are ignored.
  • proc:system, proc:execute: Output that is invalid in the chosen encoding is rejected unless the fallback option is enabled. String input is encoded with the specified encoding instead of UTF-8. proc:property and related functions require CREATE permission.

HTTP

  • http:send-request: Response header names are returned in lower case; binary response bodies of GET requests are returned as lazy items. Compressed responses are requested by default. Redirects to another origin are followed without credentials, and redirects from https to http are not followed. A BaseX user agent is sent by default. Bodies of type application/xquery are returned as strings. Error codes follow the EXPath specification: HC0003 became HC0004, HC0004 became HC0005, and timeouts raise HC0006.
  • web:response-header: The message attribute was removed.
  • web:error: $err:value contains the message instead of the status code.

Jobs and Evaluation

  • job:execute: If the calling query holds locks, the job may only read databases read-locked by it; otherwise, a job:deadlock error is raised. Move the database access of the calling query to a separate job, or use job:eval.
  • job:eval: Services require an id, and the memory, timeout and permission options are rejected for services. job:list-details returns the full job string.
  • xquery:eval: The permission option also applies to returned function items.
  • Repository: All functions and commands require ADMIN permission (before: CREATE). REPO LIST returns absolute paths.

Other Modules

Serialization

  • The default of html-version changed from 4.0 to 5.0.
  • With html-version and version, control characters that the requested version does not allow raise an error.
  • The custom parameters indents, newline and tabulator are superseded by the standard parameters indent-unit and line-ending.
  • Adaptive method: Doubles and floats are output as JSON numbers (1.0e0 → 1), and QNames as #prefix:local (before: #Q{uri}local). BaseX method: Floats are output as JSON numbers.
  • The output of fn:trace and fn:message is serialized with the BaseX method instead of the adaptive method ("A" → A), and it is dropped for users without CREATE permission.

Graphical User Interface

Only a single GUI instance is opened per project directory, and the fullscreen mode was removed (see Graphical User Interface).

⚡Generated with XQuery
 
BaseX 13