Parsers

This article presents the available parsers that can be used to import various data sources into BaseX databases. Please visit the Serialization article if you want to know how to export data.

Each input format can be addressed in three ways: in the GUI, via the PARSER option, and in XQuery. The following table gives an overview; the format-specific parsing options are documented in the linked articles:

Format	Parser	XQuery functions	Options
XML	xml	fn:parse-xml, fn:doc, db:add	Options
HTML	html	html:parse, fn:parse-html	HTML Functions
JSON	json	json:parse, fn:parse-json	JSON Functions
CSV	csv	csv:parse, fn:parse-csv	CSV Functions
Binary data	raw	db:put-binary	–
Other text formats	–	fn:invisible-xml	Invisible XML

XML Parsers

BaseX provides two alternatives for parsing XML:

• By default, Java’s SAXParser is used to parse XML documents.
• The internal, built-in XML parser is more fault-tolerant than Java’s XML parser. It supports standard HTML entities out of the box, and it is faster than the default parser, in particular if small documents are to be parsed. However, the internal parser does not support the full range of DTD features and cannot resolve catalogs.

GUI

Go to Menu Database → New, then choose the Parsing tab and (de)activate Use internal XML parser. The parsing of DTDs can be turned on/off by selecting the checkbox below.

Command Line

To turn the internal XML parser and DTD parsing on/off, modify the INTPARSE and DTD options. Further options, such as XINCLUDE, STRIPNS and CATALOG, are listed in XML Parsing:

SET INTPARSE true
SET DTD true

XQuery

The db:add and db:put functions can also be used to add new XML documents to the database. The following example query uses the internal XML parser and adds all files to the database DB that are found in the directory 2Bimported:

for $file in file:list("2Bimported")
return db:add('DB', $file, '', { 'intparse': true() })

HTML Parser

HTML documents are converted to well-formed XML by an external parser: TagSoup or Validator.nu. If neither is available, the input is returned as text. The available parsers, their installation and their parsing options are described in the documentation of the HTML Functions.

GUI

Go to Menu Database → New and select “HTML” in the input format combo box. The “Parsing” tab indicates whether an HTML parser is available. The same applies to the “Resources” tab in the “Database Properties” dialog.

These two dialogs come with an input field 'Parameters' where options of the HTML parser can be entered. Keys and values are separated by the equal sign, and multiple options are separated by commas, for example: method=tagsoup,nons=true.

Command Line

Turn on the HTML Parser before parsing documents, set a file filter and optional parsing options:

SET PARSER html
SET CREATEFILTER *.html
SET HTMLPARSER method=tagsoup,nons=true

XQuery

With the HTML Functions, HTML can be converted to XML. In addition, all functions that accept Parsing Options can be used as well to convert HTML:

fetch:doc('index.html', {
  'parser': 'html',
  'htmlparser': { 'method': 'tagsoup', 'nons': true() }
})

JSON Parser

With BaseX, JSON documents can be imported. The JSON parsing options are described in the documentation of the JSON Functions.

GUI

Go to Menu Database → New and select “JSON” in the input format combo box. The options for parsing JSON documents can be set in the “Parsing” tab.

Command Line

Turn on the JSON Parser before parsing documents, and set some optional, parser-specific options and a file filter:

SET PARSER json
SET JSONPARSER encoding=utf-8,format=jsonml
SET CREATEFILTER *.json

XQuery

With the JSON Functions, JSON can be converted to XML.

In addition, all functions that accept Parsing Options can be used as well to convert JSON. The following query converts a JSON string to an XML document, using the conversion format w3-xml:

db:create('json-db', '{ "key": "value" }', 'example.json', {
  'parser': 'json',
  'jsonparser': { 'format': 'w3-xml' }
})

CSV Parser

BaseX can be used to import CSV documents. The CSV parsing options are described in the documentation of the CSV Functions.

GUI

Go to Menu Database → New and select “CSV” in the input format combo box. The options for parsing CSV documents can be set in the “Parsing” tab.

Command Line

Enable CSV Parsing before parsing documents, and set optional, parser-specific options and a file filter:

SET PARSER csv
SET CSVPARSER encoding=utf-8,header=true,separator=;
SET CREATEFILTER *.csv

XQuery

With the CSV Functions, CSV can be converted to XML.

In addition, all functions that accept Parsing Options can be used as well to convert CSV. In the following, all CSV files from a directory are added to a database, with the first line of each file interpreted as column header:

for $file in file:list('2Bimported', false(), '*.csv')
return db:add('csv-db', $file, '', {
  'parser': 'csv',
  'csvparser': { 'header': true() }
})

Raw Parser

With the raw parser, files are not parsed at all: they are stored unchanged as Binary Data. No parsing options are available. See also the ADDRAW option, which stores all resources that are filtered out by the CREATEFILTER option as binary data.

GUI

Go to Menu Database → New and select “RAW” in the input format combo box. The “Parsing” tab will be disabled.

Command Line

Enable the raw parser before adding documents, and adjust the file filter, which is restricted to XML documents by default:

SET PARSER raw
SET CREATEFILTER *

XQuery

Single binary resources are best added with db:put-binary. The parser can also be assigned via the parsing options, e.g. to import a full directory:

db:create('media-db', 'archive', 'archive', {
  'parser': 'raw',
  'createfilter': '*'
})

Changelog

Version 12.0

• Added: HTMLPARSER option method=nu.

Version 11.0

• Removed: Text Parser.

Version 7.8

• Updated: parser options.

Version 7.7.2

• Removed: CSV option format.

Version 7.3

• Updated: the CSV SEPARATOR option may now be assigned arbitrary single characters.

Version 7.2

• Updated: Enhanced support for TagSoup options.