Main Page » Advanced User Guide » Parsers

Parsers

This article presents the available parsers that can be used to import various data sources in BaseX databases. Please visit the Serialization article if you want to know how to export data.

XML Parsers

BaseX provides two alternatives for parsing XML:

  • By default, Java’s SAXParser is used to parse XML documents.
  • The internal, built-in XML parser is more fault-tolerant than Java’s XML parser. It supports standard HTML entities out of the box, and it is faster than the default parser, in particular if small documents are to be parsed. However, the internal parser does not support the full range of DTD features and cannot resolve catalogs.

GUI

Go to Menu Database → New, then choose the Parsing tab and (de)activate Use internal XML parser. The parsing of DTDs can be turned on/off by selecting the checkbox below.

Command Line

To turn the internal XML parser and DTD parsing on/off, modify the INTPARSE and DTD options:

SET INTPARSE true
SET DTD true

XQuery

The db:add and db:put functions can also be used to add new XML documents to the database. The following example query uses the internal XML parser and adds all files to the database DB that are found in the directory 2Bimported:

for $file in file:list("2Bimported")
return db:add('DB', $file, '', { 'intparse': true() })

HTML Parser

The HTML parser can be selected using the HTMLPARSER option method:

If this option is explicitly set, the corresponding parser must be available in the classpath. The selected parser will convert HTML input into well-formed XML documents.

By default, TagSoup is used for HTML processing. If TagSoup is unavailable in the classpath, Validator.nu will be used instead. If neither parser is available, the default XML parser will be used. In this case, the import will only succeed if the input is already well-formed XML, which is typically true for XHTML documents.

Installation

Downloads

TagSoup and Validator.nu are included in the full distributions of BaseX. They can also be manually downloaded and added to the classpath.

Maven

An easy way to add TagSoup to your project is as follows:

  • Visit MVN TagSoup Repository
  • Click on the version you want
  • On the first tab, you can see an XML snippet like this:
<dependency>
  <groupId>org.ccil.cowan.tagsoup</groupId>
  <artifactId>tagsoup</artifactId>
  <version>1.2.1</version>
</dependency>
  • Insert the XML fragment into the <dependencies> element of your project’s pom.xml file.

For Validator.nu, go to MVN Validator.nu Repository, select the version, and find a snippet like the following for inclusion into your project:

<dependency>
  <groupId>nu.validator</groupId>
  <artifactId>htmlparser</artifactId>
  <version>1.4.16</version>
</dependency>

Debian

With Debian, TagSoup will automatically be detected and included after it has been installed via:

apt-get install libtagsoup-java

Validator.nu can be installed with this command:

apt-get install libhtml5parser-java

Options

TagSoup

TagSoup offers a variety of options to customize the HTML conversion. For the complete list, please visit the TagSoup website. BaseX supports most options, with a few exceptions:

  • encoding: BaseX tries to guess the input encoding, but this can be overwritten by this option.
  • files: not supported, as the input documents are piped directly to the XML parser.
  • html, method: not supported, as they are only used to make TagSoup output HTML.
  • version: dismissed, as TagSoup always falls back to 'version 1.0', no matter what the input is.
  • standalone: deactivated.
  • pyx, pyxin: not supported, as the XML parser cannot handle this kind of input.
  • output-encoding: not supported, BaseX already takes care of that.
  • reuse, help: not supported.

Validator.nu

The following list shows the names to use for setting Validator.nu options (for details of these options, see the corresponding description in the Validor.nu HtmlParser Javadoc):

  • comment-policy
  • content-non-xml-char-policy
  • content-space-policy
  • encoding
  • heuristics
  • mapping-lang-to-xml-lang
  • name-policy
  • scripting-enabled
  • streamability-violation-policy
  • unicode-normalization-checking
  • xml-policy
  • xmlns-policy

Option heuristics has extra classpath requirements, depending on its value:

  • heuristics=icu: requires class com.ibm.icu.text.CharsetDetector from com.ibm.icu:icu4j. This is included in the full distributions of BaseX.
  • heuristics=chardet: requires class org.mozilla.intl.chardet.nsICharsetDetectionObserver from net.sourceforge.jchardet:jchardet:1.0. This is not included in a BaseX distribution.

When the respective class is unavailable, using either of these option values will result in an error message.

GUI

Go to Menu Database → New and select “HTML” in the input format combo box. There is an info in the “Parsing” tab about whether the HTML parser is available or not. The same applies to the “Resources” tab in the “Database Properties” dialog.

These two dialogs come with an input field 'Parameters' where options of the HTML parser can be entered. Keys and values are separated by the equal sign, multiple options are separated by commas, e.g.: method=tagsoup,nons=true.

Command Line

Turn on the HTML Parser before parsing documents, set a file filter and optional parsing options:

SET PARSER html
SET CREATEFILTER *.html
SET HTMLPARSER method=tagsoup,nons=true

XQuery

With the HTML Functions, HTML can be converted to XML. In addition, all functions that accept Parsing Options can be used as well to convert HTML:

fetch:doc("index.html", {
  'parser': 'html',
  'htmlparser': { 'method': 'tagsoup', 'nons': true() }
})

JSON Parser

With BaseX, JSON documents can be imported. The JSON parsing options are described in the documentation of the JSON Functions.

GUI

Go to Menu Database → New and select “JSON” in the input format combo box. You can set the following options for parsing JSON documents in the “Parsing” tab.

Command Line

Turn on the JSON Parser before parsing documents, and set some optional, parser-specific options and a file filter:

SET PARSER json
SET JSONPARSER encoding=utf-8, jsonml=true
SET CREATEFILTER *.json

XQuery

With the JSON Functions, JSON can be converted to XML.

In addition, all functions that accept Parsing Options can be used as well to convert JSON. The following query converts a JSON string to an XML document, using the conversion format basic:

db:create('json-db', '{ "key": "value" }', 'example.json', {
  'parser': 'json',
  'jsonparser': { 'format': 'basic' }
})

CSV Parser

BaseX can be used to import CSV documents. The CSV parsing options are described in the documentation of the CSV Functions.

GUI

Go to Menu Database → New and select “CSV” in the input format combo box.

You can set the following options for parsing CSV documents in the “Parsing” tab.

Command Line

Enable CSV Parsing before parsing documents, and set optional, parser-specific options and a file filter:

SET PARSER csv
SET CSVPARSER encoding=utf-8,header=true,separator=semicolon
SET CREATEFILTER *.csv

XQuery

With the CSV Functions, CSV can be converted to XML.

In addition, all functions that accept Parsing Options can be used as well to convert CSV. In the following, all CSV files from a directory are added to a database, with the first line of each file interpreted as column header:

for $file in file:list('2Bimported', false(), '*.csv')
return db:add("csv-db", $file, "", {
  'parser': 'csv',
  'csvparser': { 'header': true() }
})

Changelog

Version 12.0
  • Added: HTMLPARSER option method=nu.
Version 11.0
  • Removed: Text Parser.
Version 7.8
  • Updated: parser options.
Version 7.7.2
  • Removed: CSV option format.
Version 7.3
  • Updated: the CSV SEPARATOR option may now be assigned arbitrary single characters.
Version 7.2
  • Updated: Enhanced support for TagSoup options.
⚡Generated with XQuery
 
Login | Changelog | TOC | Imprint
Parsers