Parsers

This article is part of the Getting Started Section. It presents the available parsers that can be used to import various data sources in BaseX databases. Please visit the Serialization article if you want to know how to export data.

=XML Parsers=

BaseX provides two alternatives for parsing XML:


 * By default, Java’s SAXParser is used to parse XML documents.
 * The internal, built-in XML parser is more fault-tolerant than Java’s XML parser. It supports standard HTML entities out of the box, and it is faster than the default parser, in particular if small documents are to be parsed. However, the internal parser does not support the full range of DTD features and cannot resolve catalogs.

GUI
Go to Menu Database → New, then choose the Parsing tab and (de)activate Use internal XML parser. The parsing of DTDs can be turned on/off by selecting the checkbox below.

Command Line
To turn the internal XML parser and DTD parsing on/off, modify the  and   options:

SET true SET true

XQuery
The db:add and db:replace functions can also be used to add new XML documents to the database. The following example query uses the internal XML parser and adds all files to the database  that are found in the directory  :

 for $file in file:list("2Bimported") return db:add('DB', $file, '', map { 'intparse': true })

=HTML Parser=

If TagSoup is found in the classpath, HTML can be imported in BaseX without any problems. TagSoup ensures that only well-formed HTML arrives at the XML parser (correct opening and closing tags, etc.).

If TagSoup is not available on a system, the default XML parser will be used. (Only) if the input is well-formed XML, the import will succeed.

Downloads
TagSoup is already included in the full BaseX distributions (BaseX.zip, BaseX.exe, etc.). It can also be manually downloaded and embedded on the appropriate platforms.

Maven
An easy way to add TagSoup to your project is to follow this steps:

1. visit MVN TagSoup Repository

2. click on the version you want

3. on the first tab, you can see an XML snippet like this:

 org.ccil.cowan.tagsoup tagsoup 1.2.1

4. copy that in your own maven project’s  file into the   element.

5. don’t forget to run  again

Debian
With Debian, TagSoup will be automatically detected and included after it has been installed via:

apt-get install libtagsoup-java

Options
TagSoup offers a variety of options to customize the HTML conversion. For the complete list please visit the TagSoup website. BaseX supports most of these options with a few exceptions:


 * encoding: BaseX tries to guess the input encoding, but this can be overwritten by this option.
 * files: not supported as input documents are piped directly to the XML parser.
 * method: set to 'xml' as default. If this is set to 'html' ending tags may be missing for instance.
 * version: dismissed, as TagSoup always falls back to 'version 1.0', no matter what the input is.
 * standalone: deactivated.
 * pyx, pyxin: not supported as the XML parser can't handle this kind of input.
 * output-encoding: not supported, BaseX already takes care of that.
 * reuse, help: not supported.

GUI
Go to Menu Database → New and select "HTML" in the input format combo box. There's an info in the "Parsing" tab about whether TagSoup is available or not. The same applies to the "Resources" tab in the "Database Properties" dialog.

These two dialogs come with an input field 'Parameters' where TagSoup options can be entered.

Command Line
Turn on the HTML Parser before parsing documents, and set a file filter:

SET html SET method=xml,nons=true,nocdata=true,nodefaults=true,nobogons=true,nocolons=true,ignorable=true SET *.html

XQuery
The HTML Module provides a function for converting HTML to XML documents.

Documents can also be converted by specifying the parser and additional options as function arguments:

 fetch:xml("index.html", map { 'parser': 'html',  'htmlparser': map { 'html': false, 'nodefaults': true } })

=JSON Parser=

BaseX can also import JSON documents. The resulting format is described in the documentation for the XQuery JSON Module:

GUI
Go to Menu Database → New and select "JSON" in the input format combo box. You can set the following options for parsing JSON documents in the "Parsing" tab:


 * Encoding: Choose the appropriate encoding of the JSON file.
 * JsonML: Activate this option if the incoming file is a JsonML file.

Command Line
Turn on the JSON Parser before parsing documents, and set some optional, parser-specific options and a file filter:

SET json SET encoding=utf-8, jsonml=true SET *.json

XQuery
The JSON Module provides functions for converting JSON objects to XML documents.

=CSV Parser=

BaseX can be used to import CSV documents. Different alternatives how to proceed are shown in the following:

GUI
Go to Menu Database → New and select "CSV" in the input format combo box. You can set the following options for parsing CSV documents in the "Parsing" tab:


 * Encoding: Choose the appropriate encoding of the CSV file.
 * Separator: Choose the column separator of the CSV file. Possible:,  ,   or   or an arbitrary character.
 * Header: Activate this option if the incoming CSV files have a header line.

Command Line
Turn on the CSV Parser before parsing documents, and set some optional, parser-specific options and a file filter. Unicode code points can be specified as separators; 32 is the code point for spaces:

SET csv SET encoding=utf-8, lines=true, header=false, separator=space SET *.csv

XQuery
The CSV Module provides a function for converting CSV to XML documents.

Documents can also be converted by specifying the parser in an XQuery function. The following example query adds all CSV files that are located in the directory 2Bimported to the database DB and interprets the first lines as column headers:

 for $file in file:list("2Bimported", false, "*.csv") return db:add("DB", $file, "", map { 'parser': 'csv',  'csvparser': map { 'header': true } })

=Text Parser=

Plain text can be imported as well:

GUI
Go to Menu Database → New and select "TEXT" in the input format combobox. You can set the following option for parsing text documents in the "Parsing" tab:


 * Encoding: Choose the appropriate encoding of the text file.
 * Lines: Activate this option to create a  element for each line of the input text file.

Command Line
Turn on the CSV Parser before parsing documents and set some optional, parser-specific options and a file filter:

SET text SET lines=yes SET *

XQuery
Similar to the other formats, the text parser can also be specified via XQuery:

 for $file in file:list("2Bimported", true, "*.txt") return db:add("DB", $file, "", map { 'parser': 'text' })

=Changelog=


 * Version 7.8


 * Updated: parser options


 * Version 7.7.2


 * Removed: CSV option "format"


 * Version 7.3


 * Updated: the CSV SEPARATOR option may now be assigned arbitrary single characters


 * Version 7.2


 * Updated: Enhanced support for TagSoup options