Parsers

This article is part of the Advanced User's Guide. It presents various parsers for importing various data source into BaseX databases.

XML Parsers

BaseX provides two parsers to import XML data:

• by default, the standard Java SAXParser is used to parse the input XML documents. Some documents may not be fully processed by the default parser. This may e.g. be the case if they are too large or contain too many or incorrect entities.
• the internal, built-in XML parser of BaseX is more fault-tolerant than the default parser, supports standard HTML entities out-of-the-box, and is faster in most cases. In turn, it does not support all oddities specified by DTDs, and cannot resolve catalogs.

GUI

Go to Menu Database → New, then choose the Parsing tab and (de)activate Use internal XML parser. The parsing of DTDs can be turned on/off by selecting the checkbox below.

Command Line

To turn the internal XML parser and DTD parsing on/off, modfify the INTPARSE and DTD options:

 SET INTPARSE ON
 SET DTD ON

XQuery

The db:add() or db:replace() function can be used as well to add new XML documents to the database. The following example query uses the internal parser and adds all files to the database DB that are found in the directory 2Bimported:

declare option db:intparse "yes";
for $file in file:list("2Bimported")
return db:add('DB', $file)

CSV Parser

BaseX can be used to import CSV documents. Different alternatives how to proceed are shown in the following:

GUI

Go to Menu Database → New, choose the Parsing tab and activate "csv" in the input format combobox. You can set the following options for parsing CSV documents:

• Header: Activate this option if the incoming CSV files have a header line.
• Column Separator: Choose the column separator of the CSV file (possible: comma, semicolon, tab).
• XML format: Choose the XML format (possible: verbose, simple).

Command Line

Turn on the CSV Parser before parsing documents, and set some optional, parser-specific options and a file filter:

 SET PARSER CSV
 SET PARSEROPT encoding=utf-8, lines=true, format=verbose, header=false, separator=comma
 SET CREATEFILTER *.csv

XQuery

The CSV parser can also be specified in the prolog of an XQuery expression. The db:add() or db:replace() function can be used to add the specified source files into the database. The following example query adds all CSV files to the database DB that are found in the directory 2Bimported, and interprets the first lines as column headers:

declare option db:parser "csv";
declare option db:parseropt "header=yes";
for $file in file:list("2Bimported", false(), "*.csv")
return db:add('DB', $file)

Text Parser

Plain text can be imported as well:

GUI

Go to Menu Database → New, choose the Parsing tab and activate "text" in the input format combobox. You can set the following option for parsing text documents:

• Lines: Activate this option to create a <line>...</line> element for each line of the input text file.

Command Line

Turn on the CSV Parser before parsing documents and set some optional, parser-specific options:

 SET PARSER text
 SET PARSEROPT lines=yes

XQuery

Again, the text parser can also be specified in the prolog of an XQuery expression, and the db:add() or db:replace() function can be used to add the specified source files into the database. The following example query adds all text files to the database DB that are found in the directory 2Bimported and its sub-directories:

declare option db:parser "text";
for $file in file:list("2Bimported", true(), "*.txt")
return db:add('DB', $file)