Parsers

This article is part of the Advanced User's Guide. It presents various parsers for importing various data source into BaseX databases.

XML Parsers

By default, the standard Java SAXParser is used to parse the input XML documents. Some documents may not be fully processed by the default parser. This may e.g. be the case if they are too large or contain too many or incorrect entities.

The internal, built-in XML parser of BaseX is more fault-tolerant than the default parser. Additionally, it allows you to switch off DTD and/or entity parsing. There are two ways to switch to the internal XML parser:

GUI

Go to Menu Database → New, then choose the Parsing tab and activate Use internal XML parser.

The parsing of entities and DTDs can be turned on/off by clicking on the two check boxes below.

Command Line

Turn on the database option INTPARSE before parsing documents: SET INTPARSE ON.

Entity and DTD parsing can be changed by modifying the DTD options.

XQuery

The db:add() or db:replace() function can be used as well to add new XML documents to the database. The following example query adds all files to the database DB that are found in the directory 2Bimported:

for $file in file:list("2Bimported")
return db:add('DB', $file)

CSV Parser

BaseX can be used to import CSV documents. Different alternatives how to proceed are shown in the following:

GUI

Go to Menu Database → New, choose the Parsing tab and activate "csv" in the input format combobox. You can set the following options for parsing CSV documents:

• Header: Activate this option if the incoming CSV files have a header line.
• Column Separator: Choose the column separator of the CSV file (possible: comma, semicolon, tab).
• XML format: Choose the XML format (possible: verbose, simple).

Command Line

Turn on the CSV Parser before parsing documents: SET PARSER csv. Next, you may set parser-specific options. The default is:

 SET PARSEROPT encoding=utf-8, lines=true, format=verbose, header=false, separator=comma

XQuery

The CSV parser can also be specified in the prolog of an XQuery expression. The db:add() or db:replace() function can be used to add the specified source files into the database. The following example query adds all CSV files to the database DB that are found in the directory 2Bimported, and interprets the first lines as column headers:

declare option db:parser "csv";
declare option db:parseropt "header=yes";
for $file in file:list("2Bimported", false(), "*.csv")
return db:add('DB', $file)

Text Parser

Plain text can be imported as well:

GUI

Go to Menu Database → New, choose the Parsing tab and activate "text" in the input format combobox. You can set the following option for parsing text documents:

Lines

Activate this option to create a <line>...</line> element for each line of the input text file.

Command Line

Turn on the text parser before parsing documents: SET PARSER TEXT. Next, you may set parser-specific options. The default is:

 SET PARSEROPT lines=yes

XQuery

Again, the text parser can also be specified in the prolog of an XQuery expression, and the db:add() or db:replace() function can be used to add the specified source files into the database. The following example query adds all text files to the database DB that are found in the directory 2Bimported and its sub-directories:

declare option db:parser "text";
for $file in file:list("2Bimported", true(), "*.txt")
return db:add('DB', $file)