Parsers

From BaseX Documentation

Jump to: navigation, search

This article is part of the Getting Started Section. It presents the available parsers that can be used to import various data sources in BaseX databases. Please visit the Serialization article if you want to know how to export data. The names of the parser options have been updated with Version 7.8.

Contents

[edit] XML Parsers

BaseX provides two parsers to import XML data:

[edit] GUI

Go to Menu DatabaseNew, then choose the Parsing tab and (de)activate Use internal XML parser. The parsing of DTDs can be turned on/off by selecting the checkbox below.

[edit] Command Line

To turn the internal XML parser and DTD parsing on/off, modfify the INTPARSE and DTD options:

 SET INTPARSE true
 SET DTD true

[edit] XQuery

The db:add and db:replace functions can also be used to add new XML documents to the database. The following example query uses the internal XML parser and adds all files to the database DB that are found in the directory 2Bimported:

declare option db:intparse "yes";
for $file in file:list("2Bimported")
return db:add('DB', $file)

[edit] HTML Parser

With TagSoup, HTML can be imported in BaseX without any problems. TagSoup ensures that only well-formed HTML arrives at the XML parser (correct opening and closing tags, etc.). Hence, if TagSoup is not available on a system, there will be a lot of cases where importing HTML fails, no matter whether you use the GUI or the standalone mode.

[edit] Installation

[edit] Downloads

TagSoup is already included in the full BaseX distributions (BaseX.zip, BaseX.exe, etc.). It can also be manually downloaded and embedded on the appropriate platforms.

[edit] Maven

An easy way to add TagSoup to your own project is to follow this steps:

1. visit MVN TagSoup Repository

2. click on the version you want

3. you can see on the first tab called Maven a XML like this :

<dependency>
  <groupId>org.ccil.cowan.tagsoup</groupId>
  <artifactId>tagsoup</artifactId>
  <version>1.2.1</version>
</dependency>

4. copy that in your own maven project's pom.xml under the <dependencies> tag.

5. don't forget to run mvn jetty:run again

[edit] Debian

With Debian, TagSoup will be automatically detected and included after it has been installed via:

 apt-get install libtagsoup-java

[edit] TagSoup Options

TagSoup offers a variety of options to customize the HTML conversion. For the complete list please visit the TagSoup website. BaseX supports most of these options with a few exceptions:

[edit] GUI

Go to Menu DatabaseNew and select "HTML" in the input format combo box. There's an info in the "Parsing" tab about whether TagSoup is available or not. The same applies to the "Resources" tab in the "Database Properties" dialog.

These two dialogs come with an input field 'Parameters' where TagSoup options can be entered.

[edit] Command Line

Turn on the HTML Parser before parsing documents, and set a file filter:

 SET PARSER html
 SET HTMLPARSER method=xml,nons=true,ncdata=true,nodefaults=true,nobogons=true,nocolons=true,ignorable=true
 SET CREATEFILTER *.html

[edit] XQuery

The HTML Module provides a function for converting HTML to XML documents.

Documents can also be converted by specifying the parser and additional options in the query prolog:

declare option db:parser "html";
declare option db:htmlparser "html=false,nodefaults=true";
doc("index.html")

[edit] JSON Parser

BaseX can also import JSON documents. The resulting format is described in the documentation for the XQuery JSON Module:

[edit] GUI

Go to Menu DatabaseNew and select "JSON" in the input format combo box. You can set the following options for parsing JSON documents in the "Parsing" tab:

[edit] Command Line

Turn on the JSON Parser before parsing documents, and set some optional, parser-specific options and a file filter:

 SET PARSER json
 SET JSONPARSER encoding=utf-8, jsonml=true
 SET CREATEFILTER *.json

[edit] XQuery

The JSON Module provides functions for converting JSON objects to XML documents.

[edit] CSV Parser

BaseX can be used to import CSV documents. Different alternatives how to proceed are shown in the following:

[edit] GUI

Go to Menu DatabaseNew and select "CSV" in the input format combo box. You can set the following options for parsing CSV documents in the "Parsing" tab:

[edit] Command Line

Turn on the CSV Parser before parsing documents, and set some optional, parser-specific options and a file filter. Unicode code points can be specified as separators; 32 is the code point for spaces:

 SET PARSER csv
 SET CSVPARSER encoding=utf-8, lines=true, header=false, separator=space
 SET CREATEFILTER *.csv

[edit] XQuery

The CSV Module provides a function for converting CSV to XML documents.

Documents can also be converted by specifying the parser in the query prolog. The following example query adds all CSV files that are located in the directory 2Bimported to the database DB and interprets the first lines as column headers:

declare option db:parser "csv";
declare option db:csvparser "header=yes";
for $file in file:list("2Bimported", false(), "*.csv")
return db:add('DB', $file)

[edit] Text Parser

Plain text can be imported as well:

[edit] GUI

Go to Menu DatabaseNew and select "TEXT" in the input format combobox. You can set the following option for parsing text documents in the "Parsing" tab:

[edit] Command Line

Turn on the CSV Parser before parsing documents and set some optional, parser-specific options and a file filter:

 SET PARSER text
 SET TEXTPARSER lines=yes
 SET CREATEFILTER *

[edit] XQuery

Similar to the other formats the text parser can be specified in the prolog of an XQuery expression:

declare option db:parser "text";
for $file in file:list("2Bimported", true(), "*.txt")
return db:add('DB', $file)

[edit] Changelog

Version 7.8
Version 7.7.2
Version 7.3
Version 7.2
Personal tools
Namespaces
Variants
Actions
Navigation
Print/export