Difference between revisions of "Parsers"

From BaseX Documentation
Jump to navigation Jump to search
(27 intermediate revisions by 4 users not shown)
Line 2: Line 2:
 
It presents the available parsers that can be used to import various data sources in BaseX databases.
 
It presents the available parsers that can be used to import various data sources in BaseX databases.
 
Please visit the [[Serialization]] article if you want to know how to export data.
 
Please visit the [[Serialization]] article if you want to know how to export data.
The names of the parser options have been updated with {{Version|7.8}}.
 
  
==XML Parsers==
+
=XML Parsers=
  
BaseX provides two parsers to import XML data:
+
BaseX provides two alternatives for parsing XML:
  
* By default, the internal, built-in XML parser is used, which is more fault-tolerant than Java’s XML parser. It supports standard HTML entities out-of-the-box, and is faster in most cases. In turn, it does not support all oddities specified by DTDs, and cannot resolve [[Catalog_Resolver|catalogs]].
+
* By default, Java’s [http://download.oracle.com/javase/6/docs/api/javax/xml/parsers/SAXParser.html SAXParser] is used to parse XML documents.
* Java’s [http://download.oracle.com/javase/6/docs/api/javax/xml/parsers/SAXParser.html SAXParser] can also be selected for parsing XML documents. This parser is stricter than the built-in parser, but it refuses to process some large documents.
+
* The internal, built-in XML parser is more fault-tolerant than Java’s XML parser. It supports standard HTML entities out of the box, and it is faster than the default parser, in particular if small documents are to be parsed. However, the internal parser does not support the full range of DTD features and cannot resolve [[Catalog_Resolver|catalogs]].
  
===GUI===
+
==GUI==
  
 
Go to Menu ''Database'' → ''New'', then choose the ''Parsing'' tab and (de)activate ''Use internal XML parser''. The parsing of DTDs can be turned on/off by selecting the checkbox below.
 
Go to Menu ''Database'' → ''New'', then choose the ''Parsing'' tab and (de)activate ''Use internal XML parser''. The parsing of DTDs can be turned on/off by selecting the checkbox below.
  
===Command Line===
+
==Command Line==
  
To turn the internal XML parser and DTD parsing on/off, modfify the <code>INTPARSE</code> and <code>DTD</code> options:
+
To turn the internal XML parser and DTD parsing on/off, modify the <code>INTPARSE</code> and <code>DTD</code> options:
  
   SET [[Options#INTPARSE|INTPARSE]] true
+
   SET {{Option|INTPARSE}} true
   SET [[Options#DTD|DTD]] true
+
   SET {{Option|DTD}} true
  
===XQuery===
+
==XQuery==
  
 
The [[Database Module#db:add|db:add]] and [[Database Module#db:replace|db:replace]] functions can also be used to add new XML documents to the database. The following example query uses the internal XML parser and adds all files to the database <code>DB</code> that are found in the directory <code>2Bimported</code>:
 
The [[Database Module#db:add|db:add]] and [[Database Module#db:replace|db:replace]] functions can also be used to add new XML documents to the database. The following example query uses the internal XML parser and adds all files to the database <code>DB</code> that are found in the directory <code>2Bimported</code>:
  
 
<pre class="brush:xquery">
 
<pre class="brush:xquery">
declare option db:intparse "yes";
 
 
for $file in file:list("2Bimported")
 
for $file in file:list("2Bimported")
return db:add('DB', $file)
+
return db:add('DB', $file, '', map { 'intparse': true() })
 
</pre>
 
</pre>
  
==HTML Parser==
+
=HTML Parser=
  
With [http://home.ccil.org/~cowan/XML/tagsoup/ TagSoup], HTML can be imported in BaseX without any problems. TagSoup ensures that only well-formed HTML arrives at the XML parser (correct opening and closing tags, etc.). Hence, if TagSoup is not available on a system, there will be a lot of cases where importing HTML fails, no matter whether you use the GUI or the standalone mode.  
+
If [http://vrici.lojban.org/~cowan/XML/tagsoup/ TagSoup] is found in the [[Startup#Distributions|classpath]], HTML can be imported in BaseX without any problems. TagSoup ensures that only well-formed HTML arrives at the XML parser (correct opening and closing tags, etc.).
  
===Installation===
+
If TagSoup is not available on a system, the default XML parser will be used. (Only) if the input is well-formed XML, the import will succeed.
  
=====Downloads=====
+
==Installation==
 +
 
 +
====Downloads====
  
 
TagSoup is already included in the full BaseX distributions ({{Code|BaseX.zip}}, {{Code|BaseX.exe}}, etc.). It can also be manually downloaded and embedded on the appropriate platforms.
 
TagSoup is already included in the full BaseX distributions ({{Code|BaseX.zip}}, {{Code|BaseX.exe}}, etc.). It can also be manually downloaded and embedded on the appropriate platforms.
  
=====Maven=====
+
====Maven====
  
An easy way to add TagSoup to your own project is to follow this steps:
+
An easy way to add TagSoup to your project is to follow this steps:
  
 
1. visit [http://mvnrepository.com/artifact/org.ccil.cowan.tagsoup/tagsoup/ MVN TagSoup Repository]
 
1. visit [http://mvnrepository.com/artifact/org.ccil.cowan.tagsoup/tagsoup/ MVN TagSoup Repository]
Line 50: Line 50:
 
2. click on the version you want
 
2. click on the version you want
  
3. you can see on the first tab called Maven a XML like this :
+
3. on the first tab, you can see an XML snippet like this:
  
 
<pre class="brush:xml">
 
<pre class="brush:xml">
Line 60: Line 60:
 
</pre>
 
</pre>
  
4. copy that in your own maven project's pom.xml under the <dependencies> tag.  
+
4. copy that in your own maven project’s <code>pom.xml</code> file into the <code><dependencies></code> element.  
  
5. don't forget to run <code>mvn jetty:run</code> again
+
5. don’t forget to run <code>mvn jetty:run</code> again
  
=====Debian=====
+
====Debian====
  
 
With Debian, TagSoup will be automatically detected and included after it has been installed via:
 
With Debian, TagSoup will be automatically detected and included after it has been installed via:
Line 70: Line 70:
 
   apt-get install libtagsoup-java
 
   apt-get install libtagsoup-java
  
===TagSoup Options===
+
==Options==
  
 
TagSoup offers a variety of options to customize the HTML conversion. For the complete list
 
TagSoup offers a variety of options to customize the HTML conversion. For the complete list
please visit the [http://home.ccil.org/~cowan/XML/tagsoup#program TagSoup] website. BaseX supports
+
please visit the [http://vrici.lojban.org/~cowan/XML/tagsoup/#program TagSoup] website. BaseX supports
 
most of these options with a few exceptions:
 
most of these options with a few exceptions:
  
* '''encoding''': BaseX tries to guess the input encoding but this can be overwritten by the user if necessary.
+
* '''encoding''': BaseX tries to guess the input encoding, but this can be overwritten by this option.
 
* '''files''': not supported as input documents are piped directly to the XML parser.
 
* '''files''': not supported as input documents are piped directly to the XML parser.
 
* '''method''': set to 'xml' as default. If this is set to 'html' ending tags may be missing for instance.
 
* '''method''': set to 'xml' as default. If this is set to 'html' ending tags may be missing for instance.
Line 98: Line 98:
 
Turn on the HTML Parser before parsing documents, and set a file filter:
 
Turn on the HTML Parser before parsing documents, and set a file filter:
  
   SET [[Options#PARSER|PARSER]] html
+
   SET {{Option|PARSER}} html
   SET [[Options#HTMLPARSER|HTMLPARSER]] method=xml,nons=true,ncdata=true,nodefaults=true,nobogons=true,nocolons=true,ignorable=true
+
   SET {{Option|HTMLPARSER}} method=xml,nons=true,nocdata=true,nodefaults=true,nobogons=true,nocolons=true,ignorable=true
   SET [[Options#CREATEFILTER|CREATEFILTER]] *.html
+
   SET {{Option|CREATEFILTER}} *.html
  
 
===XQuery===
 
===XQuery===
Line 107: Line 107:
  
 
Documents can also be converted by specifying the parser and additional options
 
Documents can also be converted by specifying the parser and additional options
in the query prolog:
+
as function arguments:
  
 
<pre class="brush:xquery">
 
<pre class="brush:xquery">
declare option db:parser "html";
+
fetch:xml("index.html", map {
declare option db:htmlparser "html=false";
+
  'parser': 'html',
doc("index.html")
+
  'htmlparser': map { 'html': false(), 'nodefaults': true() }
 +
})
 
</pre>
 
</pre>
  
==JSON Parser==
+
=JSON Parser=
  
 
BaseX can also import JSON documents. The resulting format is described in the documentation for the XQuery [[JSON Module]]:
 
BaseX can also import JSON documents. The resulting format is described in the documentation for the XQuery [[JSON Module]]:
  
===GUI===
+
==GUI==
  
 
Go to Menu ''Database'' → ''New'' and select "JSON" in the input format combo box.
 
Go to Menu ''Database'' → ''New'' and select "JSON" in the input format combo box.
Line 127: Line 128:
 
* '''JsonML''': Activate this option if the incoming file is a JsonML file.
 
* '''JsonML''': Activate this option if the incoming file is a JsonML file.
  
===Command Line===
+
==Command Line==
  
 
Turn on the JSON Parser before parsing documents, and set some optional, parser-specific options and a file filter:
 
Turn on the JSON Parser before parsing documents, and set some optional, parser-specific options and a file filter:
  
   SET [[Options#PARSER|PARSER]] json
+
   SET {{Option|PARSER}} json
   SET [[Options#JSONPARSER|JSONPARSER]] encoding=utf-8, jsonml=true
+
   SET {{Option|JSONPARSER}} encoding=utf-8, jsonml=true
   SET [[Options#CREATEFILTER|CREATEFILTER]] *.json
+
   SET {{Option|CREATEFILTER}} *.json
  
===XQuery===
+
==XQuery==
  
 
The [[JSON Module]] provides functions for converting JSON objects to XML documents.
 
The [[JSON Module]] provides functions for converting JSON objects to XML documents.
  
==CSV Parser==
+
=CSV Parser=
  
 
BaseX can be used to import CSV documents. Different alternatives how to proceed are shown in the following:
 
BaseX can be used to import CSV documents. Different alternatives how to proceed are shown in the following:
  
===GUI===
+
==GUI==
  
 
Go to Menu ''Database'' → ''New'' and select "CSV" in the input format combo box.
 
Go to Menu ''Database'' → ''New'' and select "CSV" in the input format combo box.
Line 151: Line 152:
 
* '''Separator''': Choose the column separator of the CSV file. Possible: <code>comma</code>, <code>semicolon</code>, <code>tab</code> or <code>space</code> or an arbitrary character.
 
* '''Separator''': Choose the column separator of the CSV file. Possible: <code>comma</code>, <code>semicolon</code>, <code>tab</code> or <code>space</code> or an arbitrary character.
 
* '''Header''': Activate this option if the incoming CSV files have a header line.
 
* '''Header''': Activate this option if the incoming CSV files have a header line.
* {{Mark|Removed in Version 7.7.2}}: '''XML format''': Choose the XML format. Possible: <code>verbose</code>, <code>simple</code>.
 
  
===Command Line===
+
==Command Line==
  
 
Turn on the CSV Parser before parsing documents, and set some optional, parser-specific options and a file filter. Unicode code points can be specified as separators; {{Code|32}} is the code point for spaces:
 
Turn on the CSV Parser before parsing documents, and set some optional, parser-specific options and a file filter. Unicode code points can be specified as separators; {{Code|32}} is the code point for spaces:
  
   SET [[Options#PARSER|PARSER]] csv
+
   SET {{Option|PARSER}} csv
   SET [[Options#CSVPARSER|CSVPARSER]] encoding=utf-8, lines=true, header=false, separator=space
+
   SET {{Option|CSVPARSER}} encoding=utf-8, lines=true, header=false, separator=space
   SET [[Options#CREATEFILTER|CREATEFILTER]] *.csv
+
   SET {{Option|CREATEFILTER}} *.csv
  
===XQuery===
+
==XQuery==
  
With {{Version|7.7.2}}, the [[CSV Module]] was added. It provides a function for converting CSV to XML documents.
+
The [[CSV Module]] provides a function for converting CSV to XML documents.
  
Documents can also be converted by specifying the parser in the query prolog.
+
Documents can also be converted by specifying the parser in an XQuery function.
 
The following example query adds all CSV files that are located in the
 
The following example query adds all CSV files that are located in the
 
directory {{Code|2Bimported}} to the database {{Code|DB}} and interprets
 
directory {{Code|2Bimported}} to the database {{Code|DB}} and interprets
Line 171: Line 171:
  
 
<pre class="brush:xquery">
 
<pre class="brush:xquery">
declare option db:parser "csv";
 
declare option db:csvparser "header=yes";
 
 
for $file in file:list("2Bimported", false(), "*.csv")
 
for $file in file:list("2Bimported", false(), "*.csv")
return db:add('DB', $file)
+
return db:add("DB", $file, "", map {
 +
  'parser': 'csv',
 +
  'csvparser': map { 'header': true() }
 +
})
 
</pre>
 
</pre>
  
==Text Parser==
+
=Text Parser=
  
 
Plain text can be imported as well:
 
Plain text can be imported as well:
  
===GUI===
+
==GUI==
  
 
Go to Menu ''Database'' → ''New'' and select "TEXT" in the input format combobox.
 
Go to Menu ''Database'' → ''New'' and select "TEXT" in the input format combobox.
Line 189: Line 190:
 
* '''Lines''': Activate this option to create a <code>&lt;line&gt;...&lt;/line&gt;</code> element for each line of the input text file.
 
* '''Lines''': Activate this option to create a <code>&lt;line&gt;...&lt;/line&gt;</code> element for each line of the input text file.
  
===Command Line===
+
==Command Line==
  
 
Turn on the CSV Parser before parsing documents and set some optional, parser-specific options and a file filter:
 
Turn on the CSV Parser before parsing documents and set some optional, parser-specific options and a file filter:
  
   SET [[Options#PARSER|PARSER]] text
+
   SET {{Option|PARSER}} text
   SET [[Options#TEXTPARSER|TEXTPARSER]] lines=yes
+
   SET {{Option|TEXTPARSER}} lines=yes
   SET [[Options#CREATEFILTER|CREATEFILTER]] *
+
   SET {{Option|CREATEFILTER}} *
  
===XQuery===
+
==XQuery==
  
Similar to the other formats the text parser can be specified in the prolog of an
+
Similar to the other formats, the text parser can also be specified via XQuery:
XQuery expression:
 
  
 
<pre class="brush:xquery">
 
<pre class="brush:xquery">
declare option db:parser "text";
 
 
for $file in file:list("2Bimported", true(), "*.txt")
 
for $file in file:list("2Bimported", true(), "*.txt")
return db:add('DB', $file)
+
return db:add("DB", $file, "", map { 'parser': 'text' })
 
</pre>
 
</pre>
  
Line 225: Line 224:
  
 
* Updated: Enhanced support for TagSoup options
 
* Updated: Enhanced support for TagSoup options
 
[[Category: Beginner]]
 

Revision as of 15:56, 10 April 2019

This article is part of the Getting Started Section. It presents the available parsers that can be used to import various data sources in BaseX databases. Please visit the Serialization article if you want to know how to export data.

XML Parsers

BaseX provides two alternatives for parsing XML:

  • By default, Java’s SAXParser is used to parse XML documents.
  • The internal, built-in XML parser is more fault-tolerant than Java’s XML parser. It supports standard HTML entities out of the box, and it is faster than the default parser, in particular if small documents are to be parsed. However, the internal parser does not support the full range of DTD features and cannot resolve catalogs.

GUI

Go to Menu DatabaseNew, then choose the Parsing tab and (de)activate Use internal XML parser. The parsing of DTDs can be turned on/off by selecting the checkbox below.

Command Line

To turn the internal XML parser and DTD parsing on/off, modify the INTPARSE and DTD options:

 SET INTPARSE true
 SET DTD true

XQuery

The db:add and db:replace functions can also be used to add new XML documents to the database. The following example query uses the internal XML parser and adds all files to the database DB that are found in the directory 2Bimported:

for $file in file:list("2Bimported")
return db:add('DB', $file, '', map { 'intparse': true() })

HTML Parser

If TagSoup is found in the classpath, HTML can be imported in BaseX without any problems. TagSoup ensures that only well-formed HTML arrives at the XML parser (correct opening and closing tags, etc.).

If TagSoup is not available on a system, the default XML parser will be used. (Only) if the input is well-formed XML, the import will succeed.

Installation

Downloads

TagSoup is already included in the full BaseX distributions (BaseX.zip, BaseX.exe, etc.). It can also be manually downloaded and embedded on the appropriate platforms.

Maven

An easy way to add TagSoup to your project is to follow this steps:

1. visit MVN TagSoup Repository

2. click on the version you want

3. on the first tab, you can see an XML snippet like this:

<dependency>
  <groupId>org.ccil.cowan.tagsoup</groupId>
  <artifactId>tagsoup</artifactId>
  <version>1.2.1</version>
</dependency>

4. copy that in your own maven project’s pom.xml file into the <dependencies> element.

5. don’t forget to run mvn jetty:run again

Debian

With Debian, TagSoup will be automatically detected and included after it has been installed via:

 apt-get install libtagsoup-java

Options

TagSoup offers a variety of options to customize the HTML conversion. For the complete list please visit the TagSoup website. BaseX supports most of these options with a few exceptions:

  • encoding: BaseX tries to guess the input encoding, but this can be overwritten by this option.
  • files: not supported as input documents are piped directly to the XML parser.
  • method: set to 'xml' as default. If this is set to 'html' ending tags may be missing for instance.
  • version: dismissed, as TagSoup always falls back to 'version 1.0', no matter what the input is.
  • standalone: deactivated.
  • pyx, pyxin: not supported as the XML parser can't handle this kind of input.
  • output-encoding: not supported, BaseX already takes care of that.
  • reuse, help: not supported.

GUI

Go to Menu DatabaseNew and select "HTML" in the input format combo box. There's an info in the "Parsing" tab about whether TagSoup is available or not. The same applies to the "Resources" tab in the "Database Properties" dialog.

These two dialogs come with an input field 'Parameters' where TagSoup options can be entered.

Command Line

Turn on the HTML Parser before parsing documents, and set a file filter:

 SET PARSER html
 SET HTMLPARSER method=xml,nons=true,nocdata=true,nodefaults=true,nobogons=true,nocolons=true,ignorable=true
 SET CREATEFILTER *.html

XQuery

The HTML Module provides a function for converting HTML to XML documents.

Documents can also be converted by specifying the parser and additional options as function arguments:

fetch:xml("index.html", map {
  'parser': 'html',
  'htmlparser': map { 'html': false(), 'nodefaults': true() }
})

JSON Parser

BaseX can also import JSON documents. The resulting format is described in the documentation for the XQuery JSON Module:

GUI

Go to Menu DatabaseNew and select "JSON" in the input format combo box. You can set the following options for parsing JSON documents in the "Parsing" tab:

  • Encoding: Choose the appropriate encoding of the JSON file.
  • JsonML: Activate this option if the incoming file is a JsonML file.

Command Line

Turn on the JSON Parser before parsing documents, and set some optional, parser-specific options and a file filter:

 SET PARSER json
 SET JSONPARSER encoding=utf-8, jsonml=true
 SET CREATEFILTER *.json

XQuery

The JSON Module provides functions for converting JSON objects to XML documents.

CSV Parser

BaseX can be used to import CSV documents. Different alternatives how to proceed are shown in the following:

GUI

Go to Menu DatabaseNew and select "CSV" in the input format combo box. You can set the following options for parsing CSV documents in the "Parsing" tab:

  • Encoding: Choose the appropriate encoding of the CSV file.
  • Separator: Choose the column separator of the CSV file. Possible: comma, semicolon, tab or space or an arbitrary character.
  • Header: Activate this option if the incoming CSV files have a header line.

Command Line

Turn on the CSV Parser before parsing documents, and set some optional, parser-specific options and a file filter. Unicode code points can be specified as separators; 32 is the code point for spaces:

 SET PARSER csv
 SET CSVPARSER encoding=utf-8, lines=true, header=false, separator=space
 SET CREATEFILTER *.csv

XQuery

The CSV Module provides a function for converting CSV to XML documents.

Documents can also be converted by specifying the parser in an XQuery function. The following example query adds all CSV files that are located in the directory 2Bimported to the database DB and interprets the first lines as column headers:

for $file in file:list("2Bimported", false(), "*.csv")
return db:add("DB", $file, "", map {
  'parser': 'csv',
  'csvparser': map { 'header': true() }
})

Text Parser

Plain text can be imported as well:

GUI

Go to Menu DatabaseNew and select "TEXT" in the input format combobox. You can set the following option for parsing text documents in the "Parsing" tab:

  • Encoding: Choose the appropriate encoding of the text file.
  • Lines: Activate this option to create a <line>...</line> element for each line of the input text file.

Command Line

Turn on the CSV Parser before parsing documents and set some optional, parser-specific options and a file filter:

 SET PARSER text
 SET TEXTPARSER lines=yes
 SET CREATEFILTER *

XQuery

Similar to the other formats, the text parser can also be specified via XQuery:

for $file in file:list("2Bimported", true(), "*.txt")
return db:add("DB", $file, "", map { 'parser': 'text' })

Changelog

Version 7.8
  • Updated: parser options
Version 7.7.2
  • Removed: CSV option "format"
Version 7.3
  • Updated: the CSV SEPARATOR option may now be assigned arbitrary single characters
Version 7.2
  • Updated: Enhanced support for TagSoup options