Difference between revisions of "Parsers"

From BaseX Documentation
Jump to navigation Jump to search
m (Text replacement - "syntaxhighlight" to "pre")
 
(114 intermediate revisions by 6 users not shown)
Line 1: Line 1:
This article is part of the [[Getting Started]] Section.
+
This article is part of the [[Advanced User's Guide]]. It presents the available parsers that can be used to import various data sources in BaseX databases. Please visit the [[Serialization]] article if you want to know how to export data.
It presents different parsers for importing various data
 
source into BaseX databases. For export see [[Serialization]].
 
  
==XML Parsers==
+
=XML Parsers=
  
BaseX provides two parsers to import XML data:
+
BaseX provides two alternatives for parsing XML:
  
* By default, the internal, built-in XML parser is used, which is more fault-tolerant than Java’s XML parser. It supports standard HTML entities out-of-the-box, and is faster in most cases. In turn, it does not support all oddities specified by DTDs, and cannot resolve [[Catalog_Resolver|catalogs]].
+
* By default, Java’s [https://docs.oracle.com/en/java/javase/11/docs/api/java.xml/javax/xml/parsers/SAXParser.html SAXParser] is used to parse XML documents.
* Java’s [http://download.oracle.com/javase/6/docs/api/javax/xml/parsers/SAXParser.html SAXParser] can also be selected for parsing XML documents. This parser is stricter than the built-in parser, but it refuses to process some large documents.
+
* The internal, built-in XML parser is more fault-tolerant than Java’s XML parser. It supports standard HTML entities out of the box, and it is faster than the default parser, in particular if small documents are to be parsed. However, the internal parser does not support the full range of DTD features and cannot resolve [[Catalog_Resolver|catalogs]].
  
===GUI===
+
==GUI==
  
 
Go to Menu ''Database'' → ''New'', then choose the ''Parsing'' tab and (de)activate ''Use internal XML parser''. The parsing of DTDs can be turned on/off by selecting the checkbox below.
 
Go to Menu ''Database'' → ''New'', then choose the ''Parsing'' tab and (de)activate ''Use internal XML parser''. The parsing of DTDs can be turned on/off by selecting the checkbox below.
  
===Command Line===
+
==Command Line==
  
To turn the internal XML parser and DTD parsing on/off, modfify the <code>INTPARSE</code> and <code>DTD</code> options:
+
To turn the internal XML parser and DTD parsing on/off, modify the <code>INTPARSE</code> and <code>DTD</code> options:
  
   SET [[Options#INTPARSE|INTPARSE]] true
+
   SET {{Option|INTPARSE}} true
   SET [[Options#DTD|DTD]] true
+
   SET {{Option|DTD}} true
  
===XQuery===
+
==XQuery==
  
The [[Database Module#db:add|db:add()]] or [[Database Module#db:replace|db:replace()]]
+
The {{Function|Database|db:add}} and {{Function|Database|db:put}} functions can also be used to add new XML documents to the database. The following example query uses the internal XML parser and adds all files to the database <code>DB</code> that are found in the directory <code>2Bimported</code>:
function can be used as well to add new XML documents to the database.
 
The following example query uses the internal parser and adds all files
 
to the database <code>DB</code> that are found in the directory
 
<code>2Bimported</code>:
 
  
<pre class="brush:xquery">
+
<pre lang='xquery'>
declare option db:intparse "yes";
 
 
for $file in file:list("2Bimported")
 
for $file in file:list("2Bimported")
return db:add('DB', $file)
+
return db:add('DB', $file, '', map { 'intparse': true() })
 
</pre>
 
</pre>
  
==HTML Parsers==
+
=HTML Parser=
  
With [http://home.ccil.org/~cowan/XML/tagsoup/ TagSoup], HTML can be imported in BaseX without any problems. TagSoup ensures that only well-formed HTML arrives at the XML parser (correct opening and closing tags, etc.). Hence, if TagSoup is not available on a system, there will be a lot of cases where importing HTML fails, no matter whether you use the GUI or the standalone mode.  
+
If [http://vrici.lojban.org/~cowan/tagsoup/ TagSoup] is found in the [[Startup#Distributions|classpath]], HTML input is converted to well-formed XML documents.
  
===Installation===
+
If TagSoup is not available on a system, the default XML parser will be used, and the import will only succeed if the input is already well-formed XML (which is usually the case for XHTML documents).
  
If the TagSoup classes are accessible via the classpath, or if you run BaseX from the sources and use the Maven build manager, BaseX will automatically use TagSoup to prepare HTML input. TagSoup is also included in the complete BaseX distributions (BaseX.zip, BaseX.exe, etc.) or can be manually downloaded and embedded on the appropriate platforms. Using Debian, TagSoup will be automatically included after it has been installed via:
+
==Installation==
  
  apt-get install libtagsoup-java
+
===Downloads===
  
===GUI===
+
TagSoup is already included in the full BaseX distributions ({{Code|BaseX.zip}}, {{Code|BaseX.exe}}, etc.). It can also be manually downloaded and embedded on the appropriate platforms.
  
Go to Menu ''Database'' → ''New'' and select "HTML" in the input format combo box.
+
===Maven===
There's an info in the "Parsing" tab about whether TagSoup is available or not.
 
  
The same applied to the "Resources" tab in the "Database Properties" dialog.
+
An easy way to add TagSoup to your project is to as follows:
  
===Command Line===
+
1. Visit [https://mvnrepository.com/artifact/org.ccil.cowan.tagsoup/tagsoup/ MVN TagSoup Repository]
  
Turn on the HTML Parser before parsing documents, and set a file filter:
+
2. Click on the version you want
  
  SET [[Options#PARSER|PARSER]] html
+
3. On the first tab, you can see an XML snippet like this:
  SET [[Options#CREATEFILTER|CREATEFILTER]] *.html
 
  
===XQuery===
+
<pre lang="xml">
 
+
<dependency>
<pre class="brush:xquery">
+
  <groupId>org.ccil.cowan.tagsoup</groupId>
declare option db:parser "html";
+
  <artifactId>tagsoup</artifactId>
doc("index.html")
+
  <version>1.2.1</version>
 +
</dependency>
 
</pre>
 
</pre>
  
===TagSoup Options===
+
4. Insert the XML fragment into the <code><dependencies></code> element of your project’s <code>pom.xml</code> file.
  
TagSoup offers a variety of options to customize the import of HTML. For the complete list
+
===Debian===
please visit the [http://home.ccil.org/~cowan/XML/tagsoup/ TagSoup] website. BaseX supports
 
most of this options in general with a few exceptions:
 
  
* '''encoding''': BaseX tries to guess the input encoding but this can be overwritten by the user if necessary.
+
With Debian, TagSoup will automatically be detected and included after it has been installed via:
* '''files''': Not supported as input documents are piped directly to the XML parser.
 
* '''method''': Set to 'xml' as default. If this is set to 'html' for instance ending tags may be missing f.i.
 
* '''version''': Dismissed, as TagSoup always falls back to 'version 1.0', no matter what the input is.
 
* '''standalone''': Deactivated.
 
* '''pyx''': Not supported as the XML parser can't handle this kind of input.
 
* '''pyxin''': See pyx option.
 
* '''reuse''': Not supported.
 
* '''output-encoding''': Not supported, BaseX already takes care of that.
 
* '''help''': Not supported.
 
* '''version''': Not supported.
 
  
These options can be changed like any other option in BaseX, for example via
+
  apt-get install libtagsoup-java
XQuery:
 
  
<pre class="brush:xquery">
+
==Options==
declare option db:parser "html";
+
 
declare option db:htmlopt "html=false";
+
TagSoup offers a variety of options to customize the HTML conversion. For the complete list, please visit the [http://vrici.lojban.org/~cowan/tagsoup/ TagSoup] website. BaseX supports most options, with a few exceptions:
doc("index.html")
 
</pre>
 
  
And also via the [[Commands#SET|SET]] command, i.e.
+
* '''encoding''': BaseX tries to guess the input encoding, but this can be overwritten by this option.
 +
* '''files''': not supported as the input documents are piped directly to the XML parser.
 +
* '''method''': set to 'xml' as default. If this is set to 'html' ending tags may be missing for instance.
 +
* '''version''': dismissed, as TagSoup always falls back to 'version 1.0', no matter what the input is.
 +
* '''standalone''': deactivated.
 +
* '''pyx''', '''pyxin''': not supported as the XML parser can't handle this kind of input.
 +
* '''output-encoding''': not supported, BaseX already takes care of that.
 +
* '''reuse''', '''help''': not supported.
  
  SET [[Options#HTMLOPT|HTMLOPT]] method=xml
+
==GUI==
  
==JSON Parser==
+
Go to Menu ''Database'' → ''New'' and select "HTML" in the input format combo box. There's an info in the "Parsing" tab about whether TagSoup is available or not. The same applies to the "Resources" tab in the "Database Properties" dialog.
  
BaseX can also import JSON documents:
+
These two dialogs come with an input field 'Parameters' where TagSoup options can be entered. Keys and values are separated by the equal sign, multiple options are separated by commas, e.g.: <code>method=xml,nons=true</code>.
  
===GUI===
+
==Command Line==
  
Go to Menu ''Database'' → ''New'' and select "JSON" in the input format combo box.
+
Turn on the HTML Parser before parsing documents, set a file filter and optional parsing options:
You can set the following options for parsing JSON documents in the "Parsing" tab:
 
  
* '''Encoding''': Choose the appropriate encoding of the JSON file.
+
  SET {{Option|PARSER}} html
* '''JsonML''': Activate this option if the incoming file is a JsonML file.
+
  SET {{Option|CREATEFILTER}} *.html
 +
  SET {{Option|HTMLPARSER}} method=xml,nons=true
  
===Command Line===
+
==XQuery==
  
Turn on the JSON Parser before parsing documents, and set some optional, parser-specific options and a file filter:
+
The [[HTML Module]] provides XQuery functions for converting HTML to XML. In addition, all functions that accept [[Options#Parsing|Parsing Options]] can be used as well to convert HTML:
  
   SET [[Options#PARSER|PARSER]] json
+
<pre lang='xquery'>
   SET [[Options#PARSEROPT|PARSEROPT]] encoding=utf-8, jsonml=true
+
fetch:doc("index.html", map {
  SET [[Options#CREATEFILTER|CREATEFILTER]] *.json
+
   'parser': 'html',
 +
   'htmlparser': map { 'method': 'xml', 'nons': true() }
 +
})
 +
</pre>
  
==CSV Parser==
+
=JSON Parser=
  
BaseX can be used to import CSV documents. Different alternatives how to proceed are shown in the following:
+
With BaseX, JSON documents can be imported. The JSON parsing options are described in the documentation of the [[JSON Module]].
  
===GUI===
+
==GUI==
  
Go to Menu ''Database'' → ''New'' and select "CSV" in the input format combo box.
+
Go to Menu ''Database'' → ''New'' and select "JSON" in the input format combo box. You can set the following options for parsing JSON documents in the "Parsing" tab.
You can set the following options for parsing CSV documents in the "Parsing" tab:
 
  
* '''Encoding''': Choose the appropriate encoding of the CSV file.
+
==Command Line==
* '''Separator''': Choose the column separator of the CSV file (possible: <code>comma</code>, <code>semicolon</code>, <code>tab</code>).
 
* '''XML format''': Choose the XML format (possible: <code>verbose</code>, <code>simple</code>).
 
* '''Header''': Activate this option if the incoming CSV files have a header line.
 
  
===Command Line===
+
Turn on the JSON Parser before parsing documents, and set some optional, parser-specific options and a file filter:
  
Turn on the CSV Parser before parsing documents, and set some optional, parser-specific options and a file filter:
+
  SET {{Option|PARSER}} json
 +
  SET {{Option|JSONPARSER}} encoding=utf-8, jsonml=true
 +
  SET {{Option|CREATEFILTER}} *.json
  
  SET [[Options#PARSER|PARSER]] csv
+
==XQuery==
  SET [[Options#PARSEROPT|PARSEROPT]] encoding=utf-8, lines=true, format=verbose, header=false, separator=comma
 
  SET [[Options#CREATEFILTER|CREATEFILTER]] *.csv
 
  
===XQuery===
+
The [[JSON Module]] provides functions for converting JSON to XML.
  
The CSV parser can also be specified in the prolog of an XQuery expression.
+
In addition, all functions that accept [[Options#Parsing|Parsing Options]] can be used as well to convert JSON. The following query converts a JSON string to an XML document, using the conversion format {{Code|[[JSON Module#Basic|basic]]}}:
The [[Database Module#db:add|db:add()]] or [[Database Module#db:replace|db:replace()]]
 
function can be used to add the specified source files into the database.
 
The following example query adds all CSV files to the database <code>DB</code>
 
that are found in the directory <code>2Bimported</code>, and interprets the
 
first lines as column headers:
 
  
<pre class="brush:xquery">
+
<pre lang='xquery'>
declare option db:parser "csv";
+
db:create('json-db', '{ "key": "value" }', 'example.json', map {
declare option db:parseropt "header=yes";
+
  'parser': 'json',
for $file in file:list("2Bimported", false(), "*.csv")
+
  'jsonparser': map { 'format': 'basic' }
return db:add('DB', $file)
+
})
 
</pre>
 
</pre>
  
==Text Parser==
+
=CSV Parser=
  
Plain text can be imported as well:
+
BaseX can be used to import CSV documents. The CSV parsing options are described in the documentation of the [[CSV Module]].
  
===GUI===
+
==GUI==
  
Go to Menu ''Database'' → ''New'' and select "TEXT" in the input format combobox.
+
Go to Menu ''Database'' → ''New'' and select "CSV" in the input format combo box.
You can set the following option for parsing text documents in the "Parsing" tab:
+
You can set the following options for parsing CSV documents in the "Parsing" tab.
  
* '''Encoding''': Choose the appropriate encoding of the text file.
+
==Command Line==
* '''Lines''': Activate this option to create a <code>&lt;line&gt;...&lt;/line&gt;</code> element for each line of the input text file.
 
  
===Command Line===
+
Enable CSV Parsing before parsing documents, and set optional, parser-specific options and a file filter:
  
Turn on the CSV Parser before parsing documents and set some optional, parser-specific options and a file filter:
+
  SET {{Option|PARSER}} csv
 +
  SET {{Option|CSVPARSER}} encoding=utf-8,header=true,separator=semicolon
 +
  SET {{Option|CREATEFILTER}} *.csv
  
  SET [[Options#PARSER|PARSER]] text
+
==XQuery==
  SET [[Options#PARSEROPT|PARSEROPT]] lines=yes
 
  SET [[Options#CREATEFILTER|CREATEFILTER]] *
 
  
===XQuery===
+
The [[CSV Module]] provides a function for converting CSV to XML.
  
Again, the text parser can also be specified in the prolog of an XQuery expression, and
+
In addition, all functions that accept [[Options#Parsing|Parsing Options]] can be used as well to convert CSV. In the following, all CSV files from a directory are added to a database, with the first line of each file interpreted as column header:
the [[Database Module#db:add|db:add()]] or [[Database Module#db:replace|db:replace()]]
 
function can be used to add the specified source files into the database.
 
The following example query adds all text files to the database <code>DB</code>
 
that are found in the directory <code>2Bimported</code> and its sub-directories:
 
  
<pre class="brush:xquery">
+
<pre lang='xquery'>
declare option db:parser "text";
+
for $file in file:list("2Bimported", false(), "*.csv")
for $file in file:list("2Bimported", true(), "*.txt")
+
return db:add("csv-db", $file, "", map {
return db:add('DB', $file)
+
  'parser': 'csv',
 +
  'csvparser': map { 'header': true() }
 +
})
 
</pre>
 
</pre>
  
[[Category: Beginner]]
+
=Changelog=
 +
 
 +
;Version 11.0
 +
* Removed: Text Parser
 +
 
 +
;Version 7.8
 +
* Updated: parser options
 +
 
 +
;Version 7.7.2
 +
* Removed: CSV option "format"
 +
 
 +
;Version 7.3
 +
* Updated: the CSV {{Code|SEPARATOR}} option may now be assigned arbitrary single characters
 +
 
 +
;Version 7.2
 +
* Updated: Enhanced support for TagSoup options

Latest revision as of 17:39, 1 December 2023

This article is part of the Advanced User's Guide. It presents the available parsers that can be used to import various data sources in BaseX databases. Please visit the Serialization article if you want to know how to export data.

XML Parsers[edit]

BaseX provides two alternatives for parsing XML:

  • By default, Java’s SAXParser is used to parse XML documents.
  • The internal, built-in XML parser is more fault-tolerant than Java’s XML parser. It supports standard HTML entities out of the box, and it is faster than the default parser, in particular if small documents are to be parsed. However, the internal parser does not support the full range of DTD features and cannot resolve catalogs.

GUI[edit]

Go to Menu DatabaseNew, then choose the Parsing tab and (de)activate Use internal XML parser. The parsing of DTDs can be turned on/off by selecting the checkbox below.

Command Line[edit]

To turn the internal XML parser and DTD parsing on/off, modify the INTPARSE and DTD options:

 SET INTPARSE true
 SET DTD true

XQuery[edit]

The db:add and db:put functions can also be used to add new XML documents to the database. The following example query uses the internal XML parser and adds all files to the database DB that are found in the directory 2Bimported:

for $file in file:list("2Bimported")
return db:add('DB', $file, '', map { 'intparse': true() })

HTML Parser[edit]

If TagSoup is found in the classpath, HTML input is converted to well-formed XML documents.

If TagSoup is not available on a system, the default XML parser will be used, and the import will only succeed if the input is already well-formed XML (which is usually the case for XHTML documents).

Installation[edit]

Downloads[edit]

TagSoup is already included in the full BaseX distributions (BaseX.zip, BaseX.exe, etc.). It can also be manually downloaded and embedded on the appropriate platforms.

Maven[edit]

An easy way to add TagSoup to your project is to as follows:

1. Visit MVN TagSoup Repository

2. Click on the version you want

3. On the first tab, you can see an XML snippet like this:

<dependency>
  <groupId>org.ccil.cowan.tagsoup</groupId>
  <artifactId>tagsoup</artifactId>
  <version>1.2.1</version>
</dependency>

4. Insert the XML fragment into the <dependencies> element of your project’s pom.xml file.

Debian[edit]

With Debian, TagSoup will automatically be detected and included after it has been installed via:

 apt-get install libtagsoup-java

Options[edit]

TagSoup offers a variety of options to customize the HTML conversion. For the complete list, please visit the TagSoup website. BaseX supports most options, with a few exceptions:

  • encoding: BaseX tries to guess the input encoding, but this can be overwritten by this option.
  • files: not supported as the input documents are piped directly to the XML parser.
  • method: set to 'xml' as default. If this is set to 'html' ending tags may be missing for instance.
  • version: dismissed, as TagSoup always falls back to 'version 1.0', no matter what the input is.
  • standalone: deactivated.
  • pyx, pyxin: not supported as the XML parser can't handle this kind of input.
  • output-encoding: not supported, BaseX already takes care of that.
  • reuse, help: not supported.

GUI[edit]

Go to Menu DatabaseNew and select "HTML" in the input format combo box. There's an info in the "Parsing" tab about whether TagSoup is available or not. The same applies to the "Resources" tab in the "Database Properties" dialog.

These two dialogs come with an input field 'Parameters' where TagSoup options can be entered. Keys and values are separated by the equal sign, multiple options are separated by commas, e.g.: method=xml,nons=true.

Command Line[edit]

Turn on the HTML Parser before parsing documents, set a file filter and optional parsing options:

 SET PARSER html
 SET CREATEFILTER *.html
 SET HTMLPARSER method=xml,nons=true

XQuery[edit]

The HTML Module provides XQuery functions for converting HTML to XML. In addition, all functions that accept Parsing Options can be used as well to convert HTML:

fetch:doc("index.html", map {
  'parser': 'html',
  'htmlparser': map { 'method': 'xml', 'nons': true() }
})

JSON Parser[edit]

With BaseX, JSON documents can be imported. The JSON parsing options are described in the documentation of the JSON Module.

GUI[edit]

Go to Menu DatabaseNew and select "JSON" in the input format combo box. You can set the following options for parsing JSON documents in the "Parsing" tab.

Command Line[edit]

Turn on the JSON Parser before parsing documents, and set some optional, parser-specific options and a file filter:

 SET PARSER json
 SET JSONPARSER encoding=utf-8, jsonml=true
 SET CREATEFILTER *.json

XQuery[edit]

The JSON Module provides functions for converting JSON to XML.

In addition, all functions that accept Parsing Options can be used as well to convert JSON. The following query converts a JSON string to an XML document, using the conversion format basic:

db:create('json-db', '{ "key": "value" }', 'example.json', map {
  'parser': 'json',
  'jsonparser': map { 'format': 'basic' }
})

CSV Parser[edit]

BaseX can be used to import CSV documents. The CSV parsing options are described in the documentation of the CSV Module.

GUI[edit]

Go to Menu DatabaseNew and select "CSV" in the input format combo box. You can set the following options for parsing CSV documents in the "Parsing" tab.

Command Line[edit]

Enable CSV Parsing before parsing documents, and set optional, parser-specific options and a file filter:

 SET PARSER csv
 SET CSVPARSER encoding=utf-8,header=true,separator=semicolon
 SET CREATEFILTER *.csv

XQuery[edit]

The CSV Module provides a function for converting CSV to XML.

In addition, all functions that accept Parsing Options can be used as well to convert CSV. In the following, all CSV files from a directory are added to a database, with the first line of each file interpreted as column header:

for $file in file:list("2Bimported", false(), "*.csv")
return db:add("csv-db", $file, "", map {
  'parser': 'csv',
  'csvparser': map { 'header': true() }
})

Changelog[edit]

Version 11.0
  • Removed: Text Parser
Version 7.8
  • Updated: parser options
Version 7.7.2
  • Removed: CSV option "format"
Version 7.3
  • Updated: the CSV SEPARATOR option may now be assigned arbitrary single characters
Version 7.2
  • Updated: Enhanced support for TagSoup options