Difference between revisions of "Parsers"
Line 41: | Line 41: | ||
===Installation=== | ===Installation=== | ||
− | ===== | + | =====In your own project built with Maven===== |
− | An easy way to add TagSoupe to your own project is follow this steps : | + | An easy way to add TagSoupe to your own project is to follow this steps : |
+ | |||
1. visit [http://mvnrepository.com/artifact/org.ccil.cowan.tagsoup/tagsoup/ MVN TagSoup Repository] | 1. visit [http://mvnrepository.com/artifact/org.ccil.cowan.tagsoup/tagsoup/ MVN TagSoup Repository] | ||
− | 2. click on the version | + | |
− | 3. you can see on the first tab Maven an XML like : | + | 2. click on the version you want |
+ | |||
+ | 3. you can see on the first tab called Maven an XML like this : | ||
+ | |||
<dependency> | <dependency> | ||
<groupId>org.ccil.cowan.tagsoup</groupId> | <groupId>org.ccil.cowan.tagsoup</groupId> | ||
Line 51: | Line 55: | ||
<version>1.2.1</version> | <version>1.2.1</version> | ||
</dependency> | </dependency> | ||
+ | |||
4. copy that in your own maven project's pom.xml under the <dependencies> tag. | 4. copy that in your own maven project's pom.xml under the <dependencies> tag. | ||
+ | |||
+ | 5. don't forget to run <code>mvn jetty:run</code> again | ||
=====BaseX build with Maven build manager===== | =====BaseX build with Maven build manager===== | ||
If the TagSoup classes are accessible via the classpath, or if you run BaseX from the sources and use the Maven build manager, BaseX will automatically use TagSoup to prepare HTML input. | If the TagSoup classes are accessible via the classpath, or if you run BaseX from the sources and use the Maven build manager, BaseX will automatically use TagSoup to prepare HTML input. | ||
− | ===== | + | =====In BaseX distributions===== |
TagSoup is also included in the complete BaseX distributions (BaseX.zip, BaseX.exe, etc.) or can be manually downloaded and embedded on the appropriate platforms. | TagSoup is also included in the complete BaseX distributions (BaseX.zip, BaseX.exe, etc.) or can be manually downloaded and embedded on the appropriate platforms. | ||
− | ===== | + | =====On Debian===== |
Using Debian, TagSoup will be automatically included after it has been installed via: | Using Debian, TagSoup will be automatically included after it has been installed via: | ||
Revision as of 15:13, 3 August 2012
This article is part of the Getting Started Section. It presents different parsers for importing various data source into BaseX databases. For export see Serialization.
XML Parsers
BaseX provides two parsers to import XML data:
- By default, the internal, built-in XML parser is used, which is more fault-tolerant than Java’s XML parser. It supports standard HTML entities out-of-the-box, and is faster in most cases. In turn, it does not support all oddities specified by DTDs, and cannot resolve catalogs.
- Java’s SAXParser can also be selected for parsing XML documents. This parser is stricter than the built-in parser, but it refuses to process some large documents.
GUI
Go to Menu Database → New, then choose the Parsing tab and (de)activate Use internal XML parser. The parsing of DTDs can be turned on/off by selecting the checkbox below.
Command Line
To turn the internal XML parser and DTD parsing on/off, modfify the INTPARSE
and DTD
options:
SET INTPARSE true SET DTD true
XQuery
The db:add() or db:replace()
function can be used as well to add new XML documents to the database.
The following example query uses the internal parser and adds all files
to the database DB
that are found in the directory
2Bimported
:
declare option db:intparse "yes"; for $file in file:list("2Bimported") return db:add('DB', $file)
HTML Parser
With TagSoup, HTML can be imported in BaseX without any problems. TagSoup ensures that only well-formed HTML arrives at the XML parser (correct opening and closing tags, etc.). Hence, if TagSoup is not available on a system, there will be a lot of cases where importing HTML fails, no matter whether you use the GUI or the standalone mode.
Installation
In your own project built with Maven
An easy way to add TagSoupe to your own project is to follow this steps :
1. visit MVN TagSoup Repository
2. click on the version you want
3. you can see on the first tab called Maven an XML like this :
<dependency> <groupId>org.ccil.cowan.tagsoup</groupId> <artifactId>tagsoup</artifactId> <version>1.2.1</version> </dependency>
4. copy that in your own maven project's pom.xml under the <dependencies> tag.
5. don't forget to run mvn jetty:run
again
BaseX build with Maven build manager
If the TagSoup classes are accessible via the classpath, or if you run BaseX from the sources and use the Maven build manager, BaseX will automatically use TagSoup to prepare HTML input.
In BaseX distributions
TagSoup is also included in the complete BaseX distributions (BaseX.zip, BaseX.exe, etc.) or can be manually downloaded and embedded on the appropriate platforms.
On Debian
Using Debian, TagSoup will be automatically included after it has been installed via:
apt-get install libtagsoup-java
Options
TagSoup offers a variety of options to customize the import of HTML. For the complete list please visit the TagSoup website. BaseX supports most of these options with a few exceptions:
- encoding: BaseX tries to guess the input encoding but this can be overwritten by the user if necessary.
- files: Not supported as input documents are piped directly to the XML parser.
- method: Set to 'xml' as default. If this is set to 'html' ending tags may be missing for instance.
- version: Dismissed, as TagSoup always falls back to 'version 1.0', no matter what the input is.
- standalone: Deactivated.
- pyx, pyxin: Not supported as the XML parser can't handle this kind of input.
- output-encoding: Not supported, BaseX already takes care of that.
- reuse, help: Not supported.
GUI
Go to Menu Database → New and select "HTML" in the input format combo box. There's an info in the "Parsing" tab about whether TagSoup is available or not. The same applies to the "Resources" tab in the "Database Properties" dialog.
These two dialogs come with an input field 'Parameters' where TagSoup options can be entered.
Command Line
Turn on the HTML Parser before parsing documents, and set a file filter:
SET PARSER html SET HTMLOPT method=xml,nons=true,ncdata=true,nodefaults=true,nobogons=true,nocolons=true,ignorable=true SET CREATEFILTER *.html
XQuery
declare option db:parser "html"; declare option db:htmlopt "html=false"; doc("index.html")
JSON Parser
BaseX can also import JSON documents:
GUI
Go to Menu Database → New and select "JSON" in the input format combo box. You can set the following options for parsing JSON documents in the "Parsing" tab:
- Encoding: Choose the appropriate encoding of the JSON file.
- JsonML: Activate this option if the incoming file is a JsonML file.
Command Line
Turn on the JSON Parser before parsing documents, and set some optional, parser-specific options and a file filter:
SET PARSER json SET PARSEROPT encoding=utf-8, jsonml=true SET CREATEFILTER *.json
CSV Parser
BaseX can be used to import CSV documents. Different alternatives how to proceed are shown in the following:
GUI
Go to Menu Database → New and select "CSV" in the input format combo box. You can set the following options for parsing CSV documents in the "Parsing" tab:
- Encoding: Choose the appropriate encoding of the CSV file.
- Separator: Choose the column separator of the CSV file. Possible:
comma
,semicolon
,tab
orspace
or an arbitrary character. - XML format: Choose the XML format. Possible:
verbose
,simple
. - Header: Activate this option if the incoming CSV files have a header line.
Command Line
Turn on the CSV Parser before parsing documents, and set some optional, parser-specific options and a file filter. Since Version 7.3, Unicode code points can be specified as separators; 32
is the code point for spaces:
SET PARSER csv SET PARSEROPT encoding=utf-8, lines=true, format=verbose, header=false, separator=32 SET CREATEFILTER *.csv
XQuery
The CSV parser can also be specified in the prolog of an XQuery expression.
The db:add() or db:replace()
function can be used to add the specified source files into the database.
The following example query adds all CSV files to the database DB
that are found in the directory 2Bimported
, and interprets the
first lines as column headers:
declare option db:parser "csv"; declare option db:parseropt "header=yes"; for $file in file:list("2Bimported", false(), "*.csv") return db:add('DB', $file)
Text Parser
Plain text can be imported as well:
GUI
Go to Menu Database → New and select "TEXT" in the input format combobox. You can set the following option for parsing text documents in the "Parsing" tab:
- Encoding: Choose the appropriate encoding of the text file.
- Lines: Activate this option to create a
<line>...</line>
element for each line of the input text file.
Command Line
Turn on the CSV Parser before parsing documents and set some optional, parser-specific options and a file filter:
SET PARSER text SET PARSEROPT lines=yes SET CREATEFILTER *
XQuery
Again, the text parser can also be specified in the prolog of an XQuery expression, and
the db:add() or db:replace()
function can be used to add the specified source files into the database.
The following example query adds all text files to the database DB
that are found in the directory 2Bimported
and its sub-directories:
declare option db:parser "text"; for $file in file:list("2Bimported", true(), "*.txt") return db:add('DB', $file)
Changelog
- Version 7.3
- Updated: SEPARATOR option of CSV parser may now contain arbitrary code points.
- Version 7.2
- Updated: Enhanced support for TagSoup options.