HTML Functions

This module provides functions for converting HTML to XML. Conversion will only take place if TagSoup or Validator.nu is included in the classpath. See Parsers if you want to import HTML documents into a database.

Conventions

All functions and errors are in the http://basex.org/modules/html namespace, to which the html prefix is statically bound.

Parsers

Updated: If no parser is available, the HTML input will be wrapped in an html element.

The HTML parser can be selected with the BaseX-specific method option (see HTMLPARSER for the equivalent database option). It is not to be confused with TagSoup's own method option, which is listed as unsupported below:

• method=tagsoup: uses the TagSoup parser.
• method=nu: uses the Validator.nu HTML parser.

If this option is explicitly set, the corresponding parser must be available in the classpath. The selected parser will convert HTML input into well-formed XML documents.

By default, TagSoup is used for HTML processing. If TagSoup is unavailable in the classpath, Validator.nu will be used instead. If neither parser is available, a document with an html root element will be returned, containing the HTML input as child text. The name of the default parser is returned by html:parser.

Downloads

TagSoup and Validator.nu are included in the full distributions of BaseX. They can also be manually downloaded and added to the classpath.

Maven

An easy way to add TagSoup to your project is as follows:

• Visit MVN TagSoup Repository
• Click on the version you want
• On the first tab, you can see an XML snippet like this:

<dependency>
  <groupId>org.ccil.cowan.tagsoup</groupId>
  <artifactId>tagsoup</artifactId>
  <version>1.2.1</version>
</dependency>

• Insert the XML fragment into the <dependencies> element of your project’s pom.xml file.

For Validator.nu, go to MVN Validator.nu Repository, select the version, and find a snippet like the following for inclusion into your project:

<dependency>
  <groupId>nu.validator</groupId>
  <artifactId>htmlparser</artifactId>
  <version>1.4.16</version>
</dependency>

Debian

With Debian, TagSoup will automatically be detected and included after it has been installed via:

apt-get install libtagsoup-java

Validator.nu can be installed with this command:

apt-get install libhtml5parser-java

Options

TagSoup

TagSoup offers a variety of options to customize the HTML conversion. For the complete list, please visit the TagSoup website. BaseX supports most options, with a few exceptions:

• encoding: BaseX tries to guess the input encoding; this option overrides the detected value.
• files: not supported, as the input documents are piped directly to the XML parser.
• html, method: not supported, as they are only used to make TagSoup output HTML.
• version: not supported, as TagSoup always falls back to version 1.0, no matter what the input is.
• standalone: not supported.
• pyx, pyxin: not supported, as the XML parser cannot handle this kind of input.
• output-encoding: not supported; BaseX already takes care of that.
• reuse, help: not supported.

Validator.nu

Added: fail-on-error option.

The following list shows the names to use for setting Validator.nu options (for details of these options, see the corresponding description in the Validator.nu HtmlParser Javadoc):

• comment-policy
• content-non-xml-char-policy
• content-space-policy
• encoding
• fail-on-error: if enabled, a parsing error will be raised instead of being recovered from (yes, no; boolean if supplied in a map; default: no)
• heuristics
• mapping-lang-to-xml-lang
• name-policy
• scripting-enabled
• streamability-violation-policy
• unicode-normalization-checking
• xml-policy
• xmlns-policy

Option heuristics has extra classpath requirements, depending on its value:

• heuristics=icu: requires class com.ibm.icu.text.CharsetDetector from com.ibm.icu:icu4j. This is included in the full distributions of BaseX.
• heuristics=chardet: requires class org.mozilla.intl.chardet.nsICharsetDetectionObserver from net.sourceforge.jchardet:jchardet:1.0. This is not included in a BaseX distribution.

When the respective class is unavailable, using either of these option values will result in an error message.

Functions

html:doc

html:doc(  $source   as xs:string?,  $options  as map(*)?  := {}) as document-node()?create

html:parse

html:parse(  $value    as (xs:string|xs:base64Binary|xs:hexBinary)?,  $options  as map(*)?  := {}) as document-node()?

html:parser

html:parser() as xs:string

Examples

Basic Example

The following query converts the specified string to an XML document node.

html:parse("<html>")

<html/>

Specifying Options

The next query creates an XML document with namespaces:

html:parse("<a href='ok.html'/>", { 'nons': false() })

<html xmlns="http://www.w3.org/1999/xhtml">
  <body>
    <a shape="rect" href="ok.html"/>
  </body>
</html>

Parsing Binary Input

If the input encoding is unknown, the data to be processed can be passed on in its binary representation. The HTML parser will automatically try to detect the correct encoding:

html:parse(fetch:binary("https://en.wikipedia.org"))

<html xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <title>Wikipedia, the free encyclopedia</title>
    <meta charset="UTF-8"/>
    ...

Changelog

• Added: Validator.nu option fail-on-error.
• Updated: If no parser is available, the HTML input will be wrapped in an html element.

• Added: support for using Validator.nu

• Added: html:doc

• Updated: error codes updated; errors now use the module namespace

• Added: New module added.