Main Page » XQuery » Functions » HTTP Client Functions

HTTP Client Functions

This module contains a single function to send HTTP requests and handle HTTP responses. The function send-request is based on the EXPath HTTP Client Module. It gives full control over the available request and response parameters. For simple GET requests, the Fetch Functions may be sufficient.

If <http:header name="Accept-Encoding" value="gzip"/> is specified and if the addressed web server provides support for the gzip compression algorithm, the response will automatically be decompressed.

Please note that BaseX provides extensions to the specification:

Updated: csv, json and html parser options can be supplied to influence the conversion of the response.

Since BaseX 10, the module is based on the Java HTTP Client, which provides a better overall performance, uses internal connection pools and follows redirects across different protocols (http, https).

Conventions

All functions in this module are assigned to the http://expath.org/ns/http-client namespace, which is statically bound to the http prefix.

All errors are assigned to the http://expath.org/ns/error namespace, which is statically bound to the experr prefix.

Functions

http:send-request

Updated: csv, json and html attributes added.

Signature
http:send-request(
  $request  as element(http:request)?,
  $href     as xs:string?              := (),
  $bodies   as item()*                 := ()
) as item()+
SummarySends an HTTP request and interprets the corresponding response:
  • $request contains an <http:request/> element with a method attribute, an href attribute with the target URI, and optional header and body elements.
  • The request is either sent to the URI of the $href argument or (if empty) to the URI supplied via the href attribute.
  • In addition to the attributes of the official specification, csv, json, html and text attributes can be supplied to define how to convert the response body (see Response Conversion for an example).

Notes:

  • Both basic and digest authentication is supported.
  • While the contents of the request can be supplied as child of the http:body element, it is faster and safer to pass them on via the third argument.
  • Certificate verification can be globally disabled via the IGNORECERT option.
  • For further information, please check out the EXPath specification.
Errors
HC0001An HTTP error occurred.
HC0002Error parsing the entity content as XML or HTML.
HC0003With a multipart response, the override-media-type must be either a multipart media type or application/octet-stream.
HC0004The src attribute on the body element is mutually exclusive with all other attribute (except the media-type).
HC0005The request element is not valid.
HC0006A timeout occurred waiting for the response.

Examples

Status Only

Simple GET request. As the attribute status-only is set to true, only the response element is returned.

Query 
http:send-request(<http:request method='get' status-only='true'/>, 'http://basex.org')
Result 
<http:response status="200" message="OK">
  <http:header name="Date" value="Mon, 14 Mar 2011 20:55:53 GMT"/>
  <http:header name="Content-Length" value="12671"/>
  <http:header name="Expires" value="Mon, 14 Mar 2011 20:57:23 GMT"/>
  <http:header name="Set-Cookie" value="fe_typo_user=d10c955; path=/"/>
  <http:header name="Connection" value="close"/>
  <http:header name="Content-Type" value="text/html; charset=utf-8"/>
  <http:header name="Server" value="Apache/2.2.16"/>
  <http:header name="X-Powered-By" value="PHP/5.3.5"/>
  <http:header name="Cache-Control" value="max-age=90"/>
  <http:body media-type="text/html; charset=utf-8"/>
</http:response>

Google Homepage

Retrieve the Google search home page with a timeout of 10 seconds. In order to parse HTML, TagSoup must be contained in the class path.

Query 
http:send-request(<http:request method='get' href='http://www.google.com' timeout='10'/>)
Result 
<http:response status="200" message="OK">
  <http:header name="Date" value="Mon, 14 Mar 2011 22:03:25 GMT"/>
  <http:header name="Transfer-Encoding" value="chunked"/>
  <http:header name="Expires" value="-1"/>
  <http:header name="X-XSS-Protection" value="1; mode=block"/>
  <http:header name="Set-Cookie" value="...; expires=Tue, 13-Sep-2011 22:03:25 GMT; ..."/>
  <http:header name="Content-Type" value="text/html; charset=ISO-8859-1"/>
  <http:header name="Server" value="gws"/>
  <http:header name="Cache-Control" value="private, max-age=0"/>
  <http:body media-type="text/html; charset=ISO-8859-1"/>
</http:response>
<html xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
    <title>Google</title>
    ...
  </body>
</html>

The response content type can also be overwritten in order to retrieve HTML pages and other textual data as plain string (using text/plain) or in its binary representation (using application/octet-stream). With the http:header element, a custom user agent can be set. See the following example:

Query 
let $response := http:send-request(
  <http:request method='get'
     override-media-type='application/octet-stream'
     href='http://www.google.com'>
    <http:header name='User-Agent' value='Opera'/>
  </http:request>
)
let $body := tail($response)
return try {
  html:parse($body)
} catch * {
  'Conversion to XML failed: ' || $err:description
}

SVG Data

Content-type ending with +xml, e.g. image/svg+xml.

Query 
http:send-request(
  <http:request method='get'/>,
  'http://upload.wikimedia.org/wikipedia/commons/6/6b/Bitmap_VS_SVG.svg'
)
Result 
<http:response status="200" message="OK">
  <http:header name="ETag" value="W/"11b6d-4ba15ed4""/>
  <http:header name="Age" value="9260"/>
  <http:header name="Date" value="Mon, 14 Mar 2011 19:17:10 GMT"/>
  <http:header name="Content-Length" value="72557"/>
  <http:header name="Last-Modified" value="Wed, 17 Mar 2010 22:59:32 GMT"/>
  <http:header name="Content-Type" value="image/svg+xml"/>
  <http:header name="X-Cache-Lookup" value="MISS from knsq22.knams.wikimedia.org:80"/>
  <http:header name="Connection" value="keep-alive"/>
  <http:header name="Server" value="Sun-Java-System-Web-Server/7.0"/>
  <http:header name="X-Cache" value="MISS from knsq22.knams.wikimedia.org"/>
  <http:body media-type="image/svg+xml"/>
</http:response>
<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"
    version="1.1" height="638">
  <defs>
    <linearGradient id="lg0">
      <stop stop-color="#3333ff" offset="0"/>
      <stop stop-color="#3f3fff" stop-opacity="0" offset="1"/>
    </linearGradient>
    ...
</svg>

POST Request

POST request to the BaseX REST Service, specifying a username and password.

Query 
http:send-request(
  <http:request method='post' username='admin' password='admin'>
    <http:body media-type='application/xml'/>
  </http:request>,
  'http://localhost:8080/rest',
  <query>
    <text>
      <html>{
        for $i in 1 to 3
        return Section { $i }
      }</html>
    </text>
  </query>
)
Result 
<http:response xmlns:http="http://expath.org/ns/http-client" status="200" message="OK">
  <http:header name="Content-Length" value="135"/>
  <http:header name="Content-Type" value="application/xml"/>
  <http:header name="Server" value="Jetty(11.0.20)"/>
  <http:body media-type="application/xml"/>
</http:response>
<html>
  Section 1
  Section 2
  Section 3
</html>

File Upload

Performs an HTML file upload. In the RESTXQ code, the uploaded file is written to the temporary directory:

Query 
let $path := 'file-to-be.uploaded'
return http:send-request(
  <http:request method='POST'>
    <http:multipart media-type='multipart/form-data'>
      <http:header name='content-disposition'
        value='form-data; name="files"; filename="{ file:name($path) }"'/>
      <http:body media-type='application/octet-stream'/>
    </http:multipart>
  </http:request>,
  'http://localhost:8080/write-to-temp',
  file:read-binary($path)
)
RESTXQ service 
declare
  %rest:POST
  %rest:path('/write-to-temp')
  %rest:form-param('files', '{ $files }')
function dba:file-upload(
  $files  as map(xs:string, xs:base64Binary)
) as empty-sequence() {
  map:for-each($files, fn($file, $content) {
    file:write-binary(file:temp-dir() || $file, $content)
  });
};

Response Conversion

CSV, JSON and HTML responses are automatically converted to an XML representation. The target format can be influenced by supplying csv, json and html attributes:

Query 
http:send-request(
  <http:request method='GET' json='format=xquery,lax=true'/>,
  'http://localhost:8080/json'
)
Result 
{ "abcde": 12345 }

Without the json attribute, the response body is converted to the default XML representation:

<json type="object">
  <abcde>12345</abcde>
</json>
RESTXQ service 
declare
  %rest:path('json')
  %output:method('json')
function local:json() {
  { 'abcde': 12345 }
};

See the CSV Functions, JSON Functions and HTML Functions for a list of the available options.

Errors

CodeDescription
HC0001An HTTP error occurred.
HC0002Error parsing the entity content as XML or HTML.
HC0003With a multipart response, the override-media-type must be either a multipart media type or application/octet-stream.
HC0004The src attribute on the body element is mutually exclusive with all other attribute (except the media-type).
HC0005The request element is not valid.
HC0006A timeout occurred waiting for the response.

Changelog

Version 11.0Version 10.0Version 9.0
  • Updated: support for gzipped content encoding
Version 8.0
  • Added: digest authentication
Version 7.6
  • Updated: http:send-request: HC0002 is raised if the input cannot be parsed or converted to the final data type.
  • Updated: errors are using text/plain as media-type.
⚡Generated with XQuery
 
Login | Changelog | TOC | Imprint
HTTP Client Functions