HTTP Client Functions

This module contains a single function to send HTTP requests and handle HTTP responses. The function send-request is based on the EXPath HTTP Client Module. It gives full control over the available request and response parameters. For simple GET requests, the Fetch Functions may be sufficient.

Despite the similar name, the Client Module is unrelated: it addresses BaseX server instances, whereas this module sends generic HTTP requests.

Please note that BaseX provides extensions to the specification:

• The cookies attribute of the http:request element enables cookie handling for the requests of a query.
• The csv, json and html attributes of the http:request element control how response bodies are converted (see Request Attributes).
• Binary response bodies of GET requests are returned as lazy items, and file-based request payloads are streamed.
• If <http:header name="Accept-Encoding" value="gzip"/> is specified and if the addressed web server provides support for the gzip compression algorithm, the response will automatically be decompressed.

Since BaseX 10, the module is based on the Java HTTP Client, which provides a better overall performance, uses internal connection pools and follows redirects across different protocols (http, https).

Conventions

All functions are in the http://expath.org/ns/http-client namespace, to which the http prefix is statically bound.

All errors are in the http://expath.org/ns/error namespace, to which the experr prefix is statically bound.

Request Attributes

The following attributes can be attached to the http:request element. All of them are defined by the EXPath specification, except for cookies, csv, json and html:

Functions

http:send-request

Updated: Binary response bodies of GET requests are returned as lazy items; file-based request payloads are streamed; new cookies attribute.

http:send-request(  $request  as element(http:request)?,  $href     as xs:string?  := (),  $bodies   as item()*  := ()) as item()+create

Examples

Streaming

As the ZIP archive has a binary media type, the response body is returned as lazy item, and its contents are streamed to a file with constant memory consumption:

let $response := http:send-request(
  <http:request method='get'/>,
  'https://files.basex.org/xml/xmark111mb.zip'
)
return file:write-binary('output.zip', $response[2])

With the override-media-type attribute set to application/octet-stream, this behavior can be enforced for arbitrary GET requests.

Uploads work equally well with constant memory: The contents of the addressed file will be streamed to the target server:

http:send-request(
  <http:request method='put'>
    <http:body media-type='application/octet-stream'/>
  </http:request>,
  'http://localhost:8080/rest/archive/huge.zip',
  file:read-binary('huge.zip')
)

Status Only

Simple GET request. As the attribute status-only is set to true, only the response element is returned.

http:send-request(<http:request method='get' status-only='true'/>, 'http://basex.org')

<http:response status="200" message="OK">
  <http:header name="Date" value="Mon, 14 Mar 2011 20:55:53 GMT"/>
  <http:header name="Content-Length" value="12671"/>
  <http:header name="Expires" value="Mon, 14 Mar 2011 20:57:23 GMT"/>
  <http:header name="Set-Cookie" value="fe_typo_user=d10c955; path=/"/>
  <http:header name="Connection" value="close"/>
  <http:header name="Content-Type" value="text/html; charset=utf-8"/>
  <http:header name="Server" value="Apache/2.2.16"/>
  <http:header name="X-Powered-By" value="PHP/5.3.5"/>
  <http:header name="Cache-Control" value="max-age=90"/>
  <http:body media-type="text/html; charset=utf-8"/>
</http:response>

Cookies

With the cookies attribute, the cookies that a server assigns are attached to the subsequent requests of a query. As the cookies are also sent to redirected addresses, a login that ends with a redirect can be followed by requests to protected resources:

void(
  http:send-request(
    <http:request method='post' cookies='true' href='https://example.com/login'>
      <http:body media-type='application/x-www-form-urlencoded'/>
    </http:request>,
    (),
    'user=jack&password=...'
  )
),
http:send-request(
  <http:request method='get' cookies='true'/>,
  'https://example.com/download'
)

Google Homepage

Retrieve the Google search home page with a timeout of 10 seconds. In order to parse HTML, the HTML parser must be contained in the class path.

http:send-request(<http:request method='get' href='http://www.google.com' timeout='10'/>)

<http:response status="200" message="OK">
  <http:header name="Date" value="Mon, 14 Mar 2011 22:03:25 GMT"/>
  <http:header name="Transfer-Encoding" value="chunked"/>
  <http:header name="Expires" value="-1"/>
  <http:header name="X-XSS-Protection" value="1; mode=block"/>
  <http:header name="Set-Cookie" value="...; expires=Tue, 13-Sep-2011 22:03:25 GMT; ..."/>
  <http:header name="Content-Type" value="text/html; charset=ISO-8859-1"/>
  <http:header name="Server" value="gws"/>
  <http:header name="Cache-Control" value="private, max-age=0"/>
  <http:body media-type="text/html; charset=ISO-8859-1"/>
</http:response>
<html xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
    <title>Google</title>
    ...
  </head>
  <body>
    ...
  </body>
</html>

The response content type can also be overwritten in order to retrieve HTML pages and other textual data as plain string (using text/plain) or in its binary representation (using application/octet-stream). With the http:header element, a custom user agent can be set. See the following example:

let $response := http:send-request(
  <http:request method='get'
     override-media-type='application/octet-stream'
     href='http://www.google.com'>
    <http:header name='User-Agent' value='Opera'/>
  </http:request>
)
let $body := tail($response)
return try {
  html:parse($body)
} catch * {
  'Conversion to XML failed: ' || $err:description
}

SVG Data

Content-type ending with +xml, e.g. image/svg+xml.

http:send-request(
  <http:request method='get'/>,
  'http://upload.wikimedia.org/wikipedia/commons/6/6b/Bitmap_VS_SVG.svg'
)

<http:response status="200" message="OK">
  <http:header name="ETag" value="W/&quot;11b6d-4ba15ed4&quot;"/>
  <http:header name="Age" value="9260"/>
  <http:header name="Date" value="Mon, 14 Mar 2011 19:17:10 GMT"/>
  <http:header name="Content-Length" value="72557"/>
  <http:header name="Last-Modified" value="Wed, 17 Mar 2010 22:59:32 GMT"/>
  <http:header name="Content-Type" value="image/svg+xml"/>
  <http:header name="X-Cache-Lookup" value="MISS from knsq22.knams.wikimedia.org:80"/>
  <http:header name="Connection" value="keep-alive"/>
  <http:header name="Server" value="Sun-Java-System-Web-Server/7.0"/>
  <http:header name="X-Cache" value="MISS from knsq22.knams.wikimedia.org"/>
  <http:body media-type="image/svg+xml"/>
</http:response>
<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"
    version="1.1" height="638">
  <defs>
    <linearGradient id="lg0">
      <stop stop-color="#3333ff" offset="0"/>
      <stop stop-color="#3f3fff" stop-opacity="0" offset="1"/>
    </linearGradient>
    ...
</svg>

POST Request

POST request to the BaseX REST Service, specifying a username and password.

http:send-request(
  <http:request method='post' username='admin' password='...'>
    <http:body media-type='application/xml'/>
  </http:request>,
  'http://localhost:8080/rest',
  <query>
    <text>
      <html>{
        for $i in 1 to 3
        return 'Section ' || $i
      }</html>
    </text>
  </query>
)

<http:response xmlns:http="http://expath.org/ns/http-client" status="200" message="OK">
  <http:header name="Content-Length" value="135"/>
  <http:header name="Content-Type" value="application/xml"/>
  <http:header name="Server" value="Jetty(11.0.20)"/>
  <http:body media-type="application/xml"/>
</http:response>
<html>
  Section 1
  Section 2
  Section 3
</html>

File Upload

Performs an HTML file upload. In the RESTXQ code, the uploaded file is written to the temporary directory:

let $path := 'file-to-be.uploaded'
return http:send-request(
  <http:request method='POST'>
    <http:multipart media-type='multipart/form-data'>
      <http:header name='content-disposition'
        value='form-data; name="files"; filename="{ file:name($path) }"'/>
      <http:body media-type='application/octet-stream'/>
    </http:multipart>
  </http:request>,
  'http://localhost:8080/write-to-temp',
  file:read-binary($path)
)

declare
  %rest:POST
  %rest:path('/write-to-temp')
  %rest:form-param('files', '{ $files }')
function local:file-upload(
  $files  as map(xs:string, xs:base64Binary)
) as empty-sequence() {
  map:for-each($files, fn($file, $content) {
    file:write-binary(file:temp-dir() || $file, $content)
  })
};

Response Conversion

CSV, JSON and HTML responses are automatically converted to an XML representation. The target format can be influenced by supplying csv, json and html attributes:

http:send-request(
  <http:request method='GET' json='format=w3,lax=true'/>,
  'http://localhost:8080/json'
)

{ "abcde": 12345 }

Without the json attribute, the response body is converted to the default XML representation:

<json type="object">
  <abcde>12345</abcde>
</json>

declare
  %rest:path('json')
  %output:method('json')
function local:json() {
  { 'abcde': 12345 }
};

See the CSV Functions, JSON Functions and HTML Functions for a list of the available options.

Errors

Changelog

• Added: http:send-request: cookies attribute for collecting and resending cookies.
• Updated: http:send-request: binary response bodies of GET requests are returned as lazy items; file-based request payloads are streamed.

• Updated: http:send-request: csv, json and html attributes added.

• Updated: Implementation based on the new Java HTTP Client.

• Updated: support for gzipped content encoding

• Added: digest authentication

• Updated: http:send-request: HC0002 is raised if the input cannot be parsed or converted to the final data type.
• Updated: errors are using text/plain as media-type.