HTTP Client Functions

This module contains functions to send HTTP requests and handle HTTP responses:

• http:get, http:post and the other functions provide a map-based interface: requests are controlled by typed options, and the response is returned as a single record.
• http:send-request is based on the EXPath HTTP Client Module.

Both interfaces can be used side by side. See Extensions for the features that BaseX adds to the specification. For simple GET requests, the Fetch Functions may be sufficient.

Despite the similar name, the Client Module is unrelated: it addresses BaseX server instances, whereas this module sends generic HTTP requests.

Conventions

All functions are in the http://expath.org/ns/http-client namespace, to which the http prefix is statically bound.

The errors of http:send-request are in the http://expath.org/ns/error namespace, to which the experr prefix is statically bound. The errors of the other functions are in the namespace of the module.

Functions

http:delete

Added: New function.

http:delete(  $href     as xs:string,  $options  as map(*)?  := {}) as http:response-recordcreate

http:get

Added: New function.

http:get(  $href     as xs:string,  $options  as map(*)?  := {}) as http:response-recordcreate

http:get('https://basex.org', { 'response-body': 'none' })?status

http:head

Added: New function.

http:head(  $href     as xs:string,  $options  as map(*)?  := {}) as http:response-recordcreate

http:options

Added: New function.

http:options(  $href     as xs:string,  $options  as map(*)?  := {}) as http:response-recordcreate

http:patch

Added: New function.

http:patch(  $href     as xs:string,  $body     as item()*  := (),  $options  as map(*)?  := {}) as http:response-recordcreate

http:post

Added: New function.

http:post(  $href     as xs:string,  $body     as item()*  := (),  $options  as map(*)?  := {}) as http:response-recordcreate

http:put

Added: New function.

http:put(  $href     as xs:string,  $body     as item()*  := (),  $options  as map(*)?  := {}) as http:response-recordcreate

http:query

Added: New function.

http:query(  $href     as xs:string,  $body     as item()*  := (),  $options  as map(*)?  := {}) as http:response-recordcreate

http:send

Added: New function.

http:send(  $href     as xs:string,  $method   as xs:string,  $body     as item()*  := (),  $options  as map(*)?  := {}) as http:response-recordcreate

http:send-request

Updated: New cookies, proxy, verify and xml attributes; follow-redirect accepts a maximum number of redirects; auth-method is case-insensitive; timeout accepts fractions of a second; href and version attributes in the response element.

http:send-request(  $request  as element(http:request)?,  $href     as xs:string?  := (),  $bodies   as item()*  := ()) as item()+create

Request Options

The following options can be supplied via the $options argument of http:get, http:post and the other functions with a map-based interface:

Some options are supplied as records. Their types are known in the static context, so they can also be used in type declarations. Entries that are not declared are rejected, and entries with a ? in their type can be omitted:

declare record http:auth-record(
  username    as xs:string,
  password    as xs:string,
  method      as xs:string?,      (: 'basic' or 'digest'; default: 'basic' :)
  preemptive  as xs:boolean?      (: default: false() :)
);

declare record http:parse-options-record(
  xml   as map(*)?,
  html  as map(*)?,
  json  as map(*)?,
  csv   as map(*)?
);

declare record http:part-record(
  headers  as map(xs:string, xs:string*)?,
  body     as item()?
);

A function that processes a part can thus be declared as follows:

declare function local:name($part as http:part-record) as xs:string? {
  $part?headers?Content-Disposition
};

Certificates

The certificates option is a map with two optional entries. Both are maps that reference a key store:

keystore is the path or URL of the key store. Files with the suffix .jks are read as Java key stores, all others as PKCS #12 files (.p12, .pfx). keystore-password unlocks the key store, and password unlocks the private key in the client store. The alias entry for selecting a key is not supported yet:

http:get('https://internal.example.com/api', {
  'certificates': {
    'trust': { 'keystore': 'certs/ca.p12', 'keystore-password': 'changeit' },
    'client': { 'keystore': 'certs/client.p12', 'keystore-password': 'secret', 'password': 'secret' }
  }
})

Response Record

These functions return a record of the type http:response-record, which can equally be used in type declarations. All entries are always present:

Request 1.0 Attributes

The following attributes can be attached to the http:request element. All of them are defined by the EXPath specification, except for cookies, proxy, verify, xml, csv, json and html:

Examples

JSON API

A map is sent as JSON body, with credentials that are supplied before the server asks for them. The JSON response is parsed, and duplicate keys are rejected:

let $response := http:post(
  'https://example.com/api/items',
  { 'name': 'Anna', 'tags': [ 'red', 'blue' ] },
  {
    'auth': { 'username': 'john', 'password': '****', 'preemptive': true() },
    'parse-options': { 'json': { 'duplicates': 'reject' } }
  }
)
return if($response?status = 201) then $response?body?id else error(
  xs:QName('local:api'), 'Item not created: ' || $response?status
)

Query Parameters and Headers

Parameters are appended to the URI, and the response body is returned as string instead of being parsed. An empty sequence suppresses a header field that would otherwise be added:

http:get('https://example.com/search', {
  'query': { 'q': 'xml database', 'limit': 10 },
  'headers': { 'Accept': 'text/csv', 'User-Agent': () },
  'response-body': 'text',
  'timeout': 2.5
})?body

Streaming

As the ZIP archive has a binary media type, the response body is returned as lazy item, and its contents are streamed to a file with constant memory consumption:

file:write-binary(
  'output.zip',
  http:get('https://files.basex.org/xml/xmark/111mb.zip')?body
)

With response-body set to binary, this behavior can be enforced for arbitrary requests.

Uploads work equally well with constant memory: The contents of the addressed file will be streamed to the target server:

http:put('http://localhost:8080/rest/archive/huge.zip', file:read-binary('huge.zip'))

Cookies

With the cookies option, the cookies that a server assigns are attached to the subsequent requests of a query. As the cookies are also sent to redirected addresses, a login that ends with a redirect can be followed by requests to protected resources:

void(
  http:post('https://example.com/login', { 'user': 'jack', 'password': '...' }, {
    'cookies': true(),
    'headers': { 'Content-Type': 'application/x-www-form-urlencoded' }
  })
),
http:get('https://example.com/download', { 'cookies': true() })

Digest Authentication

If the server requests digest authentication, the credentials are supplied with the second request:

http:get('https://example.com/protected', {
  'auth': { 'username': 'jack', 'password': '...', 'method': 'digest' }
})

POST Request

POST request to the BaseX REST Service, specifying a username and password. The element is serialized as XML, and the query string is wrapped in a CDATA section, so that it is sent as text instead of being evaluated by the client:

http:post(
  'http://localhost:8080/rest',
  <query xmlns='http://basex.org/rest'>
    <text><![CDATA[
      <html>{
        for $i in 1 to 3
        return <div>Section { $i }</div>
      }</html>
    ]]></text>
  </query>,
  { 'auth': { 'username': 'admin', 'password': '...' } }
)?body

<html>
  <div>Section 1</div>
  <div>Section 2</div>
  <div>Section 3</div>
</html>

File Upload

Performs an HTML file upload. In the RESTXQ code, the uploaded file is written to the temporary directory:

let $path := 'file-to-be.uploaded'
return http:post('http://localhost:8080/write-to-temp',
  {
    'headers': {
      'Content-Disposition': 'form-data; name="files"; filename="' || file:name($path) || '"'
    },
    'body': file:read-binary($path)
  },
  { 'multipart': true() }
)

declare
  %rest:POST
  %rest:path('/write-to-temp')
  %rest:form-param('files', '{ $files }')
function local:file-upload(
  $files  as map(xs:string, xs:base64Binary)
) as empty-sequence() {
  for key $file value $content in $files
  return file:write-binary(file:temp-dir() || $file, $content)
};

Response Conversion

CSV, JSON and HTML responses are automatically converted to an XML representation. The target format can be influenced with the parse-options option. With the format w3, the JSON response body is returned as map:

http:get('http://localhost:8080/json', {
  'parse-options': { 'json': { 'format': 'w3' } }
})?body

{ "abcde": 12345 }

Without the option, the response body is converted to the default XML representation:

<json type="object">
  <abcde type="number">12345</abcde>
</json>

declare
  %rest:path('/json')
  %output:method('json')
function local:json() {
  { 'abcde': 12345 }
};

See the CSV Functions, JSON Functions and HTML Functions for a list of the available options.

Status Only

Simple GET request. As the attribute status-only is set to true, only the response element is returned.

http:send-request(<http:request method='get' status-only='true'/>, 'https://basex.org')

<http:response status="200" message="OK" href="https://basex.org" version="HTTP/2">
  ...
  <http:header name="content-type" value="text/html"/>
  ...
  <http:header name="server" value="Apache"/>
  ...
  <http:body media-type="text/html"/>
</http:response>

Extensions

BaseX provides the following extensions to the EXPath specification:

• The functions http:get, http:post, http:put, http:patch, http:delete, http:head, http:options, http:query and http:send provide a map-based interface (see Request Options and Response Record).
• The cookies attribute of the http:request element enables cookie handling for the requests of a query.
• The xml, csv, json and html attributes of the http:request element control how response bodies are converted (see Request 1.0 Attributes).
• The proxy and verify attributes of the http:request element configure the connection of a single request.
• Binary response bodies are returned as lazy items, and file-based request payloads are streamed.
• The href and version attributes of the http:response element contain the final URI and the protocol version of the response.
• Compressed responses are requested by default: an Accept-Encoding header is added unless one is supplied, and if the addressed web server provides support for the gzip compression algorithm, the response will automatically be decompressed.
• Unless they are supplied with the request, an Accept: */* header and a User-Agent string with product name, version, Java version and operating system (e.g. BaseX/13.0 (Java 25; Windows 11 amd64)) are added. The same user agent is used by all other functions that retrieve remote resources, such as fetch:binary or fn:doc.
• Response header names are returned in lower case: field names are case-insensitive, and HTTP/2 servers transmit them in lower case.
• Basic and digest authentication are supported. Digest challenges with the MD5, SHA-256 and SHA-512-256 algorithms (including their -sess variants) are answered. Credentials can also be supplied via the URI (http://user:password@host/).
• Redirects are followed by the module itself: Authorization and Cookie headers are not sent to a target with a different origin, and downgrades from https to http are not followed.

Errors

Changelog

• Added: http:get, http:head, http:delete, http:options, http:post, http:put, http:patch, http:query, http:send: map-based interface with typed options and record results.
• Added: http:send-request: proxy, verify and xml attributes.
• Added: http:send-request: cookies attribute for collecting and resending cookies.
• Added: http:send-request: href and version attributes in the http:response element; SHA-256 and SHA-512-256 digest authentication.
• Updated: http:send-request: follow-redirect accepts a maximum number of redirects; auth-method is case-insensitive; timeout accepts fractions of a second.
• Updated: Compressed responses are requested by default; redirects are no longer forwarded to another origin with credentials, and downgrades from https to http are not followed.
• Updated: http:send-request: response header names are returned in lower case.
• Updated: http:send-request: binary response bodies are returned as lazy items; file-based request payloads are streamed.

• Updated: http:send-request: csv, json and html attributes added.

• Updated: Implementation based on the new Java HTTP Client.

• Updated: support for gzipped content encoding

• Added: digest authentication

• Updated: http:send-request: HC0002 is raised if the input cannot be parsed or converted to the final data type.
• Updated: errors are using text/plain as media-type.