Use of xs:hexBinary() comes with strange results

Hi, I came around this, the other day:

(: results in � (some (unknown?) binary char) :)
declare function local:test1($string) {
  xs:hexBinary($string)
};

(: results in *[convert:string] String conversion: Decoding error: xff.* :)
declare function local:test2($string) {
  convert:binary-to-string(xs:hexBinary($string))
};

(: results in � :)
declare function local:test3($string) {
  xs:hexBinary($string)
 ,convert:binary-to-string(xs:hexBinary($string), "UTF-8", true())
};

let $input1 := "c3"

let $input2 := "28"

return ( local:test3($input1), local:test3($input2) ) I came around this, when I wanted to unescape an IRI by converting the 2 digits after the '%' to their character representation. What buffles me the most is, that in local:test1#1 I get the unrecognizable binary char for the xs:hexBinary call. If, however, as done in local:test2#1 the very same expression becomes part of a sequence, then I get back the desired character. And if I use convert:binary-to-string#1 I get an error, while using the 3-arity version, I do not get the error, but the unreadable binary char. How can I simply get back any character, readable by a human, from a hexadecimal value?

Hi Andreas, I think what you are observing is the following: UTF-8 encoded stings can optionally denote a multi-byte sequence, with the number of leading 1-s defining the multibyte pattern length. c.f. https://en.wikipedia.org/wiki/UTF-8 <https://en.wikipedia.org/wiki/UTF-8> In your example, c3 decodes to:

And the two leading 1s will tell the UTF-8 decoder to read a second byte — which is missing — hence decoding fails with an error or if you use the fallback-option it will return a � While decoding ASCII, where only 127 bits are used, this is no problem as UTF-8 shares the same character positions with the ascii table. Your „C3“ character however is not in ascii but most probably ISO-8859-1 or CP1252? So while a glance at https://tools.ietf.org/html/rfc3986 <https://tools.ietf.org/html/rfc3986> says URI Characters should be encoded in UTF-8 in practice chances are you encounter values that are encoded using some „local“ encoding. If your string is not UTF-8 encoded you may only guess what the correct encoding is. You may send a predefined string that is known to be of two bytes length in UTF-8, such as: ä that will be either converted to „%C3%A4“ if it is unicode or to a well known single byte such as for example „E4“ in ISO-8859-1. Depending and what you receive by your client for that given string you may assume it encodes its data either utf-8 or latin1. You can check what your string would be encoded to:

Sorry for the long mail, hope the explanation is useful for you, even though the solution is not sooo simple and involves guessing :-) Best Michael